Drift Detection

Check the drift periodically and track it using GitHub Issues

tfaction >= v0.6.0

Blogs:

Drift Detection is the feature to detect drift. You can track the drift using GitHub Issues and resolve the drift continuously.

caution

This feature detects drift but doesn't resolve it automatically. You have to resolve it yourself.

This feature is disabled by default. To enable this feature, please see Set up.

tfaction creates GitHub Issues per working directory and manages each working directory's drift with GitHub Issues.

tfaction opens an Issue when a drift is detected, and closes the Issue when the drift is resolved.

tfaction creates only one Issue per working directory and reuses the same issue. If drift is detected again after drift is resolved once, tfaction reopens the same issue instead of creating a new issue.

tfaction checks if the drift exists at the following timing.

A pull request is merged and terraform apply or tfmigrate apply are run
terraform plan is run periodically by a dedicated GitHub Actions Workflow

The result of terraform apply, tfmigrate apply, or terraform plan is posted to the issue. The comment has links to the GitHub Actions Workflow and the pull request.

e.g. An Issue was opened because terraform apply failed.

e.g. An Issue was closed because terraform apply succeeded and the drift was resolved.

e.g. Drift is checked periodically.

Requirements

tfaction >= v0.6.0
tfaction-go >= v0.1.2
github-comment >= v5.2.1
tfcmt >= v4.3.0

Set up

Update tfaciton-root.yaml
Install tfaction-go by aqua
Add two GitHub Actions workflows
Update the apply workflow
Run schedule-create-drift-issues.yaml manually only once

1. Update tfaciton-root.yaml

Please configure Drift Detection.

drift_detection: {} # Enable Drift Detection with the default settings

drift_detection:
  enabled: true
  issue_repo_owner: suzuki-shunsuke # Repository owner of GitHub Issues. By default, Repository where GitHub Actions is run
  issue_repo_name: tfaction-example # Repository name of GitHub Issues. By default, Repository where GitHub Actions is run
  num_of_issues: 1 # The number of issues that scheduled job handles. The default value is `1`
  minimum_detection_interval: 1 # The default value is 168 (7 days). The scheduled workflow picks out working directories whose issues were updated before `minimum_detection_interval` hours

By default, drift issues are created in the same repository where tfaction is run. If you change the repository, you have to use GitHub App token or personal access token instead of GitHub Actions token because GitHub Actions token can't access the other repository.

2. Install tfaction-go by aqua

Please install tfaction-go in the repository root aqua.yaml.

$ aqua g -i suzuki-shunsuke/tfaction-go

3. Add two GitHub Actions workflows

Please run these workflows periodically.

schedule-create-drift-issues.yaml: Create Drift Issues periodically
schedule-detect-drifts.yaml: Test if each working directory has a drift periodically
(Optional) sync-drift-issue-description.yaml: Synchronize Drift Issue's description with the latest issue comment

3.1. schedule-create-drift-issues.yaml

Create Issues
Archive Issues whose working directories are not found

tip

This workflow closes issues immediately because drift detection isn't run against those working directories when issues are created. Issues will be reopened when the drift will be detected.

3.2. schedule-detect-drifts.yaml

Pick out some Issues not checked recently and check the drift and updates Issues
Archive Issues whose working directories are not found

tfaction-root.yaml's following settings affect the workflow.

drift_detection:
  num_of_issues: 1 # The maxmum number of issues that scheduled job handles. The default value is `1`
  minimum_detection_interval: 1 # The default value is 168 (7 days). The scheduled workflow picks out working directories whose issues were updated before `minimum_detection_interval` hours

This workflow picks out at most num_of_issues working directories whose issues were updated before minimum_detection_interval hours and checks if they have drifts.

The pseudo query to pick out issues is like the following.

repo:${repo} "Terraform Drift" in:title sort:updated-asc updated:<${now - minimum_detection_interval(hour)}

This means if all drift issues were updated within minimum_detection_interval hours from now, no working directory aren't picked out.

tip

Why is the parameter minimum_detection_interval needed? That is because the drift is checked by not only scheduled workflow but also apply workflow. If the apply workflow is run recently against a working directory, the scheduled workflow doesn't have to check the same working directory. So tfaction updates drift issues by not only the scheduled workflow but also the apply workflow, and restricts the target of the scheduled workflow by issue's last updated time and minimum_detection_interval.

3.3. sync-drift-issue-description.yaml

If you want to reflect the latest drift detection's result to drift issue's description, please add the workflow to the repository drift issues are created.

sync-drift-issue-description.yaml ⚠️ Please change github.actor properly

This workflow is optional.

4. Update the apply workflow

Please add some steps to terraform-apply and tfmigrate-apply jobs.

Run tfaction get-or-create-drift-issue before tfaction/setup

example

- run: tfaction get-or-create-drift-issue
  shell: bash
  env:
    GITHUB_TOKEN: ${{ github.token }}

- uses: suzuki-shunsuke/tfaction/setup@v0.6.0

This step gets a drift issue for the working directory. If a drift issue isn't found, a new issue is created. The environment variables about the drift issue are set.

Run tfaction/update-drift-issue in the end of the jobs. Set if: always() to run the step definitely.

example

- uses: suzuki-shunsuke/tfaction/update-drift-issue@v0.6.0
  if: always()
  with:
    status: ${{job.status}}
    github_token: ${{steps.generate_token.outputs.token}}

This step closes or reopens the drift issue according to the job result. If the job fails the issue is reopened. Or if the job succeeds the issue is closed.

caution

You have to update tfaction of all steps to v0.6.0 or later. If old tfaction is used in other steps drift detection doesn't work well.

5. Run `schedule-create-drift-issues.yaml` manually only once

Drift Detection doesn't work well if GitHub Issues don't exist. So please run the workflow manually to create issues only once.

Enable drift detection against only specific working directories

By default Drift Detection is enabled against all working directories, but you can enable Drift Detection against only specific working directories.

The priority is as following.

tfaction.yaml's drift_detection
target group's drift_detection
tfaction-root.yaml's drift_detection

e.g.

tfaction-root.yaml

drift_detection:
  enabled: false

target_groups:
- working_directory: aws/
  # ...
  drift_detection:
    enabled: true

tfaction.yaml

drift_detection:
  enabled: false

GitHub Access Token

The permission issues: write is required to update GitHub Issues.

GitHub Issue's title

Please don't change GitHub Issue's title basically because tfaction identifies issues using the title. The title must be unique.

If you change a target name of a working directory and you want to keep using the same drift issue, please change the issue title properly. Otherwise, tfaction will close the issue and create a new issue.

Adjust the frequency of Drift Detection

Please adjust the frequency of Drift Detection as you like.

The schedule of GitHub Actions Workflow
Drift Detection's configuration
- num_of_issues
- minimum_detection_interval

This depends on the number of working directories. Please pay attention to the cost and API rate limiting if you increase the frequency.

The following table shows the example.

No.	the number of working directory	frequency of workflow	`num_of_issues`	frequency per working directory (/ 1 times)
1	10	1 / hour	1	10 hour
2	7	1 / day	1	1 week
3	1000	1 / 30 min	3	1 week

How to handle issues

Even if issues are created by Drift Detection, there is no meaning if you don't resolve them.

How to use this feature is completely up to you, but we have some advice.

Don't handle all issues by only you. Handle issues by teams
Create a strategy to handle issues continuously
Create rules to handle issues and write a guide so that everyone can handle issues properly
Rotate person in charge of handling issues
Leave the issue handling to each working directory's owners
Review and improve the issue handling periodically

The real time notification to the chat tool may be noisy and exhaust you, so we recommend making the time to check issues periodically (e.g. daily or weekly) rather than the real time notification.

How to save cost 💰

If you want to save cost, there are some options.

Decrease the frequency of schedule-detect-drifts workflow
Stop running schedule-detect-drifts workflow. Even if schedule-detect-drifts is stopped, drift issues are updated according to the result of terraform apply and tfmigrate apply. Maybe this is enough useful
Use GitHub Actions Self hosted runner

Requirements​

Set up​

1. Update tfaciton-root.yaml​

2. Install tfaction-go by aqua​

3. Add two GitHub Actions workflows​

3.1. schedule-create-drift-issues.yaml​

3.2. schedule-detect-drifts.yaml​

3.3. sync-drift-issue-description.yaml​

4. Update the apply workflow​

5. Run schedule-create-drift-issues.yaml manually only once​

Enable drift detection against only specific working directories​

GitHub Access Token​

GitHub Issue's title​

Adjust the frequency of Drift Detection​

How to handle issues​

How to save cost 💰​