Terraform

The opinion

After the first apply, Terraform's job is to reconcile intent against reality. That only works if the state is right.

Treat state as production data

Use remote state in S3 with a DynamoDB lock table from the first commit.

terraform {
  backend "s3" {
    bucket         = "org-tfstate-prod"
    key            = "platform/api/terraform.tfstate"
    region         = "us-east-2"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

Turn on S3 versioning. Old state versions are your only real rollback. Never hand-edit state. Use terraform state mv for renames, terraform import for resources state does not know about, and terraform state rm to forget a resource without destroying it.

Modules are components, not environments

A modules/production directory is a configuration in disguise. Nothing is shared, so environments drift.

• Build modules around components: an ECS service, an RDS cluster, a VPC.
• Give each a clear input and output contract. Callers treat it like an API.

• Do not build a module per environment.
• Do not add a boolean to make one module act like two.

Separate directories, not just variables

If dev and prod differ only by a .tfvars file, one wrong flag separates a staging apply from a production outage. Make the isolation structural.

modules/
  ecs-service/          # component, reused everywhere
  rds-cluster/
environments/
  dev/
    main.tf             # calls modules, own backend key
    backend.tf
  staging/
  production/           # separate state, separate blast radius

Separate root modules mean separate state files. A mistake in one environment cannot reach another.

Make the plan something a human reads

1. Engineer opens a PR.
2. Automation runs plan and posts it as a comment.
3. A human reviews the actual diff.
4. Apply runs before the merge, not after.
5. Merge once the apply succeeds.

A plan touching sixty resources will not get real review. Split the change.

Import is the undo button people forget

A resource that exists in the cloud but not in state is unmanaged, not lost. I used this to recover a bucket policy that a redeploy failed to restore. Importing the resource and re-applying put the intended policy back, because state still held the intent.

# tell state about a resource that already exists
terraform import aws_s3_bucket_policy.reports org-reports-prod

# now the diff is meaningful: intent vs. reality
terraform plan

Anti-patterns

• Console changes to managed resources. Drift, found during an unrelated apply.
• Committing .tfstate. It holds secrets and blocks concurrent work.
• apply -auto-approve from a laptop. Fine in CI against a reviewed plan.
• One giant root module. Blast radius becomes the whole account.
• Unpinned provider versions. A minor bump that changes a default costs an afternoon.

With Claude

A plan is a precise description of what is about to happen. That makes it a good input for a model.

Reviewing a plan for blast radius

Act as a senior infrastructure engineer reviewing this terraform plan
before I apply it to production.

Identify, as a table:
  - every resource being DESTROYED or REPLACED, and the downtime implied
  - any change to IAM, security groups, or network routing
  - anything that would drop data (volumes, snapshots, retention windows)
  - anything stateful that a replace would recreate empty

Then tell me the single riskiest line and why.
Do not summarize the safe changes.

The last two lines matter. Without them you get a summary of ninety changes.

Extracting a module

Give it the concrete resources first. Ask for the abstraction second.

Here are three near-identical ECS service definitions from our repo.

Extract a single reusable module:
  - inputs only for what genuinely differs between the three
  - sensible defaults for everything else
  - outputs for the ARNs the callers currently reference

Show me the variables.tf first and wait. I want to agree on the
interface before you write the resources.

Explaining drift

When a plan shows a change nobody made, ask what would have to be true for that diff to appear. That gives you hypotheses to check.