Blue-green deploys with one-click rollback

Use this page when designing or fixing a deployment pipeline, or when a deploy goes bad and you need to roll back. The pattern works on any platform: Kubernetes, a container service, VMs behind a load balancer, serverless aliases. The worked example uses a container service on AWS and a generic CI skeleton.

The pattern in one picture

flowchart LR
  B[Build image once] --> D[Deploy candidate]
  D --> S{Smoke tests pass?}
  S -- no --> X[Delete candidate, live untouched]
  S -- yes --> R[Record current live version]
  R --> P[Promote candidate to live]
  P --> W{Health OK for N minutes?}
  W -- no --> RB[Rollback to recorded version]
  W -- yes --> C[Clean up candidate]

The rules that make it safe:

• Build once, promote the same artifact. Dev, staging and prod run the identical image digest. Never rebuild per environment.
• The candidate gets real traffic only from tests until it is promoted.
• Record the previous version before promoting. Rollback means "go back to the recorded version", not "guess".
• Rollback is a single manual job that needs no new build and no thinking.
• Lower environments deploy automatically; production is a manual, protected step.

Environments and gates

Worked example: ECS with a candidate service

Two ECS services per environment: <service> (live, behind the main target group) and <service>-candidate (behind a test listener or a separate target group that only the smoke tests hit).

1. Register a new task definition revision with the new image digest.
2. Update <service>-candidate to that revision and wait for it to be stable.
3. Run smoke tests against the candidate endpoint.
4. Store the live service's current task definition ARN, for example in a parameter /deploy/<service>/<env>/previous.
5. Update <service> to the new revision and wait for steady state.
6. Scale the candidate to zero.

# Read-only checks you can run any time
aws --profile <readonly-profile> --region <region> ecs describe-services \
  --cluster <cluster> --services <service> <service>-candidate \
  --query 'services[].{name:serviceName,td:taskDefinition,running:runningCount,desired:desiredCount}'

aws --profile <readonly-profile> --region <region> ssm get-parameter \
  --name /deploy/<service>/<env>/previous --query Parameter.Value --output text

Pipeline jobs use a deploy role assumed through OIDC, never static keys.

Worked example: a CI skeleton

stages: [build, dev, sandbox, prod]

.deploy: &deploy
  image: <deploy-image>
  # Obtain a short-lived cloud role through OIDC here (see the OIDC page).
  script:
    - ./ci/deploy-candidate.sh "$ENV" "$IMAGE_DIGEST"
    - ./ci/smoke.sh "$ENV"
    - ./ci/promote.sh "$ENV"
    - ./ci/cleanup-candidate.sh "$ENV"

build:
  stage: build
  script:
    - ./ci/build-and-push.sh   # writes IMAGE_DIGEST to build.env
  artifacts:
    reports:
      dotenv: build.env

deploy_dev:
  <<: *deploy
  stage: dev
  variables: { ENV: dev }
  environment: { name: dev }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy_sandbox:
  <<: *deploy
  stage: sandbox
  variables: { ENV: sandbox }
  environment: { name: sandbox }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

deploy_prod:
  <<: *deploy
  stage: prod
  variables: { ENV: prod }
  environment: { name: prod }
  needs: [build, deploy_sandbox]
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

rollback_prod:
  stage: prod
  image: <deploy-image>
  variables: { ENV: prod }
  environment: { name: prod, action: access }
  script:
    - ./ci/rollback.sh "$ENV"   # reads /deploy/<service>/prod/previous and redeploys it
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  allow_failure: true

Mark prod as a protected environment in your CI so only approvers can press the button. Many platforms also offer a re-deploy of an earlier successful job, which is a second rollback path.

Smoke tests that are worth running

• Health endpoint returns 200 and reports the expected version.
• One read path and one write path against test data succeed.
• Dependencies reachable: database, cache, queue, one downstream API.
• Fails fast: whole suite under 2 minutes, non-zero exit on any failure.

Rollback runbook

• Open the pipeline for the last good production deploy, or the latest pipeline on the default branch.
• Run rollback_prod. It needs no inputs.
• Confirm the live service shows the previous task definition and is stable.
• Confirm error rate is back to baseline.
• Post in the incident channel and link the job.

Alternatives

Pitfalls

• Database migrations that are not backward compatible. Expand first, deploy, contract later, or rollback breaks.
• Rebuilding the image in the prod job. Promote the digest you tested.
• Storing "previous" only in the pipeline log. Store it somewhere the rollback job reads.
• A rollback job that itself needs the build stage. It must run on its own.