Containers & ECS

The sealed package

Most container problems I've had to debug in production were not container problems. They were applications that reached out to the environment during startup (an SDK call to fetch a parameter, a decrypt operation, a role assumption) and got a different answer than they did in the environment where they were tested.

This single constraint eliminates a whole class of failure. If the container never calls the cloud API at startup, it can't fail because of a throttled API, a missing permission on a role it only uses at boot, or a regional endpoint being slow. It also can't behave differently in QA than in production for reasons invisible in the application code.

Configuration comes in, the container doesn't go out

In practice this means the platform is responsible for resolving secrets, and the task definition is where that happens, not application startup code.

{
  "name": "api",
  "image": "…/api@sha256:9f2c…",
  "environment": [
    { "name": "LOG_LEVEL", "value": "info" }
  ],
  "secrets": [
    { "name": "DB_PASSWORD",
      "valueFrom": "arn:aws:secretsmanager:…:secret:prod/api/db" }
  ]
}

The application sees DB_PASSWORD as an ordinary environment variable. It has no idea Secrets Manager exists, holds no permission to read it, and contains no code that would break if you moved to a different secret store.

• Inject secrets through the task definition's secrets block.
• Let the infrastructure handle decryption; keep KMS out of application code.
• Scope the task role to the minimum the running application needs, which, done properly, is usually very little.
• Pin images by immutable digest, not by a mutable tag like latest.

• No SDK calls for configuration during startup: no SSM, no KMS, no STS.
• No baked-in config files that differ per environment. That's what makes an image environment-specific, which defeats the point of building it once.

If it needs cloud credentials to boot, it's not portable

My test for whether the sealed-package rule is actually being followed: can a developer run the service on a laptop with no AWS credentials at all, using a .env file?

If yes, the configuration boundary is clean. If no, the application has a hidden dependency on its runtime environment, and that dependency will eventually produce a defect that only reproduces in one environment, the most expensive kind to chase.

This is also the cheapest available fix for local-versus-QA divergence. When developers test against a materially different topology than the one QA maintains, QA stops being a validation system and becomes a reactive debugging service for problems that were structurally guaranteed.

Image size is a delivery metric, not a tidiness metric

Multi-gigabyte images are common and nearly always accidental: build toolchains, caches, and dev dependencies that were never meant to ship. It's easy to dismiss as cosmetic, but it shows up directly in the numbers that matter: every pipeline run pushes it, every scale event pulls it, and both get slower.

FROM node:22-slim AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci                      # dev deps live and die here
COPY . .
RUN npm run build

FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev && npm cache clean --force
COPY --from=build /app/dist ./dist
USER node                      # never run as root
CMD ["node", "dist/server.js"]

Order matters as much as the multi-stage split: dependency layers go before source layers, so a code change doesn't invalidate the dependency cache. That ordering is often worth more pipeline time than the size reduction itself.

Blue/green, and meaning it about rollback

Blue/green deployments give you a running previous version to shift traffic back to, which is the difference between a rollback measured in seconds and one measured in a pipeline run.

The part teams skip is the health check. A deployment strategy is only as good as the signal that decides whether the new version is healthy, and /health returning 200 because the web server started tells you almost nothing.

• Make the health check verify the dependencies the service actually needs to do its job.
• Set the deregistration delay to something realistic for in-flight requests, or you'll drop them on every deploy.
• Test the rollback path deliberately. An untested rollback is a hypothesis.

And keep the deployment separate from the release. Blue/green controls which version is serving; feature flags control which users see the new behavior. Those are different questions and they want different mechanisms.

Anti-patterns

• Running as root. Costs one line to fix, and it's the first thing any scanner flags.
• latest in a task definition. You lose the ability to say what's actually running, which makes incident timelines guesswork.
• Secrets in environment variables in the task definition JSON. Use secrets, not environment. The plaintext version ends up in revision history.
• One image per environment. If dev and prod images differ, you tested a different artifact than you shipped.
• Vertical scaling as a fix for architecture. If a service only works on a very large instance, capping the instance size is the honest move. It forces the actual problem into view.

With Claude

Auditing a Dockerfile

Ask for the audit against a named standard rather than in general. "Is this good?" produces generic advice, while naming the concerns produces findings.

Review this Dockerfile as a container security and build-performance
audit. Structure the output as a table: finding, severity, fix.

Check specifically:
  - runs as non-root
  - multi-stage, with no build toolchain in the final image
  - layer order maximizes cache reuse across code-only changes
  - no secrets or credentials in any layer (including intermediates)
  - base image is pinned and currently supported

Ignore style preferences. Only report things with a concrete impact.

Finding startup-time cloud calls

The sealed-package rule is easy to state and easy to violate quietly, especially in a codebase with several years of history. This is a good grep-plus-judgment task.

Search this service for configuration being fetched at startup rather
than injected: SSM parameter reads, Secrets Manager calls, KMS
decrypts, STS assume-role, any SDK client constructed at import time.

For each: file and line, what it fetches, and what it would take to
move it into the task definition instead.

I want to know whether this service can boot with no AWS credentials.
Answer that question explicitly at the end.

Explaining why a task won't start

ECS failures are famously terse: a task that stops immediately with an exit code and no application logs. Give the model the stopped-task reason, the task definition, and the log tail together; the answer is usually in the interaction between them rather than in any one of them.