Incident triage

Use this page when an alert fires, a user reports something broken, or a dashboard looks wrong. The goal of triage is not the root cause. The goal is to answer three questions fast: how bad is it, what changed, and what is the safest way to stop the bleeding. Root cause comes later, in the postmortem.

Follow the agent operating rules throughout. During an incident they matter more, not less.

Declare first, investigate second

If customers are affected, or might be, an incident exists. Say so early and in one place.

• Open or find the incident channel or ticket. One place, linked from everywhere else.
• Name a human incident commander. An agent can scribe and investigate; it does not command.
• Post a first status within 5 minutes, even if it says "investigating, impact unknown".

[INCIDENT] <service> elevated errors since ~<HH:MM UTC>
Impact: <who/what is affected, or "unknown, checking">
Status: investigating
Next update: <HH:MM UTC>
IC: <name>   Scribe: <agent or name>

Size the impact with numbers

Pick a severity from facts, not feelings.

Impact is severity times users affected, added up over time.

$$I \approx \int_{t_0}^{t_1} \text{severity}(t) \cdot \text{users}(t)\,dt \qquad I = \text{duration} \times \text{severity} \times \text{users}$$

For example, 45 minutes at 40% severity across 12,000 users is \(45 \times 0.4 \times 12{,}000 = 216{,}000\) user-minutes. Use the right-hand form for a quick estimate.

• Error rate and latency for the affected endpoint, compared with the same hour yesterday.
• Which environments, regions and tenants are affected. Is it everyone or a slice?
• Is the error budget for the related SLO burning fast?

Find what changed

Most incidents follow a change. Check the last few hours before the first bad data point.

• Deploys: pipeline history for the service and its dependencies.
• Config and feature flags: recent flag flips, parameter changes.
• Infrastructure: Terraform applies, scaling events, AWS Health events.
• Certificates and secrets: anything that expired or rotated (see certificate renewal).
• Upstream vendors: status pages for DNS, CDN, identity providers, and other dependencies.

# Who changed what in the last 3 hours (read-only)
aws --profile <readonly-profile> --region <region> cloudtrail lookup-events \
  --start-time "$(date -u -d '-3 hours' +%FT%TZ)" \
  --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false \
  --query 'Events[].{t:EventTime,who:Username,what:EventName,src:EventSource}' \
  --output table

# Recent ECS deployments for a service
aws --profile <readonly-profile> --region <region> ecs describe-services \
  --cluster <cluster> --services <service> \
  --query 'services[0].deployments[].{status:status,td:taskDefinition,created:createdAt,rollout:rolloutState}'

More read-only recipes are in AWS read-only investigation.

Narrow it down with a decision tree

flowchart TD
  A[Alert or report] --> B{Customers affected?}
  B -- no --> C[SEV3/4 ticket, keep watching]
  B -- yes --> D{Recent change in the window?}
  D -- yes --> E[Propose rollback of that change]
  D -- no --> F{Dependency unhealthy?}
  F -- yes --> G[Check vendor or upstream status, fail over if designed to]
  F -- no --> H{Capacity or saturation?}
  H -- yes --> I[Propose scale-out]
  H -- no --> J[Gather logs and traces, escalate to service owner]

Mitigate with the smallest reversible step

Mitigation beats diagnosis. Prefer, in order:

1. Roll back the last change (see blue-green deploys with rollback).
2. Turn off the feature flag.
3. Scale out or shift traffic away from the bad instance, zone or region.
4. Fail over to the standby.
5. Only then, a forward fix.

• Write the proposed action, its blast radius and how to undo it.
• Get the incident commander's explicit yes for this specific action.
• Do it, or hand the exact command to the human who will.
• Watch the key metric for 10 minutes before declaring it mitigated.

Verify and hand off

• Error rate and latency back to the baseline you recorded earlier.
• Synthetic checks or smoke tests pass.
• Status update: "mitigated, monitoring", with the time.
• Timeline captured: first signal, detection, declare, mitigate, resolve (UTC).
• Postmortem ticket opened and linked, owner named.

Pitfalls

• Retrying a failed mitigation blindly. Stop, read the output, report.
• Fixing forward in production while a one-click rollback exists.
• Debugging in a private DM. Everything goes in the incident channel.
• Pasting log lines that contain tokens, card numbers or customer data into the channel or a model. Redact first.
• Treating "no alerts" as "healthy". Check that the monitor itself is not in No Data or muted (see alert and monitor hygiene).