Postmortems

Use this page after any SEV1 or SEV2, any near miss that could have been one, and any incident that happened twice. A postmortem exists to change the system so the same thing cannot hurt users again. It is not a report card on people.

Rules for a useful postmortem

• Blameless: describe what people saw and why their action made sense at the time. Use roles, not names.
• Draft within 2 business days while memory is fresh. Review within 5.
• Every action item has an owner, a ticket and a due date.
• Prefer fixes that remove a class of failure (automation, guardrails) over "be more careful".
• No secrets, customer data or raw logs with identifiers in the document.

How an agent drafts one

1. Collect the timeline from the incident channel, alert history, deploy history and CloudTrail (read-only).
2. Convert every time to UTC and sort.
3. Fill the template below. Mark anything inferred as (inferred).
4. Propose 3 to 6 action items, ranked by how much risk each removes.
5. Hand the draft to the incident commander. Do not publish it yourself.

Template

## Summary
<2-3 sentences: what broke, who was affected, for how long, how it was fixed.>

## Impact
- Duration: <start> to <end> UTC (<N> min)
- Users affected: <count or percent, segment>
- SLO impact: <budget consumed>
- Data impact: <none / describe>

## Timeline (UTC)
| Time | Event |
| --- | --- |
| HH:MM | First bad data point |
| HH:MM | Alert fired / report received |
| HH:MM | Incident declared, IC assigned |
| HH:MM | Mitigation applied |
| HH:MM | Resolved |

## Root cause and contributing factors
<What happened, in causal order. Ask "why" until you reach something you can change.>

## Detection
<How we found out. Could we have found out sooner? Did monitoring work?>

## What went well
## What went poorly
## Where we got lucky

## Action items
| Action | Type | Owner | Ticket | Due |
| --- | --- | --- | --- | --- |
| | prevent / detect / mitigate / process | | PROJ-123 | |

Worked example (fictional)

Summary

For 47 minutes, about 30 percent of API requests to the orders service returned 503. A new release passed smoke tests, but the production configuration pointed the service at a connection pool size that the database could not serve under peak load. Rolling back the release restored service.

Impact

• Duration: 14:02 to 14:49 UTC (47 min)
• Users affected: roughly 30 percent of order submissions failed; clients retried, so 4 percent were lost
• SLO impact: 65 percent of the monthly availability budget consumed
• How that was worked out: \(\text{budget used} = \dfrac{\text{duration} \times \text{error fraction}}{\text{budget}}\). With a 99.95 percent SLO the 30-day budget is 21.6 minutes, and \(47 \times 0.30 = 14.1\) minutes is 65 percent of it.
• Data impact: none; failed requests were never committed

Timeline (UTC)

Root cause and contributing factors

• The release raised the per-task database pool from 10 to 50. With 20 tasks at peak, that exceeded the database connection limit.
• Smoke tests ran on one candidate task at low load, so they could not catch it.
• The config change lived in the same MR as a code change, and review focused on the code.
• Rollback took 17 minutes after it was proposed because nobody on the call had prod deploy rights.

Action items

Pitfalls

• "Human error" as a root cause. Ask what made the error easy.
• Action items with no owner. They never happen.
• Postmortems nobody reads. Share a three-line summary in the team channel and review open action items monthly.