The First Hour of an Incident

Three roles and one shared document. What a small team actually needs, and why mitigation comes before understanding.

Incidents Christopher Roos 3 min read

Small teams often skip incident process on the grounds that they are too small to need it. Then something serious happens, four people investigate the same theory in parallel, nobody talks to the customer, and afterwards nobody can reconstruct what was tried. Process is not overhead at that point, it is the thing that was missing.

You do not need a formal incident command structure. You need three roles and a shared place to write things down, and for small teams two of the roles are often the same person.

Somebody is coordinating, and it is not the person debugging

This is the single highest-value rule. The person with their hands in the system cannot also be tracking who is doing what, deciding when to escalate, and updating everyone else. Trying to do both means doing both badly, and the debugging is usually what suffers.

The coordinator does not need to be the most technical person in the room. They need to know what everyone is working on, notice when two people are duplicating, and own the decision about when to escalate or roll back.

Somebody is communicating

Regular updates, on a fixed cadence, even when the update is "no change, still investigating." The cadence matters more than the content, because without it people start asking individually, and every one of those questions interrupts the person who should be working.

A predictable heartbeat every fifteen minutes buys silence in between, which is the actual goal.

Everything goes in one place, as it happens

A single channel or document where every action and observation is written down at the time. Not afterwards, at the time. Two reasons.

During the incident, it stops the same thing being tried twice and gives anyone joining a way to catch up without asking for a summary. After the incident, it is the only honest timeline you will get. Memory reorganises events into a story that makes sense, and the story is usually wrong in the places that matter.

Mitigate before you understand

The instinct of most engineers, including me, is to find the cause. During an active incident that instinct is wrong. Restore service first, understand it afterwards.

Roll back the recent change even if you are not sure it is responsible. Fail over. Turn off the feature. Shed the load. These are cheap and reversible, and the diagnosis is much more pleasant to do when nothing is on fire. The one caveat is preserving evidence: take the logs and a snapshot before you restart the thing that is misbehaving, because otherwise the postmortem has nothing to work with.

Declare early, downgrade freely

Teams under-declare because declaring feels like escalation. It is easier to make calling an incident cheap than to make people bolder. If standing it down after ten minutes with "false alarm, all clear" carries no cost, people will call them at the right time instead of forty minutes late.