Observability - Playbook - Christopher Roos

Three questions, three tools

Signal	Answers	In an incident
Metrics	How often, how fast, how many	Shows the symptom and when it started
Logs	What happened, exactly	Explains the mechanism
Traces	Where in the system	Locates it to a service or a call

Metrics without traces means you know latency doubled, not which of nine services did it. Traces without logs means you know where, not why.

Structured logs, or you don't have logs

Unstructured lines are searchable by a person and by nothing else.

a log line worth having

{
  "ts":         "2026-07-27T18:04:11.912Z",
  "level":      "error",
  "service":    "catalog-api",
  "version":    "sha256:9f2c…",   // which build is this
  "request_id": "3f9c1a…",          // follows the request everywhere
  "route":      "POST /v1/items",
  "duration_ms": 4182,
  "event":      "upstream_timeout",  // stable, greppable
  "upstream":   "upstream-api"
}

request_id, passed across every service boundary.
version, the exact build. "Started after the 14:02 deploy" is the most useful sentence in an incident.

Use a stable event name, not a formatted sentence. Reworded sentences break the dashboards built on them.

Never log secrets, tokens, or personal information. Redact in the logging layer.
Do not run debug logging in production all the time.

SLOs and error budgets do the arguing

Shipping versus stability is usually argued on feel. An error budget turns it into a number both sides agreed to in advance.

SLI: the measurement. successful_requests / total_requests.
SLO: the target. 99.9% over a rolling 30 days.
Error budget: what the SLO leaves over.

$$\text{error budget} = (1 - \mathrm{SLO}) \times \text{window}$$

Worked example

43.2minutes of budget

At 99.9% over 30 days, that is $0.001 \times 43{,}200 = 43.2$ minutes of failure.

Alert on how fast you spend it:

$$\text{burn rate} = \frac{\text{observed error rate}}{1 - \mathrm{SLO}}$$

Worked example

14.4×burn rate

An error rate of 1.44% against a 0.1% allowance is a burn rate of 14.4. That spends 2% of the month in one hour, so it is the standard one-hour fast-burn page.

Budget healthy? Ship. Budget spent? Reliability work comes first.

Agree on the rule once, when nobody is upset. Then it applies on its own.

Set the SLO a little stricter than the contractual SLA, so the internal target trips first. An SLO negotiated during an outage is just a record of what happened.

Alert on symptoms, not causes

High CPU is not an incident. Failing requests are. Cause-based alerts page for conditions that hurt nobody, and miss new failures.

Page on user-visible symptoms: error rate, latency past a threshold, throughput falling off.
Every page needs a runbook and a plausible action. If there is nothing to do, it is a dashboard.
Treat a noisy alert as a defect with a ticket.

Latency alerts should watch utilization too. In a simple single-server queue, utilization is $\rho = \lambda/\mu$ and the average wait is:

$$W = \frac{1}{\mu - \lambda}$$

Worked example

50 msat 80% utilization

200 msat 95% utilization

With $\mu = 100$ requests a second, 80% utilization gives 50 ms and 95% gives 200 ms. Latency is flat for a long time and then climbs fast. Page on the climb, not on CPU.

Wait against utilization, one server at 100 requests a secondFifteen points more load, from 80% to 95%, takes the average wait from 50 ms to 200 ms.

Then measure the pager itself. Alert precision is pages that needed action divided by total pages. Nine useful pages out of 30 is 30%.

Logging comes before automated incident response

An agent that triages incidents has to read what happened. If the logs are prose with no request ID and no version, the agent is in the same spot as a human, guessing, with more confidence.

Structured logging is the prerequisite for automating incident response.

Fix observability before you build agents on top of it.

Anti-patterns

Logging everything at maximum verbosity. Ingest costs grow. Signal does not.
Dashboards nobody opens. They become a maintenance cost.
Alerts with no runbook. The response depends on who is on call.
A different log format per service. Cross-service queries stop getting written.
Monitoring only what is easy to measure. CPU is easy. Whether checkouts succeed is what matters.

With Claude

Correlating a spike

Give it the window, the symptom, and the deploy timeline together. The answer is usually in the overlap.

prompt

Error rate on catalog-api went from 0.1% to 4% between 14:02 and 14:19.

Attached: error logs for that window, and the deploy log for the day.

1. Group the errors by `event` and `upstream`. Which dominates?
2. Does the start time line up with any deploy? Name the version.
3. Give me three candidate causes ranked by how well they fit the
   evidence, and for each, the one query that would rule it out.

Flag explicitly if the errors started BEFORE the nearest deploy.

That last line guards against anchoring on the nearest deploy.

Turning an incident into detection

Ask what alert would have fired, with its threshold and its reason. You get a concrete artifact, not "improve monitoring."

Auditing what a service logs

prompt

Audit the logging in this service against these requirements:
  - structured JSON on every path, including error handlers
  - request_id present and propagated to downstream calls
  - build version on every line
  - no secrets, tokens, or PII (check error paths especially,
    that's where objects get dumped wholesale)
  - stable event names rather than formatted sentences

Output a table: requirement, pass/fail, file:line evidence.
Then: could an on-call engineer diagnose a 500 from these logs alone?