Observability

Three questions, three tools

The three pillars get recited so often that the actual point gets lost. They aren't three ways of doing the same thing. They answer genuinely different questions, and you need all three because incidents move between them.

The gaps are predictable. Logs without metrics means you can't alert, because you have no aggregate to threshold against. Metrics without traces means you know latency doubled but not which of nine services caused it. Traces without logs means you know which service and still don't know why.

Structured logs, or you don't really have logs

Unstructured log lines are searchable by a human and by nothing else. The moment you want to aggregate, alert, correlate across services, or hand the data to anything automated, you need structure.

{
  "ts":         "2026-07-27T18:04:11.912Z",
  "level":      "error",
  "service":    "payments-api",
  "version":    "sha256:9f2c…",   // which build is this
  "request_id": "3f9c1a…",          // follows the request everywhere
  "route":      "POST /v1/charges",
  "duration_ms": 4182,
  "event":      "upstream_timeout",  // stable, greppable
  "upstream":   "card-processor"
}

Two fields do most of the work:

• request_id, propagated across every service boundary. Without a correlation ID, reconstructing one request's path across services is manual archaeology.
• version, the exact build. "Started after the 14:02 deploy" is the single most useful sentence in an incident, and you can only say it if the logs carry the version.

Use a stable event name rather than a formatted sentence. Sentences change when someone edits the wording, and every dashboard built on them breaks silently.

• Never log secrets, tokens, card data, or personal information. Redact at the logging layer, not by asking people to remember.
• Don't log at debug level in production continuously. You'll pay for it in ingest cost and find nothing faster.

SLOs and error budgets do the arguing for you

The recurring tension between shipping and stability is usually argued on vibes: one side says "you're pushing too fast," the other says "you're blocking us," and whoever is more senior wins. An error budget converts that into a number both sides agreed to in advance.

• SLI: the measurement. successful_requests / total_requests.
• SLO: the target. 99.9% over a rolling 30 days.
• Error budget: 100% − SLO. At 99.9%, that's about 43 minutes of failure per month.

Set the SLO slightly stricter than the contractual SLA, so the internal target trips before the external commitment does. And define these before an incident. An SLO negotiated during an outage is just a description of what happened.

Alert on symptoms, not causes

Alerts should map to something a user would notice. High CPU is not an incident; requests failing is. Alerting on causes produces two failure modes at once: pages for conditions that harm nobody, and silence during novel failures that don't happen to match any of your cause-based rules.

• Page on user-visible symptoms: error rate, latency past a threshold, throughput falling off a cliff.
• Every page should have a runbook and a plausible action. If there's nothing to do, it's a dashboard, not a page.
• Treat a noisy alert as a defect with a ticket, because alert fatigue is how real pages get missed.

Logging is the prerequisite for automating incident response

This is the part I'd emphasize to anyone planning AI-assisted operations. Every proposal to have an agent triage incidents assumes the machine can read what happened. If the logs are unstructured prose with no correlation ID and no version, the agent is in exactly the same position as a human, reading line by line and guessing, except with less context and more confidence.

Anti-patterns

• Logging everything at maximum verbosity. Ingest costs scale, and signal doesn't.
• Dashboards nobody opens. A dashboard built during an incident and never revisited is a maintenance liability.
• Alerts with no runbook. Guarantees the response depends on who happens to be on call.
• Different log formats per service. Every cross-service query becomes bespoke, which means it doesn't get written.
• Monitoring only what's easy to measure. CPU is easy; whether checkouts are succeeding is what matters.

With Claude

Correlating a spike

Give it the window, the symptom, and the deploy timeline together. The answer is usually in the overlap, and that's exactly the kind of cross-referencing that's tedious by hand.

Error rate on payments-api went from 0.1% to 4% between 14:02 and 14:19.

Attached: error logs for that window, and the deploy log for the day.

1. Group the errors by `event` and `upstream`. Which dominates?
2. Does the start time line up with any deploy? Name the version.
3. Give me three candidate causes ranked by how well they fit the
   evidence, and for each, the one query that would rule it out.

Flag explicitly if the errors started BEFORE the nearest deploy.

That last line guards against the most common analytical error in incident response: anchoring on the nearest deploy and reasoning backwards from it.

Turning an incident into detection

The highest-leverage post-incident question isn't what broke, it's why nothing caught it. Ask for the alert definition that would have fired, including the threshold and the rationale for that threshold, and you get a concrete artifact instead of an action item that says "improve monitoring."

Auditing what a service actually logs

Before trusting a service in an automated triage path, it's worth asking whether it emits enough to be diagnosed at all.

Audit the logging in this service against these requirements:
  - structured JSON on every path, including error handlers
  - request_id present and propagated to downstream calls
  - build version on every line
  - no secrets, tokens, or PII (check error paths especially,
    that's where objects get dumped wholesale)
  - stable event names rather than formatted sentences

Output a table: requirement, pass/fail, file:line evidence.
Then: could an on-call engineer diagnose a 500 from these logs alone?