I'm Chris, an SRE in Rexburg.

I keep software up for the people who depend on it.
I ship small changes to a few users first.

Hire meEmail meAGENTS.mdRead my posts

Chris Roos with his two kids
Chris Roos with his two kids

Chris Roos

Site Reliability Engineer

SRE in Rexburg, Idaho, and dad of two. I use AI agents for the first hour of an incident and the routine work around it.

For your agent

> Point your agent at https://christwentyfour.com/llms.txt

Your agent can have my rules: paste AGENTS.md into its instructions. To load every page at once, fetch llms-full.txt.

Start here

  1. StrategyDocuments for incidents, deploys, access, and cost.
  2. The mathError budgets, small releases, canaries, and test suites, worked out with numbers.
  3. AutomationThe blueprint, then herdr, Jira, Claude Code, and the human approval step.
  4. PostsAlerts that earn the page, restores you've never run, staging that lies.
  5. RunbooksHow I use Terraform, containers, pipelines, observability, and Python, with example config.

What I optimize for, and what I give up to get it

Good strategy tells you what to give up. These are the trade-offs I use when an incident or a deadline forces a call.

10 min read

On this page

What a strategy is

A strategy picks the trade-offs. It isn't a roadmap or a list of projects. A good one lets any engineer make the right call without escalating.

It has three parts (Rumelt, Good Strategy/Bad Strategy):

  1. Diagnosis: an honest read of where you are, including constraints and risks.
  2. Guiding principle: the trade-off you pick, e.g. "reliability over feature count when we miss the SLA."
  3. Coherent actions: a few moves that put the principle to work and reinforce each other.

A strategy is a set of values

In chess, a bad strategy values pawns more than queens. A good one values queens more than pawns. Neither tells you which move to make. With the right values, you can work out the move yourself, in any position.

Engineering works the same way. Say "a deploy is worth more than a dollar of cost, as long as you stay above the SLO" and the next decision follows from it.

Good strategy vs bad strategy

GoodBad
Names the real problem in one sentenceRestates the goal ("be more reliable")
Picks a trade-off and says no to thingsTries to do everything
A few actions that fit togetherA pile of unrelated projects and OKRs
Plain wordsJargon
Gets revisited when the situation changesGets written once and filed

Strategy vs tactics

StrategyTactics
QuestionWhere do we spend effort?How do we do this step?
HorizonQuartersMinutes to days
Aided byModels and values: equations, charts, what each thing is worthDiscipline, training and docs: runbooks, checklists, practice
Fails whenThe model is wrongA step gets skipped
ExampleShip as many changes as the SLO allowsRoll back a bad deploy in under 5 minutes

Rule:if you're deciding, reach for models and values. If you're doing, reach for discipline and a runbook.

Why strategy matters more now

A good strategy with nobody who can execute it is worth nothing.

AI made tactics cheap. An agent can write the code, run the runbook, and draft the fix. When doing is cheap, choosing what to do is the expensive part.

Spend your error budget on changes

Your SLO is a budget of bad minutes. Every change spends a few of them. Recover faster and each change costs less, so you can ship more.

At 99.9%, you get 43 bad minutes a month

30-min fix
14 changes
5-min fix
86 changes

1 in 10 changes breaks something. A fix takes 30 minutes. Each change costs 3 minutes on average, so you can ship about 14 changes a month.

Same failure rate, but you roll back in 5 minutes. Each change costs 30 seconds, so you can ship about 86 a month.

If the rollback lands before the user tries again, the retry works and they never see the error.

\(\text{changes} = \text{budget} \div (\text{failure rate} \times \text{fix time})\)

43.2 ÷ (0.1 × 30) ≈ 14 · 43.2 ÷ (0.1 × 5) ≈ 86

Rule:Make changes cheap to get wrong. Halve the fix time and you double what you can ship.

Impact: make a bad change hurt fewer people for less time

Downtime people feel is people times minutes. You can cut both without shipping any less.

10,000 users. Each small change has a 2% chance of breaking something.

Why smaller changes break less

Every piece you bundle into a release is another chance to break something. At 2% a piece, a one-change release breaks 2 times in 100. Bundle 10 and it breaks 18 times in 100. And when it does break, you have 10 suspects instead of 1.

1 change per release
2 of 100 break
5 changes per release
10 of 100 break
10 changes per release
18 of 100 break

Each dot is one release. Gold ones broke something.

chance a release breaks = 1 − (0.98 × 0.98 × … once per change) = 1 − 0.98changes

One big weekly release

10 changes bundled, 18% chance it's bad
everyone · 30 min · 55,000 user-min

Ship them one at a time

2% each, easy to spot and roll back
everyone · 5 min · 10,000 user-min

Send each to 5% first

only the canary sees it
500 users · 5 min · 500 user-min

Retry lands on the old version

one failed click, then it works
500 users · 6 sec · 10 user-min

Each dot is 100 users. The bar is how long they notice, out of 30 minutes.

Same 10 changes, same code. The downtime people feel drops from about 55,000 user-minutes to 10.

user-minutes = changes × chance it's bad × users exposed × minutes they notice
10 × 0.02 × 500 × 0.1 = 10

Rule:Ship small, ship to a few first, and make the retry land on the old version.

Eliminate toil, and check that you did

Toil is repetitive manual work that leaves nothing behind. It inflates cost without adding change. Google's benchmark is under 50% of team time. The trend matters more than the threshold, because toil grows with the number of services.

  • Automate every routine deploy through a pipeline.
  • Self-heal recurring alerts. Page a human only when the fix fails.
  • Make security scans pipeline stages, not periodic projects.
  • Fix toil by hiring. Cost then grows linearly with the problem.
  • Count hours saved without checking that deploys went up.

Catch defects in design, where they cost a conversation

A defect caught in design costs a conversation. Caught at deploy, it costs a rollback and someone's sleep. Infrastructure people belong in requirement and design reviews.

  • Put tests in the acceptance criteria.
  • Version, lint, and test config like code.
  • Run static analysis and dependency scans in the PR pipeline.

Don't lean on end-to-end tests

End-to-end tests feel like the safest test. At scale they're the least useful. Every test that has to pass is another chance to fail for a reason that has nothing to do with your code: a timeout, a stale config, a service someone else just deployed. Grow the system and the chance the whole suite passes goes to zero, even when everything is healthy. Then people re-run until it's green, and the suite stops telling you anything.

Say every test is 99% reliable on a healthy system.

10 tests
90 of 100 runs pass
50 tests
61 of 100 runs pass
100 tests
37 of 100 runs pass
300 tests
5 of 100 runs pass

Each dot is one run of the whole suite on a healthy system. Gold ones passed.

Most of those failures are config and environment, not bugs. And when one fails, it doesn't tell you which service broke. A contract test checks one service against its direct neighbors. It grows with the number of connections, not the size of the whole system, and when it fails it names the culprit.

chance the suite passes = 0.99tests  ·  0.99100 ≈ 37%

Rule:Keep a handful of end-to-end tests for the money path. Everything else is unit tests, contract tests and a canary.

Write down what should be true

Steps don't tell you if the work is done. Checks do. Red is the work left. Green is done.

Steps

Done? No way to tell.
?Provision a server.
?Copy the site over.
?Point DNS at it.

Desired state

1 of 3 true. The red lines are the work left.
✗Exactly one web server is running.
✓example.com points at that server.
✗http redirects to https.

Rule:If you can't write the check, you don't know what done means yet.

Attention moves from how to what

Every new tool takes over more of the steps. What's left is saying what should be true, and checking that it is.

time → How to get food How to get shelter How to do business What gets produced

A rough picture, not a measurement.

an>→All green: done

Rule:If you can't write the check, you don't know what done means yet.

Deploy is not release

Deploying puts code on a server. Releasing makes it visible to users. Separate them, and you can ship often and still control exposure.

Code in production = deployed code + toggled-on code

Once those are separate, a bad change can be turned off in seconds.

Every deploy has to be reversible in 5 minutes. A schema migration usually isn't, so it needs a human to approve it. The same 5-minute line is the limit for what an agent can do unsupervised.

Agents, with guardrails that assume they will be wrong

I use agents for work too messy to script. An agent that causes an incident costs more than no agent, because it burns the trust that pays for the next automation.

RiskMitigation
Destructive actionHuman approval for anything not reversible in 5 minutes.
Confidently wrong outputAgent output is a draft. A human owns the decision.
Sensitive data exposureMinimum credentials. Log every tool call.
Silent decayTrack the agent's success rate as a metric.

What elite looks like

The DORA research behind Accelerate is the closest thing this field has to evidence.

MetricLowHighElite
Deployment frequency< once/monthdaily–weeklyon demand, many/day
Lead time for changes1–6 months1 day–1 week< 1 hour
Change failure rate46–60%0–15%0–15%
Time to restore> 6 months< 1 day< 1 hour

Elite teams deploy more often and fail less. Slowing down for safety goes against the data.

Culture predicts delivery better than tools do

Westrum's model sorts organizations by how they handle information and failure:

TypeInformationResponse to failure
PathologicalHoarded as powerFind someone to blame
BureaucraticIgnored if it crosses a boundaryAssign fault
GenerativeFlows to whoever needs itAsk: what allowed this?

Generative organizations have about 2.2 times the odds of high delivery performance.

I run blameless postmortems. Every action item has to change the system, not a person. If every item is "be more careful," nothing changed.

How I debug

People skip steps under pressure, so I work through this list in order:

  1. State the problem in one concrete sentence, and what success looks like.
  2. Name the constraints: time, blast radius, what must not change.
  3. Gather evidence only, and list three to seven plausible causes.
  4. Rule out the simplest causes first, one at a time.
  5. Make one change and measure it. If it fails, update the facts. Do not stack changes.
  6. Confirm the fix from more than one angle, including recreating the original conditions.
  7. Write down the cause, the fix, and one guardrail against recurrence.

You're done when you understand the cause.

A symptom going away doesn't mean it's fixed.

The runbooks have example config for each tool and the Claude workflow I use with it.