Reliability and change management

Ship as many changes as you can, as cheaply as you can, as long as reliability stays at or above the SLA.

8 min read

On this page

The job

Ship as many changes as you can, as cheaply as you can, as long as reliability stays at or above the SLA.

max Changes, min Cost s.t. Reliability ≥ ReliabilitySLA\htmlClass{sym}{\text{max}}\ \text{Changes},\ \htmlClass{sym}{\text{min}}\ \text{Cost s.t. Reliability} \text{ ≥ } \text{Reliability}_{\mathrm{SLA}}

Ship as many changes as you can, as cheaply as you can, as long as reliability stays at or above the SLA.

Spend your error budget on changes

Your SLO is a budget of bad minutes. Every change spends a few of them. Recover faster and each change costs less, so you can ship more.

At 99.9%, you get 43 bad minutes a month

30-min fix
14 changes
5-min fix
86 changes

1 in 10 changes breaks something. A fix takes 30 minutes. Each change costs 3 minutes on average, so you can ship about 14 changes a month.

Same failure rate, but you roll back in 5 minutes. Each change costs 30 seconds, so you can ship about 86 a month.

If the rollback lands before the user tries again, the retry works and they never see the error.

changes=budget÷(failure rate×fix time)\text{changes} = \text{budget} \div (\text{failure rate} \times \text{fix time})

43.2 ÷ (0.1 × 30) ≈ 14 · 43.2 ÷ (0.1 × 5) ≈ 86

Rule:Make changes cheap to get wrong. Halve the fix time and you double what you can ship.

Impact: make a bad change hurt fewer people for less time

Downtime people feel is people times minutes. You can cut both without shipping any less.

I=f(T,S,U)\htmlClass{sym}{I} = f(T, S, U)

Impact I is a function of duration T, severity S, and how many users U felt it.

max Changes, min Cost if R ≥ RSLA, else min I\htmlClass{sym}{\text{max}}\ \text{Changes},\ \htmlClass{sym}{\text{min}}\ \text{Cost if }R \text{ ≥ } R_{\mathrm{SLA}}\text{, else min }I

While reliability meets the SLA, maximize change and minimize cost. Once it misses, minimize impact I.

10,000 users. Each small change has a 2% chance of breaking something.

Why smaller changes break less

Every piece you bundle into a release is another chance to break something. At 2% a piece, a one-change release breaks 2 times in 100. Bundle 10 and it breaks 18 times in 100. And when it does break, you have 10 suspects instead of 1.

1 change per release
2 of 100 break
5 changes per release
10 of 100 break
10 changes per release
18 of 100 break

Each dot is one release. Gold ones broke something.

chance a release breaks = 1 − (0.98 × 0.98 × … once per change) = 1 − 0.98changes

One big weekly release

10 changes bundled, 18% chance it's bad
everyone · 30 min · 55,000 user-min

Ship them one at a time

2% each, easy to spot and roll back
everyone · 5 min · 10,000 user-min

Send each to 5% first

only the canary sees it
500 users · 5 min · 500 user-min

Retry lands on the old version

one failed click, then it works
500 users · 6 sec · 10 user-min

Each dot is 100 users. The bar is how long they notice, out of 30 minutes.

Same 10 changes, same code. The downtime people feel drops from about 55,000 user-minutes to 10.

user-minutes = changes × chance it's bad × users exposed × minutes they notice
10 × 0.02 × 500 × 0.1 = 10

Rule:Ship small, ship to a few first, and make the retry land on the old version.

Eliminate toil, and check that you did

Toil is repetitive manual work that leaves nothing behind. Google's bar is under 50% of team time. The trend matters more than the threshold, because toil grows with the number of services.

Priority=Changes/Cost\htmlClass{sym}{\text{Priority}} = \text{Changes}/\text{Cost}

Priority is changes divided by cost: prefer work that delivers more change per unit of cost.

  • Automate every routine deploy through a pipeline.
  • Self-heal recurring alerts. Page a human only when the fix fails.
  • Make security scans pipeline stages, not periodic projects.
  • Fix toil by hiring. Cost then grows linearly with the problem.
  • Count hours saved without checking that deploys went up.

Catch defects in design, where they cost a conversation

A defect caught in design costs a conversation. Caught at deploy, it costs a rollback and someone's sleep. Infrastructure people belong in requirement and design reviews.

  • Put tests in the acceptance criteria.
  • Version, lint, and test config like code.
  • Run static analysis and dependency scans in the PR pipeline.

Failure in design

Failure=f(Code, Config, Usage, Infra)\htmlClass{sym}{\text{Failure}} = f(\text{Code},\ \text{Config},\ \text{Usage},\ \text{Infra})

A failure is a function of code, configuration, usage, and infrastructure.

min Failure s.t. Performance ≥ Ptarget, Scalability ≥ Starget\htmlClass{sym}{\text{min}}\ \text{Failure s.t. Performance} \text{ ≥ } P_{\mathrm{target}}\text{, Scalability} \text{ ≥ } S_{\mathrm{target}}

Minimize failure without going below the performance and scalability targets.

Caught early

DDRearly=Dearly/Dtotal\htmlClass{sym}{DDR_{\mathrm{early}}} = D_{\mathrm{early}}/D_{\mathrm{total}}

DDRearlyDDR_{\mathrm{early}} is the share of all defects DtotalD_{\mathrm{total}} that were caught early, DearlyD_{\mathrm{early}}.

Idev=Hops/Htotal\htmlClass{sym}{I_{\mathrm{dev}}} = H_{\mathrm{ops}}/H_{\mathrm{total}}

IdevI_{\mathrm{dev}} is the share of developers’ total hours HtotalH_{\mathrm{total}} spent on operations, HopsH_{\mathrm{ops}}.

Don't lean on end-to-end tests

End-to-end tests feel like the safest test. At scale they're the least useful. Every test that has to pass is another chance to fail for a reason that has nothing to do with your code: a timeout, a stale config, a service someone else just deployed. Grow the system and the chance the whole suite passes goes to zero, even when everything is healthy. Then people re-run until it's green, and the suite stops telling you anything.

Say every test is 99% reliable on a healthy system.

10 tests
90 of 100 runs pass
50 tests
61 of 100 runs pass
100 tests
37 of 100 runs pass
300 tests
5 of 100 runs pass

Each dot is one run of the whole suite on a healthy system. Gold ones passed.

Most of those failures are config and environment, not bugs. And when one fails, it doesn't tell you which service broke. A contract test checks one service against its direct neighbors. It grows with the number of connections, not the size of the whole system, and when it fails it names the culprit.

chance the suite passes = 0.99tests  ·  0.99100 ≈ 37%

Rule:Keep a handful of end-to-end tests for the money path. Everything else is unit tests, contract tests and a canary.

Dependencies

∀ Si ∈ {S1,S2,…,Sn}, Din(Si)+Dout(Si) ≥ 1\text{∀ } S_i \text{ ∈ } \{S_1, S_2, \text{…}, S_n\},\ D_{\mathrm{in}}(S_i) + D_{\mathrm{out}}(S_i) \text{ ≥ } 1

Every service SiS_i has at least one incoming dependency DinD_{\mathrm{in}} or outgoing dependency DoutD_{\mathrm{out}}.

End-to-end cost

TE2E=∏i=1nki⋅vi ≈ (kmax⁡⋅vmax⁡)n\htmlClass{sym}{T_{\mathrm{E2E}}} = {\htmlClass{op}{∏}}_{i=1}^{n} k_i \htmlClass{op}{·} v_i \text{ ≈ } (k_{\max} \htmlClass{op}{·} v_{\max})^{n}

End-to-end scenarios TE2ET_{\mathrm{E2E}} are the product of each service’s states kik_i and versions viv_i, about (kmax⁡⋅vmax⁡)n(k_{\max} \htmlClass{op}{·} v_{\max})^{n}.

Why the suite flakes

pfailure=1−(1−pbase)n\htmlClass{sym}{p_{\mathrm{failure}}} = 1 - (1 - p_{\mathrm{base}})^{n}

The chance a test run fails, pfailurep_{\mathrm{failure}}, compounds from a per-service base rate pbasep_{\mathrm{base}} as you add n services.

pfalse-positive=c⋅n⋅T\htmlClass{sym}{p_{\mathrm{false\text{-}positive}}} = c \htmlClass{op}{·} n \htmlClass{op}{·} T

False-positive chance scales with a constant c, the number of services n, and the number of scenarios T.

Pall-pass=(1−pfalse-positive)T → 0\htmlClass{sym}{P_{\mathrm{all\text{-}pass}}} = (1 - p_{\mathrm{false\text{-}positive}})^{T} \text{ → } 0

The chance every test is a true pass, Pall-passP_{\mathrm{all\text{-}pass}}, goes to zero as the number of scenarios T grows.

Contract tests

TContract=∑i=1ndi\htmlClass{sym}{T_{\mathrm{Contract}}} = {\htmlClass{op}{∑}}_{i=1}^{n} d_i

The number of contract tests TContractT_{\mathrm{Contract}} is the sum of each service’s direct dependencies did_i.

TContract=n⋅(n−1)\htmlClass{sym}{T_{\mathrm{Contract}}} = n \htmlClass{op}{·} (n - 1)

In a fully connected system of n services, TContractT_{\mathrm{Contract}} is n times (n minus one).

pE2E-fail=1−∏i=1n(1−pi)\htmlClass{sym}{p_{\mathrm{E2E\text{-}fail}}} = 1 - {\htmlClass{op}{∏}}_{i=1}^{n} (1 - p_i)

The chance an end-to-end suite fails is one minus the product of each service holding, with chance pip_i of failing.

pContract-fail=1−∏(i,j)(1−pij)\htmlClass{sym}{p_{\mathrm{Contract\text{-}fail}}} = 1 - {\htmlClass{op}{∏}}_{(i,j)} (1 - p_{ij})

The chance a contract suite fails depends on the pairwise checks, each with chance pijp_{ij}.

Write down what should be true

Steps don't tell you if the work is done. Checks do. Red is the work left. Green is done.

Steps

Done? No way to tell.
?Provision a server.
?Copy the site over.
?Point DNS at it.

Desired state

1 of 3 true. The red lines are the work left.
✗Exactly one web server is running.
✓example.com points at that server.
✗http redirects to https.

Rule:If you can't write the check, you don't know what done means yet.

Attention moves from how to what

Every new tool takes over more of the steps. What's left is saying what should be true, and checking that it is.

time → How to get food How to get shelter How to do business What gets produced

A rough picture, not a measurement.

an>→All green: done

Rule:If you can't write the check, you don't know what done means yet.

Deploy is not release

Deploying puts code on a server. Releasing makes it visible to users. Separate them, and you can ship often and still control exposure.

Ctotal=Cdeployed+Ctoggled\htmlClass{sym}{C_{\mathrm{total}}} = C_{\mathrm{deployed}} + C_{\mathrm{toggled}}

CtotalC_{\mathrm{total}} is deployed changes plus changes still held back by a toggle.

Sreversal=1/Treversal\htmlClass{sym}{S_{\mathrm{reversal}}} = 1/T_{\mathrm{reversal}}

Reversal speed SreversalS_{\mathrm{reversal}} is one over the time TreversalT_{\mathrm{reversal}} it takes to reverse a deployment.

Every deploy has to be reversible in 5 minutes. A schema migration usually isn't, so it needs a human to approve it. The same 5-minute line is the limit for what an agent can do unsupervised.

Agents, with guardrails that assume they will be wrong

I use agents for work too messy to script. An agent that causes an incident costs more than no agent, because it burns the trust that pays for the next automation.

RiskMitigation
Destructive actionHuman approval for anything not reversible in 5 minutes.
Confidently wrong outputAgent output is a draft. A human owns the decision.
Sensitive data exposureMinimum credentials. Log every tool call.
Silent decayTrack the agent's success rate as a metric.

What elite looks like

DORA, in Accelerate, is the closest this field has to evidence.

Dauto=∑iDi\htmlClass{sym}{D_{\mathrm{auto}}} = {\htmlClass{op}{∑}}_{i} D_i

Automated deployment volume DautoD_{\mathrm{auto}} is the sum of each project’s deployment count DiD_i.

FD=Dauto/T\htmlClass{sym}{F_D} = D_{\mathrm{auto}}/T

Deployment frequency FDF_D is automated deployments DautoD_{\mathrm{auto}} in a time period T.

MetricLowHighElite
Deployment frequency< once/monthdaily–weeklyon demand, many/day
Lead time for changes1–6 months1 day–1 week< 1 hour
Change failure rate46–60%0–15%0–15%
Time to restore> 6 months< 1 day< 1 hour

Elite teams deploy more often and fail less. Slowing down for safety goes against the data.

Culture predicts delivery better than tools do

Westrum's model sorts organizations by how they handle information and failure:

TypeInformationResponse to failure
PathologicalHoarded as powerFind someone to blame
BureaucraticIgnored if it crosses a boundaryAssign fault
GenerativeFlows to whoever needs itAsk: what allowed this?

Generative organizations have about 2.2 times the odds of high delivery performance.

I run blameless postmortems. Every action item has to change the system, not a person. If every item is "be more careful," nothing changed.

How I debug

People skip steps under pressure, so I work through this list in order:

  1. State the problem in one concrete sentence, and what success looks like.
  2. Name the constraints: time, blast radius, what must not change.
  3. Gather evidence only, and list three to seven plausible causes.
  4. Rule out the simplest causes first, one at a time.
  5. Make one change and measure it. If it fails, update the facts. Do not stack changes.
  6. Confirm the fix from more than one angle, including recreating the original conditions.
  7. Write down the cause, the fix, and one guardrail against recurrence.

You're done when you understand the cause.

A symptom going away doesn't mean it's fixed.

This is the first engineering strategy document. My engineering strategy has the takeaways and what you can produce. DevOps and agentic tool use are coming.