# Reliability and change management

> Ship as many changes as you can, as cheaply as you can, as long as reliability stays at or above the SLA.

Source: https://christwentyfour.com/strategy/reliability

## The job

Ship as many changes as you can, as cheaply as you can, as long as reliability stays at or above the SLA.

max Changes, min Cost s.t. Reliability ≥ ReliabilitySLA\htmlClass{sym}{\text{max}}\ \text{Changes},\ \htmlClass{sym}{\text{min}}\ \text{Cost s.t. Reliability} \text{ ≥ } \text{Reliability}_{\mathrm{SLA}}

Ship as many changes as you can, as cheaply as you can, as long as reliability stays at or above the SLA.

[SLOs and error budgets](https://christwentyfour.com/sre/slos-and-error-budgets)

## Spend your error budget on changes

Your SLO is a budget of bad minutes. Every change spends a few of them. Recover faster and each change costs less, so you can ship more.

### At 99.9%, you get 43 bad minutes a month

30-min fix

14 changes

5-min fix

86 changes

1 in 10 changes breaks something. A fix takes 30 minutes. Each change costs 3 minutes on average, so you can ship about 14 changes a month.

Same failure rate, but you roll back in 5 minutes. Each change costs 30 seconds, so you can ship about 86 a month.

If the rollback lands before the user tries again, the retry works and they never see the error.

changes=budget÷(failure rate×fix time)\text{changes} = \text{budget} \div (\text{failure rate} \times \text{fix time})

43.2 ÷ (0.1 × 30) ≈ 14 · 43.2 ÷ (0.1 × 5) ≈ 86

Rule:Make changes cheap to get wrong. Halve the fix time and you double what you can ship.

## Impact: make a bad change hurt fewer people for less time

Downtime people feel is people times minutes. You can cut both without shipping any less.

I=f(T,S,U)\htmlClass{sym}{I} = f(T, S, U)

Impact I is a function of duration T, severity S, and how many users U felt it.

[Incident triage](https://christwentyfour.com/sre/incident-triage)

max Changes, min Cost if R ≥ RSLA, else min I\htmlClass{sym}{\text{max}}\ \text{Changes},\ \htmlClass{sym}{\text{min}}\ \text{Cost if }R \text{ ≥ } R_{\mathrm{SLA}}\text{, else min }I

While reliability meets the SLA, maximize change and minimize cost. Once it misses, minimize impact I.

[SLOs and error budgets](https://christwentyfour.com/sre/slos-and-error-budgets)

10,000 users. Each small change has a 2% chance of breaking something.

### Why smaller changes break less

Every piece you bundle into a release is another chance to break something. At 2% a piece, a one-change release breaks 2 times in 100. Bundle 10 and it breaks 18 times in 100. And when it does break, you have 10 suspects instead of 1.

1 change per release

2 of 100 break

5 changes per release

10 of 100 break

10 changes per release

18 of 100 break

Each dot is one release. Gold ones broke something.

chance a release breaks = 1 − (0.98 × 0.98 × … once per change) = 1 − 0.98changes

#### One big weekly release

10 changes bundled, 18% chance it's bad

everyone · 30 min · 55,000 user-min

#### Ship them one at a time

2% each, easy to spot and roll back

everyone · 5 min · 10,000 user-min

#### Send each to 5% first

only the canary sees it

500 users · 5 min · 500 user-min

#### Retry lands on the old version

one failed click, then it works

500 users · 6 sec · 10 user-min

Each dot is 100 users. The bar is how long they notice, out of 30 minutes.

Same 10 changes, same code. The downtime people feel drops from about 55,000 user-minutes to 10.

user-minutes = changes × chance it's bad × users exposed × minutes they notice  
10 × 0.02 × 500 × 0.1 = 10

Rule:Ship small, ship to a few first, and make the retry land on the old version.

## Eliminate toil, and check that you did

Toil is repetitive manual work that leaves nothing behind. Google's bar is under 50% of team time. The trend matters more than the threshold, because toil grows with the number of services.

Priority=Changes/Cost\htmlClass{sym}{\text{Priority}} = \text{Changes}/\text{Cost}

Priority is changes divided by cost: prefer work that delivers more change per unit of cost.

[Automation blueprint](https://christwentyfour.com/sre/automation-blueprint)

**Do:**

- Automate every routine deploy through a pipeline.
- Self-heal recurring alerts. Page a human only when the fix fails.
- Make security scans pipeline stages, not periodic projects.

**Don't:**

- Fix toil by hiring. Cost then grows linearly with the problem.
- Count hours saved without checking that deploys went up.

## Catch defects in design, where they cost a conversation

A defect caught in design costs a conversation. Caught at deploy, it costs a rollback and someone's sleep. Infrastructure people belong in requirement and design reviews.

**Do:**

- Put tests in the acceptance criteria.
- Version, lint, and test config like code.
- Run static analysis and dependency scans in the PR pipeline.

Failure in design

Failure=f(Code, Config, Usage, Infra)\htmlClass{sym}{\text{Failure}} = f(\text{Code},\ \text{Config},\ \text{Usage},\ \text{Infra})

A failure is a function of code, configuration, usage, and infrastructure.

[Incident triage](https://christwentyfour.com/sre/incident-triage)

min Failure s.t. Performance ≥ Ptarget, Scalability ≥ Starget\htmlClass{sym}{\text{min}}\ \text{Failure s.t. Performance} \text{ ≥ } P_{\mathrm{target}}\text{, Scalability} \text{ ≥ } S_{\mathrm{target}}

Minimize failure without going below the performance and scalability targets.

Caught early

DDRearly=Dearly/Dtotal\htmlClass{sym}{DDR_{\mathrm{early}}} = D_{\mathrm{early}}/D_{\mathrm{total}}

DDRearlyDDR_{\mathrm{early}} is the share of all defects DtotalD_{\mathrm{total}} that were caught early, DearlyD_{\mathrm{early}}.

Idev=Hops/Htotal\htmlClass{sym}{I_{\mathrm{dev}}} = H_{\mathrm{ops}}/H_{\mathrm{total}}

IdevI_{\mathrm{dev}} is the share of developers’ total hours HtotalH_{\mathrm{total}} spent on operations, HopsH_{\mathrm{ops}}.

### Don't lean on end-to-end tests

End-to-end tests feel like the safest test. At scale they're the least useful. Every test that has to pass is another chance to fail for a reason that has nothing to do with your code: a timeout, a stale config, a service someone else just deployed. Grow the system and the chance the whole suite passes goes to zero, even when everything is healthy. Then people re-run until it's green, and the suite stops telling you anything.

Say every test is 99% reliable on a healthy system.

10 tests

90 of 100 runs pass

50 tests

61 of 100 runs pass

100 tests

37 of 100 runs pass

300 tests

5 of 100 runs pass

Each dot is one run of the whole suite on a healthy system. Gold ones passed.

Most of those failures are config and environment, not bugs. And when one fails, it doesn't tell you which service broke. A contract test checks one service against its direct neighbors. It grows with the number of connections, not the size of the whole system, and when it fails it names the culprit.

chance the suite passes = 0.99tests  ·  0.99100 ≈ 37%

Rule:Keep a handful of end-to-end tests for the money path. Everything else is unit tests, contract tests and a canary.

Dependencies

∀ Si ∈ {S1,S2,…,Sn}, Din(Si)+Dout(Si) ≥ 1\text{∀ } S_i \text{ ∈ } \{S_1, S_2, \text{…}, S_n\},\ D_{\mathrm{in}}(S_i) + D_{\mathrm{out}}(S_i) \text{ ≥ } 1

Every service SiS_i has at least one incoming dependency DinD_{\mathrm{in}} or outgoing dependency DoutD_{\mathrm{out}}.

End-to-end cost

TE2E=∏i=1nki⋅vi ≈ (kmax⁡⋅vmax⁡)n\htmlClass{sym}{T_{\mathrm{E2E}}} = {\htmlClass{op}{∏}}_{i=1}^{n} k_i \htmlClass{op}{·} v_i \text{ ≈ } (k_{\max} \htmlClass{op}{·} v_{\max})^{n}

End-to-end scenarios TE2ET_{\mathrm{E2E}} are the product of each service’s states kik_i and versions viv_i, about (kmax⁡⋅vmax⁡)n(k_{\max} \htmlClass{op}{·} v_{\max})^{n}.

Why the suite flakes

pfailure=1−(1−pbase)n\htmlClass{sym}{p_{\mathrm{failure}}} = 1 - (1 - p_{\mathrm{base}})^{n}

The chance a test run fails, pfailurep_{\mathrm{failure}}, compounds from a per-service base rate pbasep_{\mathrm{base}} as you add n services.

pfalse-positive=c⋅n⋅T\htmlClass{sym}{p_{\mathrm{false\text{-}positive}}} = c \htmlClass{op}{·} n \htmlClass{op}{·} T

False-positive chance scales with a constant c, the number of services n, and the number of scenarios T.

Pall-pass=(1−pfalse-positive)T → 0\htmlClass{sym}{P_{\mathrm{all\text{-}pass}}} = (1 - p_{\mathrm{false\text{-}positive}})^{T} \text{ → } 0

The chance every test is a true pass, Pall-passP_{\mathrm{all\text{-}pass}}, goes to zero as the number of scenarios T grows.

Contract tests

TContract=∑i=1ndi\htmlClass{sym}{T_{\mathrm{Contract}}} = {\htmlClass{op}{∑}}_{i=1}^{n} d_i

The number of contract tests TContractT_{\mathrm{Contract}} is the sum of each service’s direct dependencies did_i.

TContract=n⋅(n−1)\htmlClass{sym}{T_{\mathrm{Contract}}} = n \htmlClass{op}{·} (n - 1)

In a fully connected system of n services, TContractT_{\mathrm{Contract}} is n times (n minus one).

pE2E-fail=1−∏i=1n(1−pi)\htmlClass{sym}{p_{\mathrm{E2E\text{-}fail}}} = 1 - {\htmlClass{op}{∏}}_{i=1}^{n} (1 - p_i)

The chance an end-to-end suite fails is one minus the product of each service holding, with chance pip_i of failing.

pContract-fail=1−∏(i,j)(1−pij)\htmlClass{sym}{p_{\mathrm{Contract\text{-}fail}}} = 1 - {\htmlClass{op}{∏}}_{(i,j)} (1 - p_{ij})

The chance a contract suite fails depends on the pairwise checks, each with chance pijp_{ij}.

## Write down what should be true

Steps don't tell you if the work is done. Checks do. Red is the work left. Green is done.

#### Steps

Done? No way to tell.

?Provision a server.

?Copy the site over.

?Point DNS at it.

#### Desired state

1 of 3 true. The red lines are the work left.

✗Exactly one web server is running.

✓example.com points at that server.

✗http redirects to https.

Rule:If you can't write the check, you don't know what done means yet.

## Attention moves from how to what

Every new tool takes over more of the steps. What's left is saying what should be true, and checking that it is.

A rough picture, not a measurement.

an>**→**All green: done
