10,000 users. Each small change has a 2% chance of breaking something.
Why smaller changes break less
Every piece you bundle into a release is another chance to break something. At 2% a piece, a one-change release breaks 2 times in 100. Bundle 10 and it breaks 18 times in 100. And when it does break, you have 10 suspects instead of 1.
1 change per release
2 of 100 break
5 changes per release
10 of 100 break
10 changes per release
18 of 100 break
Each dot is one release. Gold ones broke something.
chance a release breaks = 1 − (0.98 × 0.98 × … once per change) = 1 − 0.98changes
One big weekly release
10 changes bundled, 18% chance it's bad
everyone · 30 min · 55,000 user-min
Ship them one at a time
2% each, easy to spot and roll back
everyone · 5 min · 10,000 user-min
Send each to 5% first
only the canary sees it
500 users · 5 min · 500 user-min
Retry lands on the old version
one failed click, then it works
500 users · 6 sec · 10 user-min
Each dot is 100 users. The bar is how long they notice, out of 30 minutes.
Same 10 changes, same code. The downtime people feel drops from about 55,000 user-minutes to 10.
Rule:Ship small, ship to a few first, and make the retry land on the old version.
Eliminate toil, and check that you did
Toil is repetitive manual work that leaves nothing behind. Google's bar is under 50% of team time. The trend matters more than the threshold, because toil grows with the number of services.
Priority=Changes/Cost
Priority is changes divided by cost: prefer work that delivers more change per unit of cost.
Self-heal recurring alerts. Page a human only when the fix fails.
Make security scans pipeline stages, not periodic projects.
Fix toil by hiring. Cost then grows linearly with the problem.
Count hours saved without checking that deploys went up.
Catch defects in design, where they cost a conversation
A defect caught in design costs a conversation. Caught at deploy, it costs a rollback and someone's sleep. Infrastructure people belong in requirement and design reviews.
Put tests in the acceptance criteria.
Version, lint, and test config like code.
Run static analysis and dependency scans in the PR pipeline.
Failure in design
Failure=f(Code,Config,Usage,Infra)
A failure is a function of code, configuration, usage, and infrastructure.
Minimize failure without going below the performance and scalability targets.
Caught early
DDRearly=Dearly/Dtotal
DDRearly is the share of all defects Dtotal that were caught early, Dearly.
Idev=Hops/Htotal
Idev is the share of developers’ total hours Htotal spent on operations, Hops.
Don't lean on end-to-end tests
End-to-end tests feel like the safest test. At scale they're the least useful. Every test that has to pass is another chance to fail for a reason that has nothing to do with your code: a timeout, a stale config, a service someone else just deployed. Grow the system and the chance the whole suite passes goes to zero, even when everything is healthy. Then people re-run until it's green, and the suite stops telling you anything.
Say every test is 99% reliable on a healthy system.
10 tests
90 of 100 runs pass
50 tests
61 of 100 runs pass
100 tests
37 of 100 runs pass
300 tests
5 of 100 runs pass
Each dot is one run of the whole suite on a healthy system. Gold ones passed.
Most of those failures are config and environment, not bugs. And when one fails, it doesn't tell you which service broke. A contract test checks one service against its direct neighbors. It grows with the number of connections, not the size of the whole system, and when it fails it names the culprit.
chance the suite passes = 0.99tests · 0.99100 ≈ 37%
Rule:Keep a handful of end-to-end tests for the money path. Everything else is unit tests, contract tests and a canary.
Dependencies
∀ Si ∈ {S1,S2,…,Sn},Din(Si)+Dout(Si) ≥ 1
Every service Si has at least one incoming dependency Din or outgoing dependency Dout.
End-to-end cost
TE2E=∏i=1nki⋅vi ≈ (kmax⋅vmax)n
End-to-end scenarios TE2E are the product of each service’s states ki and versions vi, about (kmax⋅vmax)n.
Why the suite flakes
pfailure=1−(1−pbase)n
The chance a test run fails, pfailure, compounds from a per-service base rate pbase as you add n services.
pfalse-positive=c⋅n⋅T
False-positive chance scales with a constant c, the number of services n, and the number of scenarios T.
Pall-pass=(1−pfalse-positive)T → 0
The chance every test is a true pass, Pall-pass, goes to zero as the number of scenarios T grows.
Contract tests
TContract=∑i=1ndi
The number of contract tests TContract is the sum of each service’s direct dependencies di.
TContract=n⋅(n−1)
In a fully connected system of n services, TContract is n times (n minus one).
pE2E-fail=1−∏i=1n(1−pi)
The chance an end-to-end suite fails is one minus the product of each service holding, with chance pi of failing.
pContract-fail=1−∏(i,j)(1−pij)
The chance a contract suite fails depends on the pairwise checks, each with chance pij.
Write down what should be true
Steps don't tell you if the work is done. Checks do. Red is the work left. Green is done.
Steps
Done? No way to tell.
?Provision a server.
?Copy the site over.
?Point DNS at it.
Desired state
1 of 3 true. The red lines are the work left.
✗Exactly one web server is running.
✓example.com points at that server.
✗http redirects to https.
Rule:If you can't write the check, you don't know what done means yet.
Attention moves from how to what
Every new tool takes over more of the steps. What's left is saying what should be true, and checking that it is.
A rough picture, not a measurement.
an>→All green: done
Rule:If you can't write the check, you don't know what done means yet.
Deploy is not release
Deploying puts code on a server. Releasing makes it visible to users. Separate them, and you can ship often and still control exposure.
Ctotal=Cdeployed+Ctoggled
Ctotal is deployed changes plus changes still held back by a toggle.
Every deploy has to be reversible in 5 minutes. A schema migration usually isn't, so it needs a human to approve it. The same 5-minute line is the limit for what an agent can do unsupervised.
Agents, with guardrails that assume they will be wrong
I use agents for work too messy to script. An agent that causes an incident costs more than no agent, because it burns the trust that pays for the next automation.
Risk
Mitigation
Destructive action
Human approval for anything not reversible in 5 minutes.
Confidently wrong output
Agent output is a draft. A human owns the decision.
Sensitive data exposure
Minimum credentials. Log every tool call.
Silent decay
Track the agent's success rate as a metric.
What elite looks like
DORA, in Accelerate, is the closest this field has to evidence.
Dauto=∑iDi
Automated deployment volume Dauto is the sum of each project’s deployment count Di.
Elite teams deploy more often and fail less. Slowing down for safety goes against the data.
Culture predicts delivery better than tools do
Westrum's model sorts organizations by how they handle information and failure:
Type
Information
Response to failure
Pathological
Hoarded as power
Find someone to blame
Bureaucratic
Ignored if it crosses a boundary
Assign fault
Generative
Flows to whoever needs it
Ask: what allowed this?
Generative organizations have about 2.2 times the odds of high delivery performance.
I run blameless postmortems. Every action item has to change the system, not a person. If every item is "be more careful," nothing changed.
How I debug
People skip steps under pressure, so I work through this list in order:
State the problem in one concrete sentence, and what success looks like.
Name the constraints: time, blast radius, what must not change.
Gather evidence only, and list three to seven plausible causes.
Rule out the simplest causes first, one at a time.
Make one change and measure it. If it fails, update the facts. Do not stack changes.
Confirm the fix from more than one angle, including recreating the original conditions.
Write down the cause, the fix, and one guardrail against recurrence.
You're done when you understand the cause.
A symptom going away doesn't mean it's fixed.
This is the first engineering strategy document. My engineering strategy has the takeaways and what you can produce. DevOps and agentic tool use are coming.