What a strategy actually is
Most documents labelled "engineering strategy" are roadmaps in disguise: a list of things a team intends to build, with dates. That's a plan. A strategy is the thing that tells you what to do when the plan meets reality.
I use Richard Rumelt's structure from Good Strategy / Bad Strategy, because it forces the uncomfortable part into the open:
- Diagnosis: an honest assessment of where you actually are, including the parts nobody wants to say out loud.
- Guiding principles: the stated trade-offs. "Reliability over feature count when we're missing SLAs" is a principle. "Be reliable and fast" is not.
- Coherent actions: initiatives that actually follow from the principles, rather than a wishlist that happens to sit underneath them.
The test for whether a principle is real is simple: does it tell you what to give up? If it doesn't, it's a slogan. Most strategy documents fail this test on every line.
The one ratio everything reduces to
An SRE function exists to maximize the rate of change a business can safely absorb, while keeping cost down and reliability above its commitments. I've found nearly every prioritization argument collapses into a single ratio:
Priority = Changes / Cost
Work that raises the numerator (more deployments, more shipped) or lowers the denominator (less toil, less manual overhead) gets done. Work that does neither gets deprioritized, regardless of how interesting it is.
Reliability isn't in the numerator, and that's deliberate. Reliability is a constraint, not a goal to maximize: keep it at or above the SLA and spend everything left over on throughput. Teams that treat reliability as something to maximize without limit end up shipping nothing and calling it engineering rigor.
During an incident the objective function changes entirely, and it's worth being explicit
about that too: minimize duration × severity × affected users. Nothing else
matters until that number stops growing.
Eliminate toil, and measure whether you actually did
Toil is repetitive, manual, automatable work that produces no lasting value. Resolving the same alert every week, deploying by hand, managing infrastructure through a console. It inflates the cost denominator without moving the numerator at all.
The Google SRE benchmark is that toil should stay under 50% of a team's time. I care less about the exact threshold than about the trend, because the failure mode is subtle: toil grows in proportion to the number of services, so a team that isn't actively reducing it is losing ground even when the absolute number looks flat.
- Automate every routine deployment through a pipeline.
- Build self-healing responses for recurring alert classes, and page a human only when the automated response fails.
- Treat security scanning and compliance checks as pipeline stages, not as periodic projects.
- Solve toil by hiring. It scales the cost denominator linearly with the problem.
- Track "hours saved" without tracking whether deployment frequency actually improved. Saved hours that don't turn into throughput are just slack.
Shift left, because late defects are exponentially expensive
A defect caught at design costs a conversation. The same defect caught at deployment costs a rollback, an incident review, and the trust of whoever was on call. Defects come from four places (code, configuration, usage, and infrastructure) and only the first is reliably caught by the tests most teams write.
The practical implication is that infrastructure people belong in requirements and design reviews, not just at the deployment end of the process. By the time a design reaches an SRE as a deploy ticket, every expensive decision has already been made.
- Require tests as acceptance criteria on the ticket, not as a follow-up ticket.
- Version, lint, and test configuration the same way you do application code. Config is where a surprising share of production defects actually originate.
- Put static analysis and dependency scanning in the PR pipeline, where the cost of a finding is measured in minutes.
Write requirements as desired state, not as instructions
This one is small, unglamorous, and has improved my working life more than almost anything else on this page. A ticket that describes actions can't be verified and can't be safely re-run. A ticket that describes desired state is testable, idempotent, and survives being handed to someone else, or to an agent.
Create a new branch named docker-poc.
Recreate the plugin in ECS.
A branch named `docker-poc` exists in the service repository.
A Dockerfile exists in that branch, running PHP 8.0, exposing 80 and 443.
An ECS service named `plugin-api` is running the containerized application.
A request to /health returns 200 OK.
Notice what happened: the second version tells you when you're done. The first one only tells you what to type. Every line of the state-based version can be checked by a script, which means it can also be checked by CI, which means it can also be handed to an agent without supervision. Vague requirements are the actual bottleneck in most automation efforts, not the tooling.
Contract tests over end-to-end tests
This is the most contested opinion here, so let me make the argument precisely.
End-to-end test coverage across microservices scales combinatorially with the number of services and versions. Every service you add multiplies the state space the suite has to hold stable. Past a certain size, the probability that all E2E tests pass on any given run approaches zero, regardless of whether the system is actually healthy.
At that point the suite has stopped being a signal. Teams respond by re-running until green, which is functionally the same as deleting the tests but with a much worse latency cost. Worse, most of those failures turn out to be configuration and environment drift rather than genuine defects, so the suite is expensive and pointed at the wrong failure class.
Contract testing scales linearly instead. Each service verifies only the interfaces it directly depends on, without needing the whole system live. The pyramid I aim for:
- Unit tests: component correctness, fast, run on every commit.
- Contract tests: interface compliance between a service and its direct dependencies. This is what replaces E2E for integration concerns.
- Canary deployments: the real integration test is production, run against a small, controlled slice of traffic.
Keep a very small E2E suite for the critical revenue path. Retire the rest. A flaky test that everyone re-runs is worse than no test, because it costs pipeline time and teaches the team to ignore red.
Deploying and releasing are different events
Deploying puts code on a production server. Releasing makes it visible to users. Teams that conflate the two are forced to choose between shipping slowly and exposing half-finished work, when the actual answer is to decouple the two events entirely.
Code in production = deployed code + toggled-on code
Once those are separate variables, you can deploy continuously and control user exposure independently, including turning a bad change off in seconds without a rollback, a pipeline run, or a war room.
It's worth separating three things that often get collapsed into "shipped":
- Deployment: the code is on the server.
- Expression: the code path is reachable by users.
- Exposure: what percentage of users are on it.
My working rule is that any deployment must be reversible within five minutes. If it isn't (a schema migration, a data backfill, anything with a one-way door) it needs a different process and explicit human approval. That five-minute line also turns out to be the right boundary for what an agent is allowed to do unsupervised.
Agents, with guardrails that assume they'll be wrong
Agentic AI extends toil elimination into work that was previously too context-dependent to script: incident triage, log correlation, drafting requirements, classifying pipeline failures. I use it heavily, and I've written up the concrete workflow separately.
The strategic point belongs here, though: an agent that causes an incident is far worse than no agent, because it burns the organizational goodwill that pays for the next five automations. So the reliability constraint applies to agents exactly as it applies to services.
| Risk | Mitigation |
|---|---|
| Destructive action | Human approval required for anything not reversible in five minutes. |
| Confidently wrong output | Agent output is a draft. A human owns the decision, always. |
| Sensitive data exposure | Scope credentials to the minimum necessary; log every tool call. |
| Silent reliability decay | Track agent action success rate as a first-class metric, not an anecdote. |
What elite performance actually looks like
The DORA research behind Accelerate is the closest thing this field has to empirical grounding, and it's worth knowing the actual numbers rather than the vibes:
| Metric | Low | High | Elite |
|---|---|---|---|
| Deployment frequency | < once/month | daily–weekly | on demand, many/day |
| Lead time for changes | 1–6 months | 1 day–1 week | < 1 hour |
| Change failure rate | 46–60% | 0–15% | 0–15% |
| Time to restore | > 6 months | < 1 day | < 1 hour |
The finding that matters most: speed and stability are not a trade-off. Elite teams deploy more often and fail less. Low performers give up both at once, because the thing that produces speed (automation, testing, small changes) is the same thing that produces stability. Anyone arguing for slower delivery in the name of safety is arguing against the evidence.
Culture is the strongest predictor, which is inconvenient
The same research found that organizational culture predicts delivery performance more strongly than any individual technical practice. Westrum's model splits organizations three ways by how information moves and how failure is handled:
| Type | Information | Response to failure |
|---|---|---|
| Pathological | Hoarded as a source of power | Find someone to blame |
| Bureaucratic | Ignored if it crosses a boundary | Seek justice, assign fault |
| Generative | Flows freely toward whoever needs it | Inquiry: what allowed this? |
Generative organizations have roughly 2.2× the odds of high delivery performance. That's a larger effect than most tooling decisions, which is humbling if you like buying tools.
The lever I've found that actually moves this is the blameless postmortem, not as a ceremony, but as a hard rule that the output must be a change to the system rather than a change to a person. If a postmortem's action items are all "be more careful," the system stayed exactly as fragile as it was before.
How I actually debug
Under pressure, people skip steps and start changing things. This is the sequence I fall back on, and the discipline is doing them in order:
- State the problem in one concrete sentence.
- State what success looks like.
- Name the constraints: time, blast radius, what must not change.
- Gather facts. Evidence only, no assumptions dressed up as facts.
- List three to seven plausible causes.
- Eliminate the simplest explanations first, one at a time.
- Choose the next action with the highest signal per unit of risk.
- Execute one change and measure it.
- If it isn't fixed, update the facts and re-rank. Do not stack changes.
- Confirm the fix from more than one angle, including recreating the original conditions.
- Write down the cause, the fix, and one guardrail that prevents recurrence.
Facts decide the finish line, not feelings.
The most common failure in incident response isn't a lack of skill. It's declaring victory when the symptom disappears rather than when the cause is understood.