On-call is usually treated as an operational cost: a rota to fill, a payment to negotiate, a burden to distribute fairly. All of that matters, but it misses the more useful framing. On-call is a feedback mechanism, and where you point it determines what gets built.
The people who ship should carry the pager
When the team that writes a service is the team that gets woken by it, the design changes. Retries get bounded. Errors get messages that say what to do. Timeouts get set deliberately rather than left at whatever the library defaults to. Nobody needs to be persuaded to care about operability, because the consequences arrive personally at 3am.
When those are different teams, the incentives split. One group optimizes for shipping, the other absorbs the cost, and the cost is invisible to the people who could most cheaply prevent it. You end up with a system that is expensive to run and nobody responsible for that expense.
This is not about punishment. It is the shortest available feedback loop between a design decision and its operational consequence, and shortening feedback loops is most of what this job is.
A pager that is quiet is doing its job
There is a mild cultural assumption that a busy on-call shift means important work. It usually means unfinished work. Every page is either something the system should have handled itself, something that should have been caught earlier, or a genuine novel failure. Only the third category is a good use of a human being at night.
Tracking the split is worth doing. If most pages fall into the first two categories, the rotation is absorbing a backlog rather than responding to surprises, and adding people to the rota will not fix it.
Make the shift survivable
Some of this is just logistics, and getting it wrong undermines everything else:
- One primary, one backup, and a documented path to wake someone with more context.
- Shifts long enough to build continuity and short enough to recover from. A week is common and reasonable.
- An explicit expectation that a bad night means a light next day. Otherwise the cost is paid in quality and nobody records it.
- A handover with substance: what fired, what is still unresolved, what to watch.
Give the rotation time to fix things
The most common structural mistake is putting someone on call while also expecting their normal delivery output that week. The result is that pages get acknowledged and nothing gets improved, so next week's rotation inherits the same pages.
The version that works treats on-call as a real assignment with its own goal: keep the service up, and leave the rotation better than you found it. That means the person carrying the pager has the time and the standing to delete a bad alert, write the runbook that was missing, or fix the thing that woke them. Otherwise the rotation is a treadmill, and treadmills do not go anywhere.