Use this page for a periodic monitor audit, after an incident that alerting missed, or when on-call complains about noise. A monitor that pages nobody, or pages everybody for nothing, is worse than no monitor: it gives false confidence.
Tool-specific export and as-code steps are in Monitors and dashboards as code. The principles here apply across monitoring tools.
Every alert must pass four tests
| Test | Question | Fails when |
|---|---|---|
| Actionable | Is there something a human should do right now? | It fires and the answer is "ignore it" |
| Owned | Does a named team own it? | Owner tag missing or points to a dead team |
| Routed | Does it reach someone who can act? | Notify handle is a placeholder, a departed person, or empty |
| Documented | Does it link to a runbook? | The message has no runbook or dashboard link |
Measure the actionable test over a month:
$$\text{alert precision} = \frac{\text{pages that needed action}}{\text{total pages}}$$
Worked example
30%alert precision
If 9 of 30 pages needed a human, precision is 30%. Aim for it to rise each month.
Audit in a fixed order, most dangerous first
- Wildcard or endless downtimes and mutes. A mute scoped to everything with no end date silences the whole estate.
- Monitors in No Data on critical paths. No Data usually means the metric name, tag or integration changed, so the monitor can never fire.
- Placeholder or broken notification handles. For example
@team-placeholder, a deleted channel, or an email of someone who left. - Unowned monitors. No
team:orservice:tag. - Flapping and noisy monitors. More than a few alerts per week that nobody acts on.
- Duplicates. Two monitors for the same signal with different thresholds.
- Export monitors and downtimes to files (read-only API or UI export).
- Produce a report sorted by the order above.
- For each finding, propose one of: fix, re-route, tune, merge, or delete.
- Changes go through monitors-as-code in a merge request, not by clicking in the UI.
Good alert message template
{{#is_alert}}
<service> <signal> is {{value}} (threshold {{threshold}}) in <env>.
Impact: <what users see>.
Runbook: https://wiki.example.internal/runbooks/<service>-<signal>
Dashboard: https://monitoring.example.internal/dash/<id>
{{/is_alert}}
@<team-oncall-handle>
Prefer symptoms over causes
- Page on what users feel: error rate, latency, failed transactions, queue age.
- Ticket, do not page, on causes: CPU, disk 70 percent, a single host down behind a load balancer.
- Tie paging alerts to SLO burn rates where possible.
- Every paging alert has a
priorityorseveritytag matching the on-call policy.
Decide what to do with a No Data monitor
flowchart TD
A[Monitor in No Data] --> B{Does the thing it watches still exist?}
B -- no --> C[Delete monitor via MR, note why]
B -- yes --> D{Metric or tag renamed?}
D -- yes --> E[Fix query via MR, test it fires]
D -- no --> F{Integration or agent broken?}
F -- yes --> G[Fix collection first, ticket it]
F -- no --> H[Ask owner, set notify_no_data if silence is itself bad]
Verify the fix
- Trigger a test notification, or temporarily lower a threshold in a non-production copy, and confirm it reaches the right place.
- Re-run the audit: the finding is gone.
- Record before and after counts (monitors total, No Data, unowned, muted) in the ticket.
Pitfalls
- Deleting a No Data monitor that was the only guard on a rarely-used but critical path. Ask the owner first.
- Bulk-editing in the UI. You lose history and review. Use code.
- Lowering thresholds to "catch more". That buys noise, not safety.
- Muting for a deploy and forgetting to unmute. Every downtime gets an end time.