---
title: Alert and monitor hygiene
slug: alert-and-monitor-hygiene
summary: How to audit monitors so every alert is actionable, owned and routed, and nothing important is silently muted or stuck in No Data.
order: 3
updated: 2026-11-11
status: ready
---

Use this page for a periodic monitor audit, after an incident that alerting missed, or when on-call complains about noise. A monitor that pages nobody, or pages everybody for nothing, is worse than no monitor: it gives false confidence.

Tool-specific export and as-code steps are in [Monitors and dashboards as code](./monitors-and-dashboards.md). The principles here apply across monitoring tools.

## Every alert must pass four tests

| Test | Question | Fails when |
| --- | --- | --- |
| Actionable | Is there something a human should do right now? | It fires and the answer is "ignore it" |
| Owned | Does a named team own it? | Owner tag missing or points to a dead team |
| Routed | Does it reach someone who can act? | Notify handle is a placeholder, a departed person, or empty |
| Documented | Does it link to a runbook? | The message has no runbook or dashboard link |

Measure the actionable test over a month:

$$\text{alert precision} = \frac{\text{pages that needed action}}{\text{total pages}}$$

If 9 of 30 pages needed a human, precision is 30%. Aim for it to rise each month.

## Audit in a fixed order, most dangerous first

1. **Wildcard or endless downtimes and mutes.** A mute scoped to everything with no end date silences the whole estate.
2. **Monitors in No Data on critical paths.** No Data usually means the metric name, tag or integration changed, so the monitor can never fire.
3. **Placeholder or broken notification handles.** For example `@team-placeholder`, a deleted channel, or an email of someone who left.
4. **Unowned monitors.** No `team:` or `service:` tag.
5. **Flapping and noisy monitors.** More than a few alerts per week that nobody acts on.
6. **Duplicates.** Two monitors for the same signal with different thresholds.

- [ ] Export monitors and downtimes to files (read-only API or UI export).
- [ ] Produce a report sorted by the order above.
- [ ] For each finding, propose one of: fix, re-route, tune, merge, or delete.
- [ ] Changes go through monitors-as-code in a merge request, not by clicking in the UI.

## Good alert message template

```text
{{#is_alert}}
<service> <signal> is {{value}} (threshold {{threshold}}) in <env>.
Impact: <what users see>.
Runbook: https://wiki.example.internal/runbooks/<service>-<signal>
Dashboard: https://monitoring.example.internal/dash/<id>
{{/is_alert}}
@<team-oncall-handle>
```

## Prefer symptoms over causes

- [ ] Page on what users feel: error rate, latency, failed transactions, queue age.
- [ ] Ticket, do not page, on causes: CPU, disk 70 percent, a single host down behind a load balancer.
- [ ] Tie paging alerts to [SLO burn rates](./slos-and-error-budgets.md) where possible.
- [ ] Every paging alert has a `priority` or `severity` tag matching the on-call policy.

## Decide what to do with a No Data monitor

```mermaid
flowchart TD
  A[Monitor in No Data] --> B{Does the thing it watches still exist?}
  B -- no --> C[Delete monitor via MR, note why]
  B -- yes --> D{Metric or tag renamed?}
  D -- yes --> E[Fix query via MR, test it fires]
  D -- no --> F{Integration or agent broken?}
  F -- yes --> G[Fix collection first, ticket it]
  F -- no --> H[Ask owner, set notify_no_data if silence is itself bad]
```

## Verify the fix

- [ ] Trigger a test notification, or temporarily lower a threshold in a non-production copy, and confirm it reaches the right place.
- [ ] Re-run the audit: the finding is gone.
- [ ] Record before and after counts (monitors total, No Data, unowned, muted) in the ticket.

## Pitfalls

- Deleting a No Data monitor that was the only guard on a rarely-used but critical path. Ask the owner first.
- Bulk-editing in the UI. You lose history and review. Use code.
- Lowering thresholds to "catch more". That buys noise, not safety.
- Muting for a deploy and forgetting to unmute. Every downtime gets an end time.