Automation blueprint for SRE toil

Use this page to plan which parts of an SRE queue to automate and in what order. Automating the routine work does not mean an agent runs production. The agent does the reading, sorting, drafting and checking. The human spends their time on approvals, incidents and design.

Prerequisites: Jira connected, Claude Code configured, a place to run jobs such as herdr or scheduled CI pipelines, and read-only cloud access.

The architecture

flowchart LR
  J[(Ticket system)] -->|read-only query| I[Intake]
  I --> C[Classify: one label per ticket]
  C -->|closeout| W1[Closeout drafter]
  C -->|pipeline| W2[Pipeline porter]
  C -->|access| W3[Access MR drafter]
  C -->|monitor| W4[Monitor auditor]
  C -->|investigation / human| H((Human))
  AWS[(Cloud, read-only)] --> WA[Watchers]
  W1 --> D[/Drafts/]
  W4 --> D
  W2 -->|draft MR| G[(Git host)]
  W3 -->|draft MR| G
  WA --> R[/Digests/]
  D --> Q[Approval queue]
  G --> Q
  R --> Q
  Q --> H
  H -->|approve, merge, post| G
  H -->|post| J

The components

What stays human

• Approving and merging every MR, and every production deploy.
• Incidents and on-call decisions.
• Architecture, priorities and saying no.
• Security exceptions and anything touching production data.
• Talking to people, and posting under their name.

Build it in this order

Build and trust one step before starting the next. Run each in dry-run for a week before relying on it.

1. Intake, classify and closeout drafts. Highest volume, zero risk. Measure how often the label and draft are right.
2. Approval queue and the daily chain. One file to read each morning.
3. Pipeline ports. Template-driven draft MRs clear migration backlogs in batches (pipeline migration).
4. Access requests. Guardrailed least-privilege Terraform MRs (access).
5. Monitor audit. Weekly hygiene report plus fix MRs (hygiene).
6. Watchers and weekly status. Digests and a three-line Friday status.

A worker, end to end

Every worker follows the same skeleton: read, redact, ask, validate, write a draft.

#!/usr/bin/env bash
# bin/triage-closeout.sh: draft close / bounce / reject comments for In Review tickets
set -euo pipefail
source "$(dirname "$0")/../lib/common.sh"     # log, redact, ask_claude, DRY_RUN handling
common_parse "$@"                              # --limit N, --execute (default is dry-run)

out="state/drafts/triage-$(date +%F).md"
jq -c 'select(.label=="closeout")' state/tickets/classified.jsonl | head -n "${LIMIT:-25}" |
while read -r t; do
  key="$(jq -r .key <<<"$t")"
  answer="$(redact <<<"$t" | ask_claude prompts/triage-closeout.md)"   # tools disabled, one turn
  decision="$(jq -r .decision <<<"$answer" 2>/dev/null || echo human)"
  case "$decision" in close|bounce|reject) ;; *) decision=human ;; esac  # unknown -> human
  printf '## %s  decision: %s\n%s\n[ ] approve\n\n' "$key" "$decision" "$(jq -r .comment <<<"$answer")" >> "$out"
done
log "wrote $out"

The daily chain

#!/usr/bin/env bash
# bin/daily.sh: run every weekday morning, before the human starts
set -euo pipefail
bin/intake.sh          # tickets -> state/tickets/*.jsonl
bin/classify.sh        # one label each
bin/triage-closeout.sh --limit 25
bin/watchers.sh        # cloud write events, cert expiry, drift, stale tickets
bin/approval-queue.sh  # -> state/approval-queue-YYYY-MM-DD.md

The approval queue file looks like this:

# Approval queue 2026-11-10
## Draft MRs (3)
- [ ] !<mr> access: read-only role for <service> (plan: 2 to add)
## Jira drafts (11)
- [ ] PROJ-123 close: renewed and verified
## Needs a human (4)
- PROJ-124 unclear scope, asked requester
## Watchers
- 2 certificates expire within 30 days
- Drift in 1 stack: <stack>

Weekly status in three lines

Shipped: <2-3 items with links>
Reliability: <SLO status, incidents, budget>
Blocked: <top blocker and the ask>

Draft it from Done-this-week JQL, merged MRs and the incident log. The human edits and sends it.

Measure that it works

• Toil hours per week, before and after (log them; estimates are fine).
• Draft acceptance rate: approved as-is, edited, rejected.
• Classifier accuracy from a weekly spot check of 20 tickets.
• Time to first response on new tickets.
• Zero unapproved writes. Audit the write log weekly.

Pitfalls

• Starting with the scary automation (deploys, access grants). Start with drafts.
• Silent failures. Every job writes a status line to the approval queue, including "failed".
• Letting the classifier's unknowns disappear. Anything not clearly labelled goes to "human".
• Building on an unapproved machine or tool. Get the platform approved first; it is much harder to retrofit.