Use this page when a scheduled job, report or file transfer fails or does not run. These failures are dangerous because they are quiet: nobody notices until someone asks about a missing file, and re-running blindly can send the same output twice.
Answer these first
- Which run failed (job name, scheduled time in UTC, environment)?
- Did it fail, not start at all, or run and produce a bad or empty output?
- Is the job idempotent? Can it be re-run without duplicating side effects?
- Is there a deadline (a cut-off or processing window)? How long until it?
Common causes and checks
| Symptom | Likely cause | Check |
|---|---|---|
| Auth failure on a file transfer | Rotated or expired password or key, allowlist changed | Job log, contact, key age |
| Connection timeout | Firewall, NAT IP changed, remote host down | Egress IP, security groups, a port check from the runner |
| Host key mismatch | Remote host rotated its key | Compare the fingerprint out of band before accepting |
| File not found | Upstream job late or failed | Upstream job status and timing |
| Permission denied on write | Remote directory or quota change | The other side of the transfer |
| Job never started | Scheduler disabled, rule target missing, cron in wrong time zone | Scheduler history, rule targets |
| Ran but empty | Upstream query returned nothing; date window off by one around midnight or DST | Input counts, run parameters |
# Scheduled rules and whether their targets still exist (read-only)
aws --profile <readonly-profile> --region <region> events list-rules \
--query 'Rules[].{name:Name,state:State,schedule:ScheduleExpression}' --output table
aws --profile <readonly-profile> --region <region> events list-targets-by-rule --rule <rule-name>
# Recent runs of a containerised batch task
aws --profile <readonly-profile> --region <region> ecs list-tasks --cluster <cluster> \
--family <task-family> --desired-status STOPPED
aws --profile <readonly-profile> --region <region> ecs describe-tasks --cluster <cluster> \
--tasks <task-arn> --query 'tasks[].{stopped:stoppedReason,exit:containers[0].exitCode}'
# Connectivity from the runner (no login attempted)
nc -vz -w 5 <remote-host> <port>
# If the transfer is over SSH, compare the host key fingerprint out of band
ssh-keyscan -p <port> <remote-host> 2>/dev/null | ssh-keygen -lf -
Recover the missed run
- Fix the cause first, or the re-run fails the same way.
- Confirm whether any part of the failed run already reached the far side (remote listing or confirmation). Partial deliveries are the main duplicate risk.
- Re-run for the exact missed window with explicit parameters, not "now".
- Confirm receipt when the output is important or the data is regulated.
- Record the run, the parameters and the confirmation in the ticket.
- Re-running a job that writes to a shared system or sends customer files needs a human's explicit approval for that run.
Make the next failure loud
- Alert on "did not succeed by the deadline", not only on errors. A job that never starts emits no error.
- Heartbeat or dead-man's-switch monitor for every scheduled job.
- Alert on empty output when output is expected.
- Credentials for the far side stored in a secret manager with a rotation date and an owner.
- Partner host key fingerprints pinned and documented.
Pitfalls
- Accepting a changed host key without checking with the other side. That is exactly what an attacker would want.
- Cron schedules in local time across a DST change.
- Rules that still exist but point at a deleted function. They fail silently forever; sweep for them.