← CI Firefighter

The first 15 minutes of a production incident: a checklist that does not panic

The first quarter hour decides whether an incident is a short blip or a long night. Not because you fix it in that time, but because that is when people either get organised or start changing things at random. Here is a calm order of operations.

Minute 0–2: Confirm it is real

• Is the alert backed by a user-facing symptom? Check the status of one real request, not just the dashboard.
• What is the scope: one region, one endpoint, one customer, everyone?
• Write down the time you first saw it. You will need it later.

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://api.example.com/health

Minute 2–5: Declare and assign

Say it out loud in one channel: “Declaring an incident, I am incident lead until handed off.” One person leads, one person communicates. The lead does not debug deep; the lead keeps the list.

INCIDENT: checkout 5xx above 20%
Lead: @a   Comms: @b   Started: 14:02 UTC
Status: investigating   Next update: 14:20

Minute 5–10: Ask “what changed?”

Most incidents follow a change. Check, in this order:

Source	Question
Deploys	Anything shipped in the last 2 hours?
Config / flags	Feature flag flipped, env var edited, secret rotated?
Infra	Scaling event, node rotation, certificate expiry, DNS change?
Dependencies	Provider status page, database, queue, third-party API?
Traffic	Spike, new client, bot, batch job?

git log --since="2 hours ago" --oneline origin/main
kubectl rollout history deploy/checkout

Minute 10–15: Mitigate before you understand

You do not need a root cause to stop the bleeding. Prefer the most reversible lever first:

1. Roll back the last deploy or flip the flag off.
2. Shed or rate-limit the offending traffic.
3. Fail over or scale out.
4. Only then: hot-fix forward.

Note every action with a timestamp in the channel. If an action does not help within a few minutes, undo it before trying the next one, so you only change one thing at a time.

What not to do

• Do not restart everything “to see”. You erase the evidence.
• Do not let five people debug the same log in silence.
• Do not promise an ETA you do not have. Promise the next update time instead.

At minute 15

You should have: a lead, a channel, a scope, a short list of recent changes, one mitigation tried or in progress, and a next update time. That is a good first 15 minutes, even if nothing is fixed yet.

Free sample

If you want to see how checklists like this look as an agent skill, there is a free SKILL.md you can drop into Cursor, Claude Code or Codex: free sample skill.

The fuller version – declaring, “what changed” triage, mitigation choice, status updates, timeline capture and a blameless postmortem draft – is packaged as 6 skills in Incident Triage Desk, $19.