Blog · Agent skills

How to give your coding agent a flaky-test playbook

Ask a coding agent to "fix the red build" and, if the failure is a flaky test, you will usually get one of three things: a retry decorator, a longer sleep, or a quietly deleted assertion. The build goes green, the bug stays. The agent isn't being lazy; it simply has no procedure for flakiness, so it picks the shortest path to green.

The fix is to hand it a procedure. Cursor, Claude Code and Codex can all load instructions from a skill or rules file (SKILL.md, .cursor/rules, CLAUDE.md, AGENTS.md). Here is how to write one that actually works for flaky tests.

1. Start with a trigger the agent can recognize

Agents decide when to use a skill from its description. Make it concrete:

---
name: flaky-test-hunter
description: Use when a test passes sometimes and fails sometimes, when CI is red
  but the test passes locally, or before adding any retry, sleep or skip to a test.
---

The last clause matters most: it fires the skill exactly at the moment the agent is about to paper over the problem.

2. Forbid the shortcuts explicitly

Models follow explicit rules far better than implied ones. Put the non-negotiables near the top:

Do not add retries, longer timeouts or sleep calls as a fix.
Do not delete or weaken assertions.
If you cannot fix it, quarantine with a tagged skip that links an issue, and say so in the summary.

3. Make "prove it" the first step

An agent that guesses will guess wrong. Require a reproduction loop before any edit, with the exact commands for your stack:

pytest path::test_name --count=50 -x -q      # needs pytest-repeat
go test -run TestName -count=50 -race ./pkg/...
for i in $(seq 50); do npx jest path -t "name" --silent || echo FAIL $i; done

Tell it what the result means: fails locally in the loop means a timing or race bug; passes 50/50 locally but fails in CI means order or environment dependence, so shuffle next (pytest -p random_order, jest --randomize, go test -shuffle=on).

4. Give it an ordered suspect list

Agents work best with a checklist they can walk top to bottom and tick off. A good default order:

Shared state / test order – globals, singletons, DB rows or files left by another test.
Time – now(), timezones, DST, month ends. Freeze the clock.
Async and races – missing await, sleep-based waits. Wait on a condition.
Unordered results – sets, maps, SQL without ORDER BY compared as lists.
External I/O – real network, fixed ports, shared temp dirs.

Ask the agent to note which suspects it ruled out and how. That one line turns a vague "fixed flake" commit into something a reviewer can trust.

5. Define "done" as a number

"Looks fixed" is not a stopping condition an agent can check. "Rerun the loop with N=100 and get 0 failures" is. Put the number in the skill.

6. Ask for a short report

End the skill with a template so every flaky-test PR reads the same way:

Test: ...
Repro: X/50 failures before, 0/100 after
Root cause (suspect #): ...
Fix: ...
Ruled out: ...

Where to put it

For Claude Code, drop it in .claude/skills/flaky-test-hunter/SKILL.md. For Cursor, a rule file under .cursor/rules/ works, or reference it from AGENTS.md. Commit it to the repo so the whole team's agents get the same playbook.

Free copy

If you'd rather not write it from scratch, I published the version I use as a free SKILL.md, with the loop commands, suspect list and report template: free-sample-flaky-test-hunter.md. Copy it, edit the commands for your stack, done.

It is part of CI Firefighter, a small set of agent skills for CI triage, regression bisecting, dependency bumps and similar chores, if you want more of the same.

Credits: Listed on MadeWithStack