← CI Firefighter

Stop retrying flaky tests: a 5-suspect checklist (plus a free agent skill)

A retry is a bandage, not a fix. Here is the checklist I use when a test passes sometimes and fails sometimes.

1. Prove it's flaky first

Loop the single test 50 times:

# pytest
pip install pytest-repeat && pytest path::test_name --count=50 -x -q
# Go
go test -run TestName -count=50 ./pkg/...
# Jest
for i in $(seq 50); do npx jest path/to.test.ts -t "name" --silent || echo FAIL $i; done

0/50 locally but red in CI? It's order- or environment-dependent.

2. Check the five usual suspects, in this order

1. Order dependence – shuffle: pytest -p random_order, jest --randomize, go test -shuffle=on. Fails only after another test → that test leaks state.
2. Time – now(), timezones, DST, month boundaries, sleep. Freeze the clock.
3. Async/races – missing await, sleep-based waits. Wait for a condition instead; run Go with -race.
4. Unordered collections – set/map/DB results compared as lists without ORDER BY.
5. External I/O – real network, fixed ports (use port 0), shared temp dirs.

3. Re-prove the fix

Rerun the loop with N=100. The bar is 0 failures. Can't fix it today? Quarantine with a tagged skip linking an issue, never a silent delete.

Free: the whole checklist as an agent skill

I packaged this as a SKILL.md your coding agent (Cursor / Claude Code / Codex) can follow end to end, with a report template: https://ci-firefighter.pages.dev/assets/free-sample-flaky-test-hunter.md

It's free, copy it into your skills folder.

If it helps, it's one of 7 skills in a paid pack (CI triage, bisecting regressions, dependency bumps, lockfile conflicts, env drift, postmortems) plus a PDF field guide, $19: https://ci-firefighter.pages.dev/