---
name: flaky-test-hunter
description: Use when a test passes sometimes and fails sometimes. Proves flakiness statistically, finds the nondeterminism source (time, order, shared state, network, concurrency), and fixes it instead of adding retries.
---

# Flaky Test Hunter

Rule: a retry is a bandage, not a fix. Only add a retry if you also file the root cause.

## Step 1 - Prove it is flaky
Run the single test in a loop, N=50 (or until 3 failures):
- Jest/Vitest: `for i in $(seq 50); do npx jest path/to.test.ts -t "name" --silent || echo FAIL $i; done`
- Pytest: `pip install pytest-repeat; pytest path::test_name --count=50 -x -q`
- Go: `go test -run TestName -count=50 ./pkg/...`
- Record: failures / runs. 0/50 locally but fails in CI -> it is environment or order dependent; go to Step 3.

## Step 2 - Capture a failing run
Save stdout, the assertion diff, and timestamps. Compare a passing vs failing output side by side. The diff between them usually names the culprit (a timestamp, an ID ordering, a missing row).

## Step 3 - Test the five usual suspects in this order
1. **Order dependence**: run the full file in random order (`pytest -p random_order`, `jest --randomize`, `go test -shuffle=on`). If it fails only after a specific other test, that test leaks state (globals, DB rows, env vars, mocked modules not restored).
2. **Time**: grep the test for `now()`, `Date`, `time.Now`, timezones, `sleep`, timeouts. Freeze the clock (fake timers / freezegun / injected clock). Check month/day boundaries and DST.
3. **Async/race**: missing `await`, assertions before a promise resolves, `setTimeout`-based waits, goroutines without sync. Replace sleeps with explicit wait-for-condition. Run Go with `-race`.
4. **Unordered collections**: comparing lists from sets/maps/DB queries without ORDER BY. Sort before compare or assert on sets.
5. **External I/O**: real network, real ports (port already in use), shared temp dirs, filesystem case sensitivity. Mock network; use port 0; use unique temp dirs per test.

## Step 4 - Fix and re-prove
- Apply the smallest fix that removes the nondeterminism.
- Re-run the loop from Step 1 with N=100. Required: 0 failures.
- If CI-only: run CI's parallelism setting locally (`-n auto`, `--maxWorkers`).

## Step 5 - Report
```
TEST: <id>
FLAKE RATE BEFORE: x/50   AFTER: 0/100
SOURCE: <order | time | race | unordered | io> - <one line evidence>
FIX: <diff summary>
```
If you cannot fix it now: quarantine with a tagged skip that links an issue, never a silent delete.