Flaky tests break your AI coding agent before they break your CI

The agent reports that the failing test is fixed. You open the diff:

+  await page.waitForTimeout(2000)
   await expect(page.getByRole('row')).toHaveCount(3)

Two lines you did not ask for sit in the config: retries: 2, and a test.skip on the neighbor that went red somewhere mid-session. The suite is green. Nothing was fixed.

That patch is not laziness. It is the agent solving the problem you posed. You asked for a green suite, and green was two seconds of sleep away.

A flaky test does not just cost you a rerun. It corrupts the signal every agent in your repo grades itself against. Fixing the grader is a different job from fixing a bug, and it has an order.

Why flaky tests hurt AI coding agents more than they hurt you

You have compensating knowledge: which spec hates a loaded laptop, which one goes red every third Monday. None of it reaches the session. An agent has three moves when a test goes red: change the code, change the test, or run it again. Two of them make a flake vanish without touching the cause, and both are cheaper than the third.

Frequency is the part people underestimate. You run the suite a few times a day; an agent doing test-first work runs it ten to twenty times per session. A 2% per-run flake rate is roughly a one in four chance of at least one false red per session, and each false red starts an investigation into a bug that does not exist.

Parallel agents multiply it: two sessions running the same suite from two worktrees against one database, one port, one fixture folder manufacture flakes that never happen when you work alone.

Gate: you cannot delegate a fix for a signal you have not measured.

Measure the flake rate before you delegate anything

The measurement only means something on an unchanged tree. Stash your work, then run the suspect file in a loop:

git stash
for i in $(seq 1 20); do <test command> tests/checkout || echo "FAIL $i"; done

No code changed between iterations, so every variation belongs to the test. Record the name, failures out of twenty, and whether the message repeats: one message points at a deterministic cause with a random trigger, varied messages point at ordering or environment. Then run the file alone, in the full suite, and with a single worker. Green alone and red in the suite tells you most of what an agent would spend an hour discovering.

Gate: you leave this step with a number and a test name, not an adjective.

Label the cause before an agent touches the test

Four archetypes cover almost everything. Each has a tell, plus the cheap fix an agent reaches for when you do not name it.

Time. Sleeps, timeouts, polling, dates near a boundary. Tell: it fails more on a loaded machine or a slow runner. Cheap fix: a bigger sleep.

Order and shared state. A row, a global, a cached singleton, a file left behind by whichever test created it. Tell: green alone, red in the suite, and reversing the order flips the result. Cheap fix: reorder the tests, or skip one.

Concurrency and resources. Two workers on the same port, temp path or database row. Tell: one worker is green. Cheap fix: pin the suite to a single worker, trading everyone's ten minutes for an unfixed race.

The outside world. Real network, real clock, an unpinned dependency, a machine that is not yours. Tell: it only fails in CI. Cheap fix: mock until the test asserts nothing.

Gate: every candidate carries one of those four labels before it becomes a task.

The brief that stops an agent from papering over a flake

Reuse this verbatim, one test per session:

tests/checkout/totals_spec fails 3 times in 20 on an unchanged tree, with different messages each time. Reproduce it first: run <test command> in a loop until it fails, and paste the failure. Then explain in one paragraph what makes the outcome depend on something other than the code under test. Only then propose the smallest fix. Do not add a sleep, a retry, or a skip, and do not weaken the assertion. Prove it with 20 consecutive green runs.

Reproduce first for the reason it works in ordinary debugging: a fix for a failure nobody saw is a guess with good grammar. The banned list is explicit because sleep, retry and skip are the local minimum any agent graded on green will find, the same instinct that relaxes an assertion during test-driven work. Twenty green runs, because one green run is the flake behaving normally. And the explanation is the real deliverable: a fix without one is a coincidence with a commit message.

Gate: the agent's output contains a before number and an after number.

Quarantine in the open, never in a skip

A skipped test is a deleted test with extra ceremony, and nobody comes back for it. Quarantine instead: tag it, keep running it, stop it gating the merge, give it an owner and thirty days to be fixed or deleted, and cap the list at a number you would be embarrassed to exceed.

Any new sleep, retry, skip or loosened assertion in an agent's diff is a review finding, one more reason reading the diff beats reading the summary.

Where flake hunting goes wrong, and what it really costs

Reruns are not free. Twenty runs of a twelve-minute suite is four hours of machine time, so measure one file, not the whole suite. Tokens cost too: Kong's engineering team pointed an agent harness at their fifteen flakiest tests, fixed twelve, gave up on three, at 4 to 5 hours and 100k to 200k tokens per test. Worth it for a test that blocks every merge, absurd for one nobody runs: the usual cost triage.

The scariest outcome is a successful fix. Some flakes are real races in your product, and the test is the only place one has surfaced. An agent that makes the symptom deterministic by mocking the queue has deleted your warning. When the label is concurrency, assume the fix belongs in the code until proven otherwise.

CI-only flakes cannot be reproduced locally. Different runner, timezone, parallelism. The loop becomes agent proposes, CI decides: slow enough to allow one written-down hypothesis per run.

The ceiling is architectural. Purpose-built systems stop short of solved: FlakyGuard, evaluated on industrial flaky tests, repaired 47.6% of the reproducible ones with 51.8% of its fixes accepted by developers. When one shared fixture sits behind fifteen flakes, per-test patches are interest payments and the real fix is the setup, which is also what makes the repo agent-ready.

A workspace where the flake hunt runs beside the feature work

Rerun loops are long, boring, and exactly the work that gets abandoned when it lives in a terminal tab behind your editor.

SanuDesk is a desktop app built for that shape. Your Claude Code, Codex and Gemini sessions tile into one grid, so the pane grinding through twenty reruns sits beside the pane doing today's feature and you glance instead of switching. A Kanban board sits next to it: one card per quarantined test carrying its label and flake rate, deployed to a fresh session with one click, parked in a review lane with the before and after numbers when the agent stops. The weekly hunt (rerun the suspect files overnight, record failures, drop new cards) becomes a Loop: same brief every run, journaled. It is bring-your-own-model, and grid and board are in the free plan (see pricing).

Run one test twenty times tonight

Pick the test you already shrug at. Stash your work, run it twenty times, and write down two numbers: how many failed, and how long the twenty runs took.

Zero in twenty means your suspicion was wrong and the last red was your code. Three in twenty means you own the first line of a brief an agent can satisfy, plus the proof standard it has to meet. Either way you traded a shrug for a number, and agents cannot act on shrugs.

Download SanuDesk free to keep quarantined tests on a board beside the work they interrupt, or see how the grid, the board and Loops fit together on the features page.