Make your agent earn the right to fix a flaky test: a shuffle-seed harness and a commit guard

A test fails in CI, passes on rerun, and somebody tells the agent to fix the flaky test. The agent reads the file, sees nothing wrong, and does the thing that makes the red go away: wraps it in a retry, adds a short sleep, or raises the timeout. CI goes green. The race is still there, now hidden behind a setting that nobody will look at again.

The fix is to stop letting the agent start from the code and make it start from a measurement. Before it changes anything, it has to reproduce the failure under a shuffled test order, get a failure rate and a seed that replays it, and afterwards show the same seeds passing. Separately, a commit guard rejects any diff that “fixes” a test by adding the things that hide flakes.

Here’s the whole thing: one Python script (the hunt), one skill (the procedure), one guard script and one hook entry. It’s built for Vitest. Most of it is stack-neutral, and the variations section says what to swap.

The scenario

A currency module keeps a rates cache at module level, and the tests share it:

// src/rates.ts
const cache = new Map<string, number>();

export function getRate(currency: string): number {
  if (!cache.has(currency)) {
    cache.set(currency, currency === "EUR" ? 0.86 : 1);
  }
  return cache.get(currency)!;
}

export function overrideRate(currency: string, rate: number): void {
  cache.set(currency, rate);
}

// test/rates.test.ts
import { describe, expect, it } from "vitest";
import { getRate, overrideRate } from "../src/rates";

describe("rates", () => {
  it("returns the default EUR rate", () => {
    expect(getRate("EUR")).toBe(0.86);
  });

  it("uses an override", () => {
    overrideRate("EUR", 0.9);
    expect(getRate("EUR")).toBe(0.9);
  });

  it("returns 1 for USD", () => {
    expect(getRate("USD")).toBe(1);
  });
});

Run in file order, all three pass. Every time. Nothing about this looks flaky, which is exactly why an agent that only reads the code can’t find the problem. The bug only exists in orders where “uses an override” runs before “returns the default EUR rate”.

Real suites hit the same shape through a shared database row, a singleton, a mutated environment variable, a fake timer left running or a mock that isn’t restored. The order that exposes it in CI depends on file scheduling and worker timing, so it looks random.

Step 1: a script that makes the order random on purpose

Vitest can randomise test order and pin the seed. The config keys are sequence.shuffle.tests and sequence.seed, and both have CLI flags. The docs note that the seed has no effect unless shuffling is on, so the flags go together.

Save this as scripts/flake-hunt.py:

#!/usr/bin/env python3
"""Run a Vitest target under shuffled orders and report how often it fails.

usage: flake-hunt.py [--runs N] [--first-seed S] [--out FILE] [vitest args...]
"""
import argparse, json, subprocess, sys, tempfile, os
from collections import Counter

ap = argparse.ArgumentParser()
ap.add_argument("--runs", type=int, default=30)
ap.add_argument("--first-seed", type=int, default=1)
ap.add_argument("--out", default="flake-report.json")
ap.add_argument("vitest_args", nargs="*")
args = ap.parse_args()

failing_seeds, by_test = [], Counter()
for seed in range(args.first_seed, args.first_seed + args.runs):
    with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as tmp:
        path = tmp.name
    subprocess.run(
        ["npx", "vitest", "run", "--sequence.shuffle.tests",
         f"--sequence.seed={seed}", "--reporter=json", f"--outputFile={path}",
         *args.vitest_args],
        stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
    )
    with open(path) as f:
        report = json.load(f)
    os.unlink(path)
    failed = [
        f"{r['name'].split('/')[-1]} > {t['fullName']}"
        for r in report["testResults"]
        for t in r["assertionResults"] if t["status"] == "failed"
    ]
    if failed:
        failing_seeds.append(seed)
        by_test.update(failed)

rate = len(failing_seeds) / args.runs
summary = {"runs": args.runs, "failed_runs": len(failing_seeds),
           "rate": round(rate, 3), "failing_seeds": failing_seeds,
           "tests": dict(by_test)}
with open(args.out, "w") as f:
    json.dump(summary, f, indent=2)
print(f"{len(failing_seeds)}/{args.runs} shuffled runs failed ({rate:.0%})")
for name, n in by_test.most_common():
    print(f"  {n:>3}  {name}")
if failing_seeds:
    print(f"replay: npx vitest run --sequence.shuffle.tests --sequence.seed={failing_seeds[0]}")
sys.exit(1 if failing_seeds else 0)

Each run uses a different seed, writes Vitest’s JSON report to a temp file and records which tests failed. Seeds are consecutive integers on purpose: the same seeds can be run again after the fix, so the before and after comparison is like for like. Add flake-report.json to .gitignore.

Run it:

python3 scripts/flake-hunt.py --runs 30

16/30 shuffled runs failed (53%)
  16  rates.test.ts > rates returns the default EUR rate
replay: npx vitest run --sequence.shuffle.tests --sequence.seed=2

Thirty runs of a three-test file finish in well under a minute. On a real suite, point the script at the one file or test name (everything after the options goes straight to Vitest) so the loop stays cheap.

Step 2: replay one seed and read the victim

The report gives a seed that fails every time it’s used. Replay it:

npx vitest run --sequence.shuffle.tests --sequence.seed=2

 RUN  v5.0.5
      Running tests with seed "2"
 ❯ test/rates.test.ts (3 tests | 1 failed) 19ms
   ❯ rates (3)
     × returns the default EUR rate 11ms
 FAIL  test/rates.test.ts > rates > returns the default EUR rate
AssertionError: expected 0.9 to be 0.86 // Object.is equality

Two things matter here. The failure is now deterministic, so the agent can iterate on it in seconds instead of waiting for CI. And the failing test is the victim, not the culprit. The assertion says 0.9, a number that appears nowhere in the failing test. Something earlier wrote it.

That’s the diagnostic move a tired human skips and an agent skips even faster: when a test fails only in some orders, search for what the other tests leave behind. Here that’s the single call to overrideRate.

The fix is a reset between tests:

export function resetRates(): void {
  cache.clear();
}

import { beforeEach, describe, expect, it } from "vitest";
import { getRate, overrideRate, resetRates } from "../src/rates";

describe("rates", () => {
  beforeEach(() => resetRates());
  // ...tests unchanged
});

Run the hunt again with the same seeds:

0/30 shuffled runs failed (0%)

That’s the proof. A single green rerun proves nothing about a failure that appears 53% of the time and even less about one that appears 3% of the time.

Step 3: how many runs prove a fix

A flake with failure rate p survives n clean runs with probability (1 - p) to the power n. To push that below 5%, n needs to be about 3 divided by p, because (1 - p) to the power 3/p is close to e to the minus 3, which is 0.05.

The skill below encodes this as “at least ceil(3 / p) runs, never fewer than 30”. The floor of 30 exists because the measured rate from a small sample is itself noisy. A flake that showed 2 failures in 30 might really be 1% or 10%.

A fix that’s checked with five reruns is a guess. Make the agent show the arithmetic.

Step 4: put the procedure in a skill

A project skill lives at .claude/skills/<name>/SKILL.md, takes arguments through $ARGUMENTS, and with disable-model-invocation: true only runs when you type it. That last flag is deliberate: this is a procedure you call on a named test, not something that should fire whenever the word “flaky” appears in conversation.

---
name: fix-flake
description: Fix an intermittently failing Vitest test by reproducing it under shuffled order first. Use when a test passes on rerun, fails only in CI, or is called flaky.
argument-hint: [test file or name]
disable-model-invocation: true
allowed-tools: Bash(python3 scripts/flake-hunt.py *) Bash(python3 scripts/no-masking.py) Bash(npx vitest *)
---

Fix the flaky test: $ARGUMENTS

Work in this order and do not skip a step.

1. Measure. Run `python3 scripts/flake-hunt.py --runs 30 $ARGUMENTS` and read `flake-report.json`. If nothing fails in 30 runs, say so and stop. Do not guess at a fix.
2. Replay. Run the first failing seed with `npx vitest run --sequence.shuffle.tests --sequence.seed=<seed>` and read the failure. The failing test is usually the victim, not the cause. Find the test or module that leaves state behind.
3. Fix the cause. Reset the shared state in a `beforeEach` or `afterEach`, or remove the shared state. Never add retries, raise timeouts, add sleeps, skip the test or loosen an assertion. `scripts/no-masking.py` blocks those at commit time.
4. Prove it. Re-run the hunt with the same seeds. If the original failure rate was p, use at least ceil(3 / p) runs, and never fewer than 30. Report the before and after counts.
5. Stop. Do not commit. Show the diff and both counts.

The allowed-tools line is documented as the set of tools Claude can use without asking during the skill’s turn. It’s there because without it the agent stops to ask before running the guard script, as the next section shows.

Step 1 has an exit hatch. If thirty shuffled runs are clean, the flake isn’t an ordering problem, and the right output is “not reproduced” rather than a speculative patch. See the variations for what to try next.

Run it with /fix-flake test/rates.test.ts. On this repo, Claude Code 2.1.309 on the Haiku model reported 16 of 30 failing before and 0 of 30 after, named overrideRate as the leak and the failing test as the victim, and added a reset. It also noticed that the 53% rate gave a computed minimum of 6 runs and applied the floor of 30 instead, which is the behaviour the step asks for. That’s one run on a tiny repo, so treat it as a smoke test, not a benchmark.

One thing it reported: it couldn’t run scripts/no-masking.py because the command needed approval. That’s the gap allowed-tools closes, and it’s worth checking on your own setup. A verification step the agent can’t execute becomes a step it skips and then apologises for.

Step 5: a guard that rejects masking

The skill says never add retries. A skill is an instruction, and instructions get ignored under pressure, particularly in a long session where the agent has already failed twice. The guard turns the rule into a check that fails.

Save this as scripts/no-masking.py:

#!/usr/bin/env python3
"""Fail if the pending diff masks a flaky test instead of fixing it."""
import re, subprocess, sys

PATTERNS = {
    r"\b(it|test|describe)\.(skip|todo)\b": "skipped test",
    r"\b(retry|retries)\b\s*[:=]": "retry setting",
    r"--retry": "retry flag",
    r"\btestTimeout\b": "raised timeout",
    r"\bsetTimeout\(": "sleep or timer in a test",
    r"\bsleep\(|\bdelay\(": "sleep in a test",
}

diff = subprocess.run(["git", "diff", "HEAD", "-U0"],
                      capture_output=True, text=True, check=True).stdout
path, hits = None, []
for line in diff.splitlines():
    if line.startswith("+++ "):
        path = line[6:]
    elif line.startswith("+") and path:
        if not re.search(r"(\.test\.|\.spec\.|vitest\.config)", path):
            continue
        for rx, why in PATTERNS.items():
            if re.search(rx, line):
                hits.append(f"{path}: {why}: {line[1:].strip()}")
if hits:
    print("Blocked: this change hides a flake instead of fixing it.", file=sys.stderr)
    print("\n".join(f"  {h}" for h in hits), file=sys.stderr)
    sys.exit(2)

It looks only at added lines in test files and Vitest config, using git diff HEAD so staged and unstaged tracked changes are both covered, which matters when the agent commits with git commit -a. It exits 2 because that’s the code a Claude Code hook treats as “block”.

Try it on a retry:

Blocked: this change hides a flake instead of fixing it.
  test/rates.test.ts: retry setting: describe("rates", { retry: 3 }, () => {

The patterns are blunt on purpose. A legitimate setTimeout in a test about debouncing will trip them, and the right response is to read the message and either restructure or deliberately allow the line, not to loosen the guard for everyone.

Step 6: run the guard where the agent commits

Two places, because they catch different things.

First, a Claude Code PreToolUse hook that fires only on git commit. The if field takes permission-rule syntax, so the script doesn’t spawn for every Bash call. In .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "if": "Bash(git commit *)",
            "command": "python3 \"$CLAUDE_PROJECT_DIR\"/scripts/no-masking.py"
          }
        ]
      }
    ]
  }
}

With the hook in place, asking the agent to commit the retry: 3 change gets a refusal. On the same Haiku setup, it reported that a pre-commit hook had blocked the commit because the change hid a flaky test, showed the offending diff line, said it hadn’t bypassed the hook, and suggested shared state as the likely root cause. git log showed no new commit.

The Claude Code docs describe the if filter as best-effort: when the tool can’t work out which commands a Bash call runs (a command assembled from a variable, say), the hook runs anyway, and the docs recommend the permission system for hard enforcement. For this guard, running more often than needed is the safe direction.

Second, the same script as a plain git hook, so a human committing outside Claude Code, or an agent that doesn’t run Claude Code, hits it too. With Husky it’s one line in .husky/pre-commit. The commit gate Playbook covers Husky wiring and the --no-verify gap, and the same caveats apply here: a git hook is a guardrail against an agent being sloppy, not against one being adversarial.

Make it yours

Shuffle files as well as tests. Test-order bugs are one class; file-order and parallelism bugs are another. Vitest has --sequence.shuffle.files, and --no-file-parallelism runs files one at a time. If the hunt is clean but CI still flakes, run the hunt with file shuffling on and then once with parallelism off. A flake that disappears when files run serially is shared external state, such as a port, a temp directory or a database.

Nightly hunt in CI. Run the hunt against the whole suite on a schedule with a larger seed range, upload flake-report.json as an artefact, and hand the file to /fix-flake the next morning. The report already lists failing seeds per test, so the agent starts at step 2.

Other stacks. The shape is portable: a runner that can randomise order with a printed or pinnable seed, a JSON or JUnit report, a loop, and a skill that insists on before and after counts. Swap the Vitest flags and the report parsing. This one hasn’t been run for xUnit, so check what your runner offers for order randomisation before assuming it matches.

Codex and other agents. The two scripts don’t depend on Claude Code. An AGENTS.md section that says “run the hunt before touching a flaky test, and never add retries” plus the git hook gets most of the way. The allowed-tools line and the PreToolUse hook are Claude Code specific.

What goes wrong

The hunt doesn’t reproduce everything. Order dependence is common but it’s one cause. A flake from real timing, network behaviour, or a race inside the code under test will sail through thirty shuffled runs. When the report is clean, the skill stops, which is correct, but it leaves you with an unsolved flake. Capture more evidence from CI (the seed, the worker count, the log around the failure) before the next attempt.

A test that fails 100% after the shuffle isn’t flaky. It’s an order-dependent test that has been failing in CI by luck of scheduling. Same cure, but the number is a warning that the suite has been passing for the wrong reasons.

The guard has false positives and false negatives. It matches text, so a retry configured in a shared helper, or a waitFor with a generous timeout, slides past. It also can’t tell whether the reset the agent added is the right one. Read the diff. The harness makes the agent’s claim checkable; it doesn’t replace the reviewer.

Fixing the cause can mean changing production code. Sometimes the shared state is the bug: a module-level cache with no way to reset it. The agent may propose adding resetRates() to the source file. That’s usually fine, but a test-only export in production code is a decision for you, not a detail.

Don’t bother with any of this for a one-off failure you can read your way to in two minutes, or for a suite that’s red every time. The harness pays off on the failures that appear one run in twenty, where reading the code gets you nowhere.

First 15 minutes

Pick the test your team re-runs most often. Copy flake-hunt.py into scripts/ and run it against that file with --runs 30. If seeds fail, replay the first one and find the test that leaves state behind. If none fail, try again with --sequence.shuffle.files added to the Vitest arguments.

Then add the skill and the guard, and the next time somebody says a test is flaky, run /fix-flake on it and read the before and after counts before you read the diff.