Make your agent earn the right to fix a flaky test: a shuffle-seed harness and a commit guard

Fri, 23 Oct 2026 07:09:00 +0000

A test fails in CI, passes on rerun, and somebody tells the agent to fix the flaky test. The agent reads the file, sees nothing wrong, and does the thing that makes the red go away: wraps it in a retry, adds a short sleep, or raises the timeout. CI goes green. The race is still there, now hidden behind a setting that nobody will look at again.

Flaky-Tests on Tokenise

Make your agent earn the right to fix a flaky test: a shuffle-seed harness and a commit guard