Let git find the commit that broke it: a reproduction test, git bisect run and an agent for the fix

A customer reports that a £10.05 item with 10% off is being charged £9.04 instead of £9.05. It worked in the last release. Between that release and today sit nine commits, one of which is a work-in-progress that doesn’t compile. Ask an agent “what changed?” and it reads the diffs, picks the one that looks most like pricing and tells you so with great confidence. Sometimes it’s right.

Git can answer this without opinions. git bisect run takes a script that says good or bad, checks out commits in a binary search and names the first bad one. The agent’s job shrinks to two tasks it’s good at: writing the script’s test from the bug report, and fixing the commit git points at.

This works with any agent that can read files and run a shell, or with no agent at all. The test, the script and the bisect are plain git and a test runner. The example below is TypeScript with Vitest 5, and the pattern is identical with xUnit and dotnet test.

The repository

Nine commits, a small pricing module, and a regression planted at commit seven. Here’s the module after that commit:

export interface Line { unitPriceCents: number; quantity: number }

export function subtotal(lines: Line[]): number {
  return lines.reduce((sum, l) => sum + l.unitPriceCents * l.quantity, 0);
}

// Work in integer arithmetic to avoid float drift
export function applyDiscount(cents: number, percent: number): number {
  return Math.floor((cents * (100 - percent)) / 100);
}

export function total(lines: Line[], discountPercent = 0): number {
  return applyDiscount(subtotal(lines), discountPercent);
}

Before that commit, applyDiscount used Math.round. The existing tests (2000 pence unchanged, 2000 less 10% is 1800) pass either way, which is the usual shape of a regression: nothing in the suite pins the behaviour that moved.

The history, oldest first:

5a95bc6 Add pricing module
e36102f Add currency formatter
8e46c12 Add VAT helper
7bee860 Add percent formatter
ec94b6e WIP: vat rate param (does not compile)
5691548 Fix vat helper
b65182d Use integer arithmetic for discounts
0fb355d Extract currency symbol
c3d248b Add readme

Step 1: turn the bug report into a failing test

This is the part to hand to the agent, and the part to check by hand. Give it the report and ask for one test, not a fix:

Customers report that a £10.05 item with 10% off is charged £9.04 instead of £9.05.
Write one Vitest test, in a new file named bisect.repro.test.ts in the repo root,
that fails on the current code for this reason and would pass once it's fixed.
Import only from ./src/pricing. Do not change any existing file. Do not fix the bug.
Run it and show me the failure message.

The constraints matter. A new file means old commits don’t need it to exist. Importing only the public function means the test runs against every revision that has that function. “Do not fix the bug” keeps the test honest: you want it red now.

What comes back:

import { expect, test } from "vitest";
import { total } from "./src/pricing";

// Bug report: a £10.05 item with 10% off is charged £9.04. Should be £9.05.
test("10% off 1005p rounds to nearest penny", () => {
  expect(total([{ unitPriceCents: 1005, quantity: 1 }], 10)).toBe(905);
});

Check two things before moving on. The test must fail for the reason in the report (expected 905, received 904), not because of a typo or a missing import. And it must pass on a commit you know to be good. Most bisects that go wrong at this stage go wrong because the test is red everywhere.

The file stays untracked. Git leaves untracked files alone when it checks out other commits, so the same test file is sitting there at every step. Run git status and you’ll see it listed under ??. Don’t commit it yet.

Step 2: write a probe that can tell “bad” from “can’t tell”

The obvious script is npx vitest run bisect.repro.test.ts. It has a flaw. Vitest exits 1 when a test fails, and also exits 1 when the file can’t be imported at all. Commit ec94b6e has a syntax error, so the obvious script would call it bad, and git would blame the wrong commit or wander off.

Git’s answer is exit code 125. According to the git documentation, a script run by git bisect run should exit 0 if the source is good or old, and between 1 and 127 inclusive, except 125, if it’s bad or new. 125 means the revision can’t be tested, and git skips it. Any other code aborts the whole bisect.

So the probe has to look at what Vitest reports, and not just its exit status. Vitest’s JSON reporter (--reporter=json --outputFile=<path>) writes counts that make the distinction:

#!/bin/sh
# exit 0 = good, 1 = bad, 125 = cannot test this commit
out=$(mktemp)
npx vitest run bisect.repro.test.ts --reporter=json --outputFile="$out" >/dev/null 2>&1
node -e '
const r = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"));
if (r.numTotalTests === 0) process.exit(125);      // suite never ran: build or import error
process.exit(r.numFailedTests > 0 ? 1 : 0);
' "$out"
code=$?
rm -f "$out"
exit $code

Save it outside the working tree (here ../probe.sh) and chmod +x it. If the script lived inside the repo, old commits wouldn’t have it, and a bisect that checks out a commit without its own probe fails in confusing ways.

Test the probe by hand on three revisions before trusting it: the known-good one, the one that doesn’t compile and the tip.

git checkout -q 5a95bc6 && ../probe.sh; echo "5a95bc6 exit=$?"
git checkout -q ec94b6e && ../probe.sh; echo "ec94b6e exit=$?"
git checkout -q main    && ../probe.sh; echo "main exit=$?"

Run it and it prints:

5a95bc6 exit=0
ec94b6e exit=125
main exit=1

Good, untestable, bad. That’s the probe doing its job.

Step 3: let git search

Mark the tip bad and the known-good commit good, then hand over the script:

git bisect start HEAD 5a95bc6 --
git bisect run ../probe.sh

The output, trimmed of blank lines:

running '../probe.sh'
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[b65182d4bf97c8833d5f8a597bdde676f79780c0] Use integer arithmetic for discounts
running '../probe.sh'
Bisecting: 2 revisions left to test after this (roughly 2 steps)
[7bee8601af1625635419915718fff6e363dc6522] Add percent formatter
running '../probe.sh'
Bisecting: 0 revisions left to test after this (roughly 1 step)
[5691548942c6205ae609fe87aa437c4c0e4a92b1] Fix vat helper
running '../probe.sh'
b65182d4bf97c8833d5f8a597bdde676f79780c0 is the first bad commit
 src/pricing.ts | 3 ++-
bisect found first bad commit

Four test runs for nine commits. Note what happened in the middle: git landed on the uncompilable commit, the probe returned 125, and git skipped it and tested its neighbours. The log records it:

git bisect skip ec94b6e9e130fd301cda9490631ca60df5d24115
git bisect bad b65182d4bf97c8833d5f8a597bdde676f79780c0
git bisect good 7bee8601af1625635419915718fff6e363dc6522
git bisect good 5691548942c6205ae609fe87aa437c4c0e4a92b1
# first bad commit: [b65182d4bf97c8833d5f8a597bdde676f79780c0] Use integer arithmetic for discounts

Then git bisect reset to return to where you started. Forget that and the next commit you make lands on a detached HEAD in the middle of history.

Step 4: give the agent the commit and the test

Now the agent has something solid. The prompt:

git bisect run identified b65182d as the first commit where bisect.repro.test.ts fails.
Run `git show b65182d`, explain in two sentences what behaviour it changed, then fix
src/pricing.ts so bisect.repro.test.ts passes without breaking the other tests.
Keep the intent of that commit (integer arithmetic). Run the whole suite and show the output.

The last sentence of the prompt does real work. The commit message says “avoid float drift”, which is a goal worth keeping, and an agent that only sees a failing test may well revert the commit wholesale. Here a fix that keeps the integer arithmetic and restores the rounding would be:

export function applyDiscount(cents: number, percent: number): number {
  return Math.round((cents * (100 - percent)) / 100);
}

1005 times 90 is 90450, divided by 100 is 904.5, and Math.round gives 905. Move the repro test into src/pricing.test.ts alongside the others and commit it with the fix. It’s now a regression test, and the next person who “simplifies” this function finds out in seconds.

Make it yours

Merge-heavy histories. If the first-bad answer lands on a merge or on a branch with broken intermediate commits, start with git bisect start --first-parent. Per the git docs, it follows only the first parent at merge commits, so the merge itself is identified as the introduction of the bug and the branch’s ancestors are ignored. That’s the right resolution when a feature branch was green at the merge but messy inside.

Other test runners. The probe’s only job is to map the runner’s result to 0, 1 or 125. For xUnit (not run here, so treat it as a sketch), a failed dotnet build should be a 125 and a failing dotnet test --filter "FullyQualifiedName~Repro" a 1. Check the build first, in its own step with || exit 125, as the git docs’ example does for make.

Slowdowns. Bisect doesn’t have to be about bugs. git bisect start --term-old fast --term-new slow lets you search for the commit that made something slower, with a probe that times a benchmark and compares it with a threshold. Treat the threshold carefully, since noisy timings turn a binary search into a coin toss.

Unattended. The whole loop is scriptable, so a CI job can bisect on a red nightly build and attach the culprit commit to the alert. The agent can then be given the commit and the test in a fresh session, as in step 4.

What goes wrong

A flaky test. Bisect assumes the answer for a commit never changes. One false “good” sends the search into the wrong half of history, and it ends at a confident wrong commit with no warning. Run the probe five times on the tip and five times on the good commit before you start. If you can’t get the same answer five times, fix the flakiness first or this technique isn’t ready for that bug.

A test that depends on later code. The repro test imports only what existed at the good commit. If the agent writes a test that uses a helper added in commit six, the test errors on older commits, the probe says 125, and git skips most of the range. Watch for “skipped” counts in the log: many skips mean the probe isn’t testing what you think.

Dependencies that changed. If an old commit needs a different node_modules or a different SDK, the probe fails to build and skips everything, or worse, passes for the wrong reason. Bisect across a package upgrade only if the probe installs what each commit needs, and expect it to be slow.

A bug with no single cause. Bisect finds the first commit where the test flips. If two commits each contribute half of the bug, git names the second one. Read the diff of the named commit and the one before it.

Don’t bother when the culprit is obvious from the stack trace, when history is a single squashed commit, or when the bug exists only in production data you can’t reproduce in a test. Git can’t search for what a test can’t express.

The first 15 minutes

Take a bug from your tracker that “used to work”. Paste the report into the step 1 prompt and get a failing test that you’ve read yourself. Save the probe from step 2 outside the repo, check it on a good, a broken and a bad commit, then run git bisect start HEAD <last-release-tag> -- and git bisect run. Hand the commit it names to the agent with the prompt from step 4.