Pin a legacy function before an agent refactors it: record its outputs, then let StrykerJS grade the safety net
A recorder, a generated test file and a mutation run give you a safety net you can measure, not just feel good about. About 20 minutes to set up.
A function nobody wants to touch decides what customers pay, and an agent has just been asked to tidy it. The tests are thin, the original author left in 2021, and the diff that comes back reads fine. That’s the trap. A refactor that reads fine and changes a rounding rule is the most expensive kind, because nothing fails until an invoice does.
The usual fix is to write tests first, and the usual way to do that with an agent is to say “write unit tests for this module”. The agent reads the code, decides what it should do, and writes expectations from that reading. Where the code is wrong, the tests agree with the wrong reading or, worse, disagree with the code and get “fixed” until they pass.
Characterization tests flip the order. They don’t say what the function should do. They record what it does now, for a deliberately chosen set of inputs, and fail if anything changes. The agent’s job shrinks to something it’s good at: choosing inputs by reading the branches. The expected values come from running the real code, so no model gets to guess them.
This Playbook builds that in three parts: a recorder that writes the expected values, a generated Vitest file that replays them, and a StrykerJS run that measures whether the inputs were good enough. Then it shows the agent prompts and the Claude Code permission rule that keeps the agent out of the source while the net goes up.
The legacy function
Here’s a stand-in for the kind of code this is for. It’s small enough to read in one go, and it has the usual layering of special cases.
export interface Order {
weightKg: number;
country: string;
subtotal: number;
express?: boolean;
coupon?: string;
}
export function shippingCost(o: Order): number {
let cost = 0;
if (o.weightKg <= 0) return 0;
if (o.weightKg <= 1) cost = 4.5;
else if (o.weightKg <= 5) cost = 4.5 + (o.weightKg - 1) * 1.2;
else cost = 9.3 + Math.ceil(o.weightKg - 5) * 2;
if (o.country !== "GB") cost = cost * 1.75;
if (o.country === "IE") cost = cost - 1;
if (o.express) cost += 6;
if (o.subtotal >= 100 && !o.express) cost = 0;
if (o.coupon === "FREESHIP") cost = 0;
if (o.coupon === "HALF") cost = cost / 2;
return Math.round(cost * 100) / 100;
}
Every line is a decision that somebody once made for a reason. Order matters: the Irish discount comes after the export multiplier, the free-shipping threshold ignores express orders, and the coupons run last. A refactor that groups “country rules” together, which is a perfectly natural thing for an agent to do, could easily reorder them.
Step 1: set up the scratch project
Any Node project works. The only requirement is a test runner and Stryker. Install them as dev dependencies, with Vitest pinned to major version 3 (the reason is in the failure section below):
npm i -D vitest@3 @stryker-mutator/core @stryker-mutator/vitest-runner typescript tsx
Stryker’s Vitest runner expects you to install Vitest yourself, per its documentation. Add a config that points Stryker at the one file you’re about to pin:
{
"testRunner": "vitest",
"mutate": ["src/shipping.ts"],
"reporters": ["clear-text"]
}
The runner always uses per-test coverage analysis, so there’s no coverageAnalysis key to set.
Step 2: write the recorder, not the expectations
This is the part to hand to the agent, and the part to constrain. The recorder is a script that builds a grid of inputs, calls the real function, and writes input and output pairs to a JSON file. The agent chooses the grid. It never types an expected value.
import { writeFileSync } from "node:fs";
import { shippingCost, type Order } from "../src/shipping";
const weights = [-1, 0, 0.5, 1, 1.01, 5, 5.01, 7.5, 20];
const countries = ["GB", "IE", "US"];
const subtotals = [0, 99.99, 100];
const express = [false, true];
const coupons = [undefined, "FREESHIP", "HALF", "freeship"];
const rows: { input: Order; output: number }[] = [];
for (const weightKg of weights)
for (const country of countries)
for (const subtotal of subtotals)
for (const e of express)
for (const coupon of coupons) {
const input: Order = { weightKg, country, subtotal, express: e, coupon };
rows.push({ input, output: shippingCost(input) });
}
writeFileSync("test/golden.json", JSON.stringify(rows, null, 1));
console.log(`${rows.length} cases recorded`);
Look at how the grid was built. Every number sits on or next to a boundary in the code: 0, 1 and 5 are the thresholds, 1.01 and 5.01 are just past them, 99.99 and 100 straddle the free-shipping line, and “freeship” in lower case probes whether the coupon match is case-sensitive. Sampling at the boundaries is the whole craft here. Random inputs would mostly land in the middle of branches and miss the edges where refactors break things.
Run it:
npx tsx tools/probe.ts
648 cases recorded
That’s 9 weights times 3 countries times 3 subtotals times 2 express flags times 4 coupons.
Step 3: replay the recording as a test
The test file is generic. It reads the golden file and asserts that the function still returns what it returned when the recording was made.
import { describe, expect, it } from "vitest";
import golden from "./golden.json";
import { shippingCost } from "../src/shipping";
describe("shippingCost (pins current behaviour, not intended behaviour)", () => {
it.each(golden.map((row) => [JSON.stringify(row.input), row] as const))(
"%s",
(_name, { input, output }) => {
expect(shippingCost(input)).toBe(output);
},
);
});
The describe title says what these tests are. That matters six months on, when someone reads a failure and wonders whether the new value or the old one is correct. The title tells them the suite has no opinion on that.
npx vitest run
Test Files 1 passed (1)
Tests 648 passed (648)
Passing on the first run proves nothing, since the expectations came from the code. What matters is the next two steps.
Step 4: break it on purpose
Before trusting the net, check it catches the mistake it exists for. Swap the two lines that handle country, as a tidy-up might, so the Irish discount lands before the multiplier:
if (o.country === "IE") cost = cost - 1;
if (o.country !== "GB") cost = cost * 1.75;
Run the suite again:
AssertionError: expected 6.13 to be 6.88 // Object.is equality
Tests 105 failed | 543 passed (648)
The failing input is a 0.5kg order to Ireland: 4.5 less 1, times 1.75, gives 6.125 and rounds to 6.13, where the original gave 4.5 times 1.75 less 1, which is 6.875 and rounds to 6.88. A fifty-pence difference on one parcel is exactly what a code reviewer skims past. Put the lines back and the suite returns to 648 passing.
Step 5: let Stryker grade the grid
A hand-picked grid is a guess about which inputs matter. Mutation testing checks the guess. StrykerJS makes small changes to the source (flips <= to <, swaps + for -, replaces a condition with true) and reruns the tests against each change. A mutant that makes a test fail is killed. One that doesn’t is a place where your suite can’t see a change.
npx stryker run
On the full grid it prints:
-------------|------------------|----------|-----------|------------|----------|----------|
| % Mutation score | | | | | |
File | total | covered | # killed | # timeout | # survived | # no cov | # errors |
-------------|--------|---------|----------|-----------|------------|----------|----------|
shipping.ts | 96.00 | 96.00 | 48 | 0 | 2 | 0 | 0 |
Two survivors out of 50. Stryker lists them:
[Survived] EqualityOperator
src/shipping.ts:12:7
- if (o.weightKg <= 1) cost = 4.5;
+ if (o.weightKg < 1) cost = 4.5;
[Survived] EqualityOperator
src/shipping.ts:13:12
- else if (o.weightKg <= 5) cost = 4.5 + (o.weightKg - 1) * 1.2;
+ else if (o.weightKg < 5) cost = 4.5 + (o.weightKg - 1) * 1.2;
Read them rather than reaching for more inputs. At exactly 1kg, the first branch gives 4.5, and the second branch would give 4.5 + 0 * 1.2, which is also 4.5. At exactly 5kg, the second branch gives 4.5 + 4.8 = 9.3 and the third gives 9.3 + 0. These are equivalent mutants: changes that produce the same output for every input, so no test can ever kill them. Stryker’s documentation says as much and says it can’t find them automatically. Note them and move on.
What a thin grid looks like
The point of the mutation run is that it tells you when the grid is weak. Cut the grid down to what a quick first pass might produce (weights 0.5 and 3, subtotals 0 and 120, same countries, flags and coupons) and the same command reports:
shipping.ts | 82.00 | 87.23 | 41 | 0 | 6 | 3 | 0 |
Six survived, three had no coverage at all. The survivors include the heavy-parcel arithmetic on line 14, because no input weighed more than 5kg:
- else cost = 9.3 + Math.ceil(o.weightKg - 5) * 2;
+ else cost = 9.3 - Math.ceil(o.weightKg - 5) * 2;
That is a mutation report an agent can act on. It names a line, a change and the input class that’s missing.
Step 6: the agent prompts
Now the agent does the work it’s suited to. Two prompts, in two stages, in a fresh session so the first doesn’t colour the second.
The first writes the recorder:
Read src/shipping.ts. Write tools/probe.ts that calls shippingCost over a
grid of inputs and writes {input, output} rows to test/golden.json.
Rules:
- Choose every value from the code: each threshold, one value either side
of it, and one value in the middle of each branch.
- Include inputs that look invalid (negative weight, lower-case coupon).
- Never write an expected output by hand. The outputs come from calling
the function.
- Do not edit anything under src/.
The second closes the gap the mutation report found:
Run `npx stryker run`. For each surviving or no-coverage mutant, decide
whether it is equivalent (cannot change any output) or whether an input
is missing from the grid in tools/probe.ts. For missing inputs, extend the
grid and re-run the probe and Stryker. For equivalent mutants, list them
in a comment at the top of tools/probe.ts with one line of reasoning each.
Do not change src/ and do not hand-edit test/golden.json.
Then a third pass, which is the one that pays for itself later:
Read test/golden.json and src/shipping.ts. List every recorded behaviour a
product owner would probably call a bug, with the input and the recorded
output. Do not fix anything. Write the list to docs/shipping-quirks.md.
From this grid, the list has real entries. A negative weight returns 0 even with express shipping, so the 6 express fee silently disappears. A coupon spelled “freeship” gives 4.5 where “FREESHIP” gives 0, so the coupon match is case-sensitive. A 1kg parcel to Ireland costs 6.88 and to the US 7.88, which means the Irish discount really is a flat pound off after the multiplier. Whether any of that is a bug is a business call. The suite now pins all of it, so a refactor can’t change it by accident, and a deliberate fix shows up as a golden-file diff that someone has to approve.
Keep the agent out of the source
The step that goes wrong most often is the agent “helping” by editing the code under test. A failing characterization test is a signal, and an agent with write access will cheerfully resolve it by changing whichever side is easier.
Claude Code’s permission rules can take that option away for the session. In .claude/settings.json:
{
"permissions": {
"deny": ["Edit(/src/**)"]
}
}
Per the Claude Code documentation, Edit rules use gitignore-style patterns, a leading / anchors the path at the project root, and the rule covers all built-in file-editing tools. It also covers file commands Claude Code recognises in Bash, such as sed and tee, and redirections like > file. It does not cover a script that opens files itself, so a node -e one-liner or a Python snippet that writes to src/ isn’t stopped by it. For that, the docs point to the sandbox, which blocks all processes from a path at the OS level. Treat the deny rule as a guard rail and the sandbox as the wall.
Remove the rule when it’s time for the refactor itself. At that point the instruction changes: the agent may edit src/, and every change must keep npx vitest run green. If it needs a golden value changed, it stops and asks.
Make it yours
For a C# codebase, the same shape works with xUnit and Stryker.NET: a [Theory] fed by a MemberData method that reads the recorded JSON, and dotnet stryker for the grade. The recorder can be a small console project or a test that regenerates the file when an environment variable is set.
For code with side effects, record at a seam. Wrap the database call, the clock and the HTTP client behind the module’s existing boundaries, feed fixed values in, and record what comes out of the function and what it sent to the wrapped dependencies. Anything you can’t make deterministic, such as the current time, has to be pinned in the test or the golden file will fail on a clean checkout.
For a whole service, record at the API: send a fixed list of requests, store the responses, and compare. That’s coarser, and mutation testing over the whole service is slow, so run Stryker only on the module being changed.
What goes wrong
Check the toolchain versions first. With Stryker 10.0.0 and the newest Vitest at the time of writing (5.0.5), the dry run passed and the report said 0 of 50 mutants killed. With Vitest pinned at 3.2.7 the identical suite killed 48. Zero kills from a suite with 648 passing assertions points at the tooling, not the tests, so check the Stryker and Vitest pairing before blaming the grid.
A recorded suite pins bugs as faithfully as features. That’s the intent, but it means a green run says “nothing changed”, never “this is right”. Don’t let the golden file be mistaken for a spec.
Floating-point output makes golden files fragile across platforms and runtimes. The function above rounds to pennies, which hides it. If yours doesn’t, compare with toBeCloseTo and say how close.
Mutation runs are slow on big modules. Scope mutate to the files you’re about to change, and run the full grid in CI only if it stays under a few minutes.
Don’t bother with any of this for code you’re about to delete, or for code with good tests already. It’s for the function that has none and has to change on Thursday.
First 15 minutes
Pick the one function in your codebase you’d least like an agent to refactor. Paste the first prompt into a fresh session with Edit(/src/**) denied, run the probe, then run npx stryker run and read only the survivors. Once the suite is green and the survivors are explained, lift the deny rule and let the refactor begin.