The bug your agent's tests can't see: let fast-check write the reproduction

An agent writes allocate(totalPence, weights), the function that splits a bill, a refund or a fee across several parties. It also writes three tests: an even split, a 1:3 split, a single share. All green. The function is wrong, and it’s wrong in the way money code is usually wrong: on some totals the shares add up to one penny more than the total.

Example tests can’t catch this, because the person (or agent) choosing the examples picks the ones the implementation already handles. A property test picks inputs by machine instead. This piece builds one with fast-check and Vitest, shows the failure it produces, and then uses that failure as the bug report for the fix. The technique needs nothing from the agent except the ability to read files and run npx vitest, so it works with any of them.

The function and the tests that pass

Here’s what an agent plausibly writes for “split an amount in pence across weights”:

// src/allocate.ts
/** Split an amount in pence across weights. Shares must add up to the total. */
export function allocate(totalPence: number, weights: number[]): number[] {
  const weightSum = weights.reduce((a, b) => a + b, 0);
  return weights.map((w) => Math.round((totalPence * w) / weightSum));
}

And the tests that come with it:

// src/allocate.test.ts
import { describe, expect, it } from "vitest";
import { allocate } from "./allocate";

describe("allocate (example tests)", () => {
  it("splits evenly", () => expect(allocate(100, [1, 1])).toEqual([50, 50]));
  it("splits by weight", () => expect(allocate(100, [1, 3])).toEqual([25, 75]));
  it("handles a single share", () => expect(allocate(999, [5])).toEqual([999]));
});

Each test is correct. None of them is a test of the sentence in the doc comment. The comment says the shares must add up to the total, and nothing checks that.

Write the property from the comment

A property is a statement that should hold for every valid input. Take it from the doc comment, the spec or the ticket, not from the implementation. Install the library:

npm install --save-dev fast-check

Then state the invariant. fc.property takes generators first and a predicate last, and fc.assert runs it against many generated inputs:

// src/allocate.property.test.ts
import fc from "fast-check";
import { describe, it } from "vitest";
import { allocate } from "./allocate";

const total = fc.integer({ min: 0, max: 1_000_000 });
const weights = fc.array(fc.integer({ min: 1, max: 100 }), { minLength: 1, maxLength: 8 });

describe("allocate (properties)", () => {
  it("shares add up to the total", () => {
    fc.assert(
      fc.property(total, weights, (t, w) => {
        const sum = allocate(t, w).reduce((a, b) => a + b, 0);
        return sum === t;
      }),
    );
  });
  it("returns one share per weight", () => {
    fc.assert(fc.property(total, weights, (t, w) => allocate(t, w).length === w.length));
  });
});

The generators matter more than the predicate. Keep them to the domain the function promises to handle: here, non-negative whole pence and positive integer weights. If a generator is wider than the contract, the failures are noise. If it’s narrower, bugs hide.

Run it and read the failure

npx vitest run

The output, trimmed:

 ❯ src/allocate.property.test.ts (2 tests | 1 failed)
   ❯ allocate (properties) (2)
     × shares add up to the total

Error: Property failed after 3 tests
{ seed: 1997849727, path: "2:1:0:2:1:3:2:3:7:1", endOnFailure: true }
Counterexample: [1,[48,48]]
Shrunk 9 time(s)

Three lines carry everything. The counterexample is one penny split across two equal weights. Each exact share is 0.5 pence, Math.round takes 0.5 up to 1, and the function returns [1, 1] for a total of 1. Two pence out of one. The generator started with a bigger failing input and fast-check shrank it nine times, which is why the report is small enough to read at a glance. (Your seed will differ, and so may the exact shrunk values; the shape of the failure won’t.)

The seed and path pair is the part to keep. Passing them back as the third argument to fc.assert replays that exact failing case first, every time, with no randomness:

it("replays the recorded failure", () => {
  fc.assert(
    fc.property(total, weights, (t, w) => allocate(t, w).reduce((a, b) => a + b, 0) === t),
    { seed: 1997849727, path: "2:1:0:2:1:3:2:3:7:1" },
  );
});

Run against the original allocate, that test fails on the first try with the same counterexample and “Shrunk 0 time(s)”. It’s a deterministic reproduction you can paste into a ticket or a prompt.

Hand the agent the report, not the diagnosis

The failure tells you what’s wrong. Resist telling the agent how to fix it, because the first fix an agent reaches for is often a patch on the symptom (clamp the last share to whatever is left), and that moves the rounding error onto one party. Give it the evidence and the constraints instead:

src/allocate.property.test.ts fails: allocate(1, [48, 48]) returns [1, 1],
which sums to 2. The doc comment says shares must add up to the total.
Replay it with: npx vitest run -t "replays the recorded failure".

Fix allocate so that every property in that file passes. Do not edit the
property tests. Keep each share within one penny of its exact value, and
don't favour the last weight. Show me the full test output when done.

Two clauses carry the weight. “Do not edit the property tests” stops the agent from loosening the invariant until it passes. “Within one penny of its exact value” states a fairness requirement the sum property doesn’t capture, which is worth adding as a property of its own so that a lazy fix can’t satisfy one invariant by breaking another:

it("each share is within one penny of its exact value", () => {
  fc.assert(
    fc.property(total, weights, (t, w) => {
      const sum = w.reduce((a, b) => a + b, 0);
      return allocate(t, w).every((s, i) => Math.abs(s - (t * w[i]) / sum) < 1);
    }),
  );
});

it("never returns a negative share", () => {
  fc.assert(fc.property(total, weights, (t, w) => allocate(t, w).every((s) => s >= 0)));
});

The fix, and what the run looks like after

The standard answer is the largest remainder method: floor every share, then hand the leftover pence to the shares with the biggest fractional parts. This version passes every property:

export function allocate(totalPence: number, weights: number[]): number[] {
  const weightSum = weights.reduce((a, b) => a + b, 0);
  const exact = weights.map((w) => (totalPence * w) / weightSum);
  const shares = exact.map(Math.floor);
  let left = totalPence - shares.reduce((a, b) => a + b, 0);
  const byRemainder = exact
    .map((x, i) => ({ i, rem: x - Math.floor(x) }))
    .sort((a, b) => b.rem - a.rem || a.i - b.i);
  for (const { i } of byRemainder) {
    if (left-- <= 0) break;
    shares[i] += 1;
  }
  return shares;
}

 Test Files  2 passed (2)
      Tests  8 passed (8)

The three original example tests still pass and the replay test stays in the file. Keep it. Once the bug is fixed, the recorded seed becomes a permanent regression test for the exact input that broke, even if a later change makes the generator produce different values.

Choosing properties that earn their place

The hard part is not fast-check’s API. It’s finding statements worth asserting. A few that work on code agents write all the time:

• Round trips: parse(format(x)) equals x, or deserialize(serialize(x)) equals x. Agents get escaping and locale wrong here.
• Conservation: totals, counts and sums before and after (as above). Anywhere money, stock or quota is split or merged.
• Idempotence: normalising, sorting or de-duplicating twice gives the same answer as once.
• Agreement with a slow, obviously correct version: a naive double loop beside the optimised one the agent wrote. This is the strongest of the four, and it’s the same idea as differential testing for migrations.
• Ordering and bounds: output length, ranges, monotonicity, “never negative”.

Ask the agent to propose properties too, but treat its list as a draft. A property that merely restates the implementation (allocate(t, w) equals the formula allocate uses) can never fail. Reject any property you couldn’t have written from the spec alone.

What goes wrong

Properties catch the cases the generator reaches. Generators with a narrow range will happily pass broken code, so check what yours actually produces; fast-check can print sampled values with fc.sample(arbitrary, 10).

Floating point bites. The fix above divides floats, so a total times a weight that lands a hair under a whole number floors one pence low and the remainder loop corrects it. That works here because the loop distributes whatever is left, but it’s the reason the sum property matters more than any single-share check. For real money code, integer arithmetic with a documented rounding rule is safer than anything involving /.

Failures vary between runs. A random seed means a flaky-looking test the first time it finds a rare bug, and a green build on the commit before. That’s the point, but it surprises teams. Log the seed (fast-check already prints it on failure), and promote every recorded failure to a fixed { seed, path } test or an explicit example case so the build is deterministic afterwards.

Don’t bother for glue code, UI wiring or anything where the “right answer” is a judgement call. Properties need an invariant you can state in one sentence.

First 15 minutes

Pick a pure function an agent wrote this month that has a sentence in its doc comment with “always”, “never” or “must” in it. Run npm install --save-dev fast-check, turn that sentence into one fc.property, and run it. If it passes, add a second property for a different sentence. If it fails, copy the counterexample and the seed into the next prompt.