tokenise by Rosvetic
Development How-to

Run your README's code blocks as Vitest tests so an agent can't leave the docs lying

Agents change code faster than anyone rereads the docs. A 40-line Vitest file turns every TypeScript example in a README into a test, so stale docs fail the build. Fifteen minutes to set up.

8 min read

A clay open book on a dark surface with a small clay stamp pressing on one page, one block glowing teal

An agent renames a function, updates every caller, fixes the tests, and ships a clean diff. The README still shows the old name. Nothing failed, because nothing reads the README except the next developer, who pastes the example, gets a compile error and loses ten minutes deciding whether to trust anything else on the page.

Docs rot because they’re the one artefact with no test. Fix that and the problem mostly goes away, whether a human or an agent made the change. The trick is small: pull every TypeScript block out of the README, turn the comments you already write (// => value) into assertions, and run the lot under Vitest. When an example lies, the build fails and names the README line.

The technique needs a Markdown file, a test runner and a shell. Any coding agent that can run npx vitest run can use it, and so can you without one. The setup below is about 40 lines, and everything printed in this piece is real output from it.

A README that can be wrong

The demo library is a tiny money module. Here’s the code under test, in src/money.ts:

export type Money = { amount: number; currency: string };

export function parse(input: string): Money {
  const m = /^([A-Z]{3}) (\d+(?:\.\d{1,2})?)$/.exec(input.trim());
  if (!m) throw new Error(`Cannot parse money: ${input}`);
  return { amount: Math.round(Number(m[2]) * 100), currency: m[1] };
}

export function format(value: Money): string {
  return `${value.currency} ${(value.amount / 100).toFixed(2)}`;
}

export function add(a: Money, b: Money): Money {
  if (a.currency !== b.currency) throw new Error("Currency mismatch");
  return { amount: a.amount + b.amount, currency: a.currency };
}

And the README, with examples written the way people already write them: a statement, then a comment saying what it returns or does.

# money

```ts
import { parse, format, add } from "./src/money";

parse("GBP 12.50"); // => { amount: 1250, currency: "GBP" }
format(add(parse("GBP 1.10"), parse("GBP 2.20"))); // => "GBP 3.30"
```

Mixing currencies throws:

```ts
import { parse, add } from "./src/money";

add(parse("GBP 1"), parse("USD 1")); // throws /mismatch/
```

Rounding is banker's style:

```ts
import { parse } from "./src/money";

parse("GBP 0.005"); // => { amount: 1, currency: "GBP" }
```

```ts skip
// pseudo-code, not run
const total = sum(items);
```

Look at the third block. The sentence above it claims banker’s rounding, and the example claims GBP 0.005 parses to one penny. The regex allows at most two decimal places, so that input throws. This is exactly what drift looks like: the code changed (or was never what the docs said), and nobody noticed.

The test file

Install the two dev dependencies and add test/readme.test.ts:

npm i -D vitest typescript

import { readFileSync, mkdirSync, writeFileSync } from "node:fs";
import { resolve } from "node:path";
import { describe, it } from "vitest";

const root = resolve(__dirname, "..");
const readme = readFileSync(resolve(root, "README.md"), "utf8");
const fence = /^```ts([^\n]*)\n([\s\S]*?)^```/gm;

const blocks = [...readme.matchAll(fence)]
  .filter((m) => !m[1].includes("skip"))
  .map((m, i) => ({
    name: `README block ${i + 1}`,
    line: readme.slice(0, m.index).split("\n").length,
    code: m[2],
  }));

// `expr; // => value`     becomes  expect(expr).toEqual(value)
// `expr; // throws /re/`  becomes  expect(() => expr).toThrow(/re/)
function rewrite(code: string): string {
  return (
    `import { expect } from "vitest";\n` +
    code
      .replace(/^(.+?);\s*\/\/ => (.+)$/gm, "expect($1).toEqual($2);")
      .replace(/^(.+?);\s*\/\/ throws (.+)$/gm, "expect(() => $1).toThrow($2);")
  ).replace(/from "\.\/src\//g, `from "${root}/src/`);
}

describe("README examples", () => {
  mkdirSync(resolve(root, ".readme-tests"), { recursive: true });
  for (const [i, b] of blocks.entries()) {
    it(`${b.name} (README.md:${b.line})`, async () => {
      const file = resolve(root, `.readme-tests/block-${i + 1}.ts`);
      writeFileSync(file, rewrite(b.code));
      await import(/* @vite-ignore */ file);
    });
  }
});

Add .readme-tests to .gitignore, since the generated files are throwaway.

Here’s what each part does. The regex finds fenced ts blocks, and any block whose info string contains skip is left out, which is how pseudo-code and fragments opt out. The line number of each fence goes into the test name, so a failure points at the README and not at a generated file.

rewrite does the useful work. A trailing // => value becomes expect(expr).toEqual(value), and // throws /re/ becomes a toThrow check. Relative imports like ./src/money are made absolute so the generated file, which lives one directory down, still resolves them.

Each block is written to .readme-tests/ and loaded with a dynamic import. Vitest transforms it like any other module, so TypeScript works without a build step. Vitest only picks up files with .test. or .spec. in the name by default, according to its guide , which is why the generated files don’t run twice. The default include pattern is **/*.{test,spec}.?(c|m)[jt]s?(x), per the config reference .

Watch it catch a lie

Run it:

npx vitest run

 ❯ test/readme.test.ts (3 tests | 1 failed) 43ms
   ❯ README examples (3)
     × README block 3 (README.md:20) 13ms

 FAIL  test/readme.test.ts > README examples > README block 3 (README.md:20)
Error: Cannot parse money: GBP 0.005
 ❯ parse src/money.ts:5:17

Blocks 1 and 2 pass. Block 3 fails on the very claim that was wrong, with the README line in the name and the stack pointing into the library. The right fix here is the docs, not the code: the module deliberately rejects a third decimal place. Change the prose and the example:

Amounts take at most two decimal places:

```ts
import { parse } from "./src/money";

parse("GBP 0.005"); // throws /Cannot parse/
```

Run again and Tests 3 passed (3).

Now the other kind of drift, a wrong value and not a crash. Change the first example’s 1250 to 125 and the failure reads:

 FAIL  test/readme.test.ts > README examples > README block 1 (README.md:3)
AssertionError: expected { amount: 1250, currency: 'GBP' } to deeply equal { amount: 125, currency: 'GBP' }

- Expected
+ Received

  {
-   "amount": 125,
+   "amount": 1250,
    "currency": "GBP",
  }

 ❯ .readme-tests/block-1.ts:4:28
      2| import { parse, format, add } from "/tmp/.../src/money";
      4| expect(parse("GBP 12.50")).toEqual({ amount: 125, currency: "GBP" });

That diff is what makes this pleasant to hand to an agent. It shows the README line, the expected value and the received value, which is everything needed to decide which side is wrong.

What the harness can’t see

A block with no // => or // throws comment is only checked for not crashing. Add this to the README:

import { format, parse } from "./src/money";

console.log(format(parse("GBP 2"))); // prints GBP 2.00

The run reports Tests 4 passed (4), and the // prints comment asserted nothing. A console.log example only proves the code runs. If you use that style, either switch to the // => form on an expression, or extend rewrite with a third rule that captures console.log and compares it. Pick one convention for the whole README and write it at the top of the test file, because a mixed style is how unchecked examples creep back in.

Three more limits worth knowing before relying on this:

  • The harness checks examples, not prose. “Rounding is banker’s style” was caught only because the example under it contradicted the sentence. A claim with no code beside it is untested.
  • Shell, JSON and YAML blocks are ignored. Install commands and config snippets still rot. The same idea extends to them (run each bash block in a temp directory), but that needs more care about side effects, so don’t do it casually.
  • The regex rewrite handles one statement per line. A multi-line call with a trailing // => on the last line won’t be matched. Put the expected value on a single-line expression, or assign to a variable first and assert on the variable.

If your README examples are long, stateful tutorials, this is the wrong tool. It suits reference snippets: call this, get that.

Let the agent repair what fails

Once docs are tests, the agent loop is the ordinary one: run the tests, read the failures, fix them. The one thing to control is which side the agent changes. Left alone, an agent asked to make a failing doc test pass will sometimes edit the library to match a stale example, which turns a docs fix into a behaviour change.

A prompt that keeps it on the right side:

The README example tests fail. For each failure, decide whether the README
or the code is wrong. Default to changing the README. Only change src/ if
the example shows the intended behaviour and the code contradicts it, and
in that case stop and tell me what you'd change instead of doing it.
Don't edit test/readme.test.ts. Re-run `npx vitest run` and report the result.

The last instruction matters. A harness the agent is allowed to edit can be made green by weakening it, for example by adding skip to the failing fence. Review any diff that adds skip the way you’d review one that adds it.skip.

It also helps to put one line in your agent instructions file, whatever the tool calls it: “After changing a public function, run npx vitest run; README failures mean the docs need updating in the same change.” That moves the check from a later cleanup into the change that causes the drift.

Make it yours

For a library published to npm, add the run to CI so a pull request that breaks an example can’t merge. It’s the same command as locally, so any CI system works.

For a monorepo, read every README.md under packages/ and prefix the test name with the package path. Keep skip for anything that needs a network or a database.

For examples that import the package by name (import { parse } from "money"), swap the import rewrite for a Vitest alias that maps the package name to src/index.ts, so the README reads exactly as a user would copy it.

First 15 minutes

Copy the test file above into your repo and point root at the right directory. Run npx vitest run and read the failures: in a README that’s a year old, expect at least one, and treat it as the first real finding. Fix the docs, mark pseudo-code with ts skip, and commit. From then on, a pull request that makes a README example untrue fails before anyone has to notice.