tokenise by Rosvetic
DevOps Playbook

Fan one migration out across a dozen repos with claude -p, and let a script decide which results you read

A driver script, a worktree per repo and an independent gate that doesn't trust the agent's own report. About 15 minutes to set up on a migration you already have.

13 min read

A blueprint drawing of a central ring joined by spokes to four smaller rings, one spoke dashed where a link has dropped

A library you publish internally drops its callback API. Twelve services still call it. The change in each one is small, the kind an agent does well, and the tedious part is everything around it: clone, branch, install, run the prompt, read twelve diffs, work out which of the twelve quietly changed a public signature.

Run the agent twelve times by hand and you’ll read the first three diffs carefully and skim the rest. That’s the failure to design against. A script can do the cloning, the branching and the running. What it should also do is decide, mechanically, which results deserve your attention, so the careful reading goes to the repos that need it.

This Playbook builds that: a driver that gives each repo its own worktree, runs claude -p with tight limits, then checks the result with a script that doesn’t believe the agent. Output is one branch per repo and one table. The whole kit is about 110 lines of shell.

Where this sits next to /batch

Claude Code has a bundled /batch skill. According to the commands reference, it researches a codebase, splits the work into 5 to 30 independent units, shows a plan, and once approved runs one background subagent per unit in its own worktree. That’s the right tool for one large repository, with you in the loop for the plan.

It isn’t built for twelve separate repositories with different test setups, or for a run that happens while nobody’s watching. For that, the shape is simpler: one claude -p per repo, driven from outside, with the verification kept outside the agent too.

The fixture used below is four small TypeScript repos with Vitest tests. Each depends on a vendored acme-http module whose deprecated get(url, callback) is being replaced by getJson(url), which returns a promise. The repos differ on purpose:

  • billing has two plain call sites, one of them already wrapped in a hand-rolled promise.
  • search has a callback-based retry loop.
  • gateway exports a callback-style function that other services call, and its test mocks get.
  • reports was migrated already.

Those four cover the interesting outcomes: a clean change, a rewrite, a trap and nothing to do.

The prompt is a contract

The prompt lives in a file so every repo gets identical words. The rules in it are the ones the gate will enforce later, which is the point: anything you ask for but can’t check is a wish.

Migrate this repository off the deprecated callback API in `vendor/acme-http`.

The old call is `get(url, (err, body) => ...)`. The replacement is `getJson(url)`, which returns a promise that resolves with the body and rejects with the error.

Rules:
- Change only files under `src/` and `test/`. Never touch `vendor/`.
- Keep every exported function's name, parameters and return type exactly as they are. Callers outside this repo depend on them. If an exported function takes a callback, it keeps taking one.
- Behaviour must stay the same, including error handling and retries.
- If a test mocks `get`, change the mock to `getJson`. Don't delete or weaken assertions.
- Do not commit. Do not run npm install.
- Run `npx vitest run` before you finish.

If the repo has nothing to migrate, change nothing and say so. If you can't migrate something without breaking the rules, leave it alone and report it as blocked.

Two details matter. The agent is told not to commit, so the driver owns the commit and only makes one when the gate passes. And “nothing to do” and “blocked” are named as legitimate answers, which keeps the agent from inventing work to look useful.

The reply is constrained with --json-schema, so the driver gets a status it can read rather than prose:

{"type":"object","properties":{"status":{"enum":["migrated","nothing_to_do","blocked"]},"call_sites_changed":{"type":"integer"},"notes":{"type":"string"}},"required":["status","call_sites_changed","notes"]}

Per the headless docs, with --output-format json and --json-schema the validated result comes back in the structured_output field, next to total_cost_usd and the session metadata.

The per-repo runner

This is the core. It makes a worktree on a fresh branch, runs the setup command, runs the agent, runs the gate, and commits only on a pass.

#!/usr/bin/env bash
# fanout-one.sh <repo-path>   (env: BRANCH, OUT, KIT, SETUP)
set -u
repo=$(cd "$1" && pwd); name=$(basename "$repo")
wt="$OUT/worktrees/$name"; log="$OUT/logs/$name"
mkdir -p "$OUT/logs" "$OUT/results"

git -C "$repo" diff --quiet && git -C "$repo" diff --cached --quiet \
  || { echo dirty > "$OUT/results/$name.verdict"; exit 0; }
git -C "$repo" worktree remove --force "$wt" 2>/dev/null
git -C "$repo" branch -D "$BRANCH" 2>/dev/null
git -C "$repo" worktree add -q -b "$BRANCH" "$wt" HEAD 2> "$log.git" \
  || { echo no-worktree > "$OUT/results/$name.verdict"; exit 0; }

( cd "$wt" && eval "$SETUP" ) > "$log.setup" 2>&1

( cd "$wt" && timeout 900 claude -p "$(cat "$KIT/prompt.md")" \
    --permission-mode acceptEdits \
    --allowedTools "Bash(npx vitest *)" "Bash(git diff *)" "Bash(git status *)" \
    --max-turns 30 --max-budget-usd 1.50 \
    --no-session-persistence \
    --output-format json --json-schema "$(cat "$KIT/schema.json")" \
) > "$OUT/results/$name.json" 2> "$log.err"
echo $? > "$OUT/results/$name.exit"

( cd "$wt" && "$KIT/check.sh" ) > "$OUT/results/$name.check" 2>&1
if [ $? -eq 0 ]; then
  if git -C "$wt" diff --quiet HEAD; then echo noop > "$OUT/results/$name.verdict"; exit 0; fi
  ( cd "$wt" && git add -A && git -c user.name=fanout -c user.email=fanout@localhost \
      commit -qm "Migrate off acme-http get() callback API" )
  echo pass > "$OUT/results/$name.verdict"
else
  echo fail > "$OUT/results/$name.verdict"
fi

The worktree is made by git directly rather than with claude --worktree. The built-in flag puts the tree under <repo>/.claude/worktrees/, which is fine interactively, but a driver wants every tree in one directory it can list, report on and delete in one go.

The flags are doing specific jobs:

  • --permission-mode acceptEdits lets the agent write files without a prompt. Shell commands outside the read-only set still need an allow rule, so the three --allowedTools entries are the entire shell it gets: the test runner, git diff and git status. No npm install, no git commit, no curl.
  • --max-turns 30 and --max-budget-usd 1.50 bound a run that wanders. The docs say the budget is checked against a client-side estimate and spend can pass it, so treat it as a smell detector rather than a hard bill limit.
  • --no-session-persistence stops twelve throwaway transcripts piling up on disk.
  • timeout 900 is the backstop for a run that hangs, because a stuck process is the one failure the flags above can’t catch.

The SETUP variable is whatever turns a fresh checkout into a runnable one, npm ci by default. A worktree starts with only tracked files, so node_modules isn’t there.

The driver

The driver reads a list of repo paths and runs the per-repo script a few at a time.

#!/usr/bin/env bash
# fanout.sh repos.txt    one repo path per line
set -eu
export KIT=$(cd "$(dirname "$0")" && pwd)
export OUT=${OUT:-$PWD/.fanout}
export BRANCH=${BRANCH:-fanout/acme-http-4}
export SETUP=${SETUP:-"npm ci"}
rm -rf "$OUT"; mkdir -p "$OUT"

grep -v '^\s*$' "$1" | xargs -P "${JOBS:-3}" -I{} "$KIT/fanout-one.sh" {}

"$KIT/report.sh" "$1"

xargs -P is the entire concurrency layer. Three at a time is a sensible default: each run is mostly waiting on the API, but twelve simultaneous test suites will fight for the same CPU and give you timing-dependent failures you then have to untangle.

report.sh is a short loop that reads each repo’s result files with jq and prints a table. Read the fields it uses from the JSON: .structured_output.status, .structured_output.notes, .total_cost_usd and .num_turns.

The gate that doesn’t believe the agent

Agents report success in good faith and are sometimes wrong. The agent here says its tests passed, and in the runs below it was right, but the driver never uses that statement. check.sh runs in the worktree after the agent exits and re-derives everything.

#!/usr/bin/env bash
# Runs in the worktree after the agent finishes. Exit 0 = pass.
set -u
fail=0
say() { echo "$1"; }

# 1. Only src/ and test/ changed.
bad=$(git diff --name-only HEAD | grep -Ev '^(src|test)/' || true)
[ -z "$bad" ] || { say "FAIL scope: touched $bad"; fail=1; }

# 2. Exported signatures unchanged (async keyword and brace placement ignored).
norm() { sed -E 's/^export async /export /; s/ \{$//' | sort; }
before=$(git grep -hE '^export ' HEAD -- src | norm)
after=$(grep -rhE '^export ' src | norm)
[ "$before" = "$after" ] || { say "FAIL api: exports differ"; diff <(echo "$before") <(echo "$after"); fail=1; }

# 3. The deprecated import is gone.
left=$(grep -rnE "import[^;]*\bget\b[^;]*acme-http|\bget\(" src test || true)
[ -z "$left" ] || { say "FAIL old-api: $left"; fail=1; }

# 4. Tests pass, run by this script.
out=$(npx vitest run 2>&1) || { say "FAIL tests"; echo "$out" | tail -15; fail=1; }

# 5. Tests were not weakened: assertion count must not drop.
before=$(git grep -hcE 'expect\(' HEAD -- test | paste -sd+ | bc)
after=$(grep -rhcE 'expect\(' test | paste -sd+ | bc)
[ "$after" -ge "$before" ] || { say "FAIL weakened: expect() calls $before -> $after"; fail=1; }

[ $fail -eq 0 ] && say "PASS"
exit $fail

Each check answers a way an agent can go wrong while looking right. Scope catches an edit to vendor/ that makes the new function easier to call. The export comparison catches a changed public signature, which is the expensive mistake in a fan-out because it breaks other repos. The grep catches a half-finished migration where tests still pass because the deprecated function still exists. The assertion count catches the oldest trick: delete the failing expect and the suite goes green.

If a checkout does all three bad things at once (touches vendor/, keeps the old import, drops an assertion), the gate prints:

FAIL scope: touched vendor/acme-http/index.ts
FAIL old-api: src/customers.ts:1:import { get } from "../vendor/acme-http";
src/customers.ts:5:    get(`/customers/${id}`, (err, body) => {
src/invoices.ts:1:import { get } from "../vendor/acme-http";
src/invoices.ts:4:  get(`/invoices/${id}`, (err, body) => {
FAIL weakened: expect() calls 3 -> 2

What a run looks like

Run it with a file listing the four repo paths:

./kit/fanout.sh repos.txt

About 25 seconds later it prints:

repo       gate     agent          cost  turns  files     +/-  notes
billing    pass     migrated       0.03      5      2  +7/-12  Migrated both `get` call sites to `getJson`. `src/invoices.ts` `loadIn
gateway    pass     migrated       0.06     10      2   +3/-3  Migrated the one call site, `fetchUpstream` in src/proxy.ts. It now ca
reports    noop     nothing_to_do  0.02      4      0   +0/-0  No callback-style `get(url, cb)` calls exist under src/ or test/. The 
search     pass     migrated       0.03      6      1 +10/-10  Only src/query.ts used the deprecated get(). I rewrote it as an async

The cost column is the client-side estimate from total_cost_usd, in dollars. The notes column is cut at 70 characters by the report script and is much longer in the JSON.

Read it in the order a reviewer would. noop means the agent changed nothing and the gate agreed, so there is nothing to commit. pass rows each have a commit on fanout/acme-http-4 in their own repo, and the files and +/- columns tell you how big the review is before you open anything.

Then read the notes on the one that could hide a trap. The full note for gateway, the repo with a public callback signature, says:

Migrated the one call site, `fetchUpstream` in src/proxy.ts. It now calls `getJson` and forwards the result to the callback with `.then(body => cb(null, body), err => cb(err))`. The exported signature is unchanged and it still takes a callback. I used the two-argument `then` so a throwing callback isn't called a second time, which matches the old vendor behaviour. The mock in test/proxy.test.ts now mocks `getJson` instead of `get`, and the assertions are unchanged.

The diff is three lines in src/proxy.ts and a three-line change to the mock. The two-argument then is worth the 30 seconds it takes to confirm: with .then(...).catch(...) a callback that throws would be called again with its own error. This is the kind of decision no gate can score, and the table sent your attention there.

A gate that fails the wrong thing is still doing its job

The first version of the check compared exported lines literally. On the first run, search failed:

FAIL api: -export function search(term: string, attempts = 2): Promise<string[]> {

The agent had rewritten the retry loop with async, which is a correct change: the name, parameters and return type are identical. A literal diff of export lines can’t tell that from a real signature change. Hence the norm function in the gate above, which drops the async keyword before comparing.

Two lessons here. A gate that’s too strict produces a pile of false failures and trains you to ignore it, so when one fires, look at why before deciding the agent was wrong. And the opposite problem is real: line-based text checks are crude. They don’t understand overloads, multi-line signatures or re-exports, and a TypeScript project with a public API should swap the grep for tsc --noEmit against the old declaration files, or an API extractor.

Make it yours

  • Another language: only check.sh, SETUP and the prompt change. For C#, SETUP is dotnet restore, the gate runs dotnet test, and the signature check compares public members, ideally with a public API analyser rather than grep.
  • More repos: put paths in repos.txt. For repos you haven’t cloned, add a step that clones into a work directory first and set BRANCH per migration so branches don’t collide across campaigns.
  • Pushing: the driver stops at a local commit on purpose. The step after is git push -u origin fanout/acme-http-4 and a PR per repo, which is a separate decision. Add it once a few campaigns have gone well, and keep the pull request, not the push, as the point where a human looks.
  • Reproducibility: the headless docs recommend --bare for scripted calls, since it skips hooks, plugins, MCP servers, auto memory and CLAUDE.md so every machine behaves the same. It authenticates with ANTHROPIC_API_KEY rather than a subscription login. The trade-off is that the agent also loses each repo’s own CLAUDE.md, so if conventions live there, either restate the important ones in the prompt file or leave --bare off.

What goes wrong

Running claude -p in a repository you didn’t write is running that repository’s configuration. The permissions page says that in a folder that was never trusted, a -p run uses hooks defined in the repository’s settings files and connects servers from its .mcp.json, without asking. Allow rules from the repository’s .claude/settings.json are not applied. For your own services that’s usually fine. For a campaign over repos from other teams, use --bare, or read what’s in their .claude/ first.

The gate proves what it checks and nothing more. Passing tests on a repo with thin coverage mean the agent didn’t break what the tests see. The batch table can’t tell you about behaviour the tests don’t pin, which is why the notes column and the diff for anything unusual still need a human. If a repo has no tests, don’t run the campaign there until it has characterisation tests, or accept that the review is the only check.

Shared state breaks the independence the design relies on. If two repos share a database in their tests, or a test binds a fixed port, JOBS=3 will produce flaky failures that look like agent errors. Drop to JOBS=1 to confirm before blaming the change.

Don’t bother with any of this for fewer than four or five repos. At that size, claude in each directory with your eyes on the diff costs about the same and you’ll catch more.

First 15 minutes

Pick a migration you’re about to do by hand and two repos that have it, one boring and one with something odd in it. Copy the prompt, check.sh and the two scripts into a kit/ folder outside both repos, change the three tool-specific lines (the deprecated pattern in check 3, the test command, SETUP), and write the two repo paths into repos.txt.

Run ./kit/fanout.sh repos.txt. Read the table, open the diff of the repo with the odd bit, and note which check you wished existed. Add it to check.sh before you add a third repo.