Microsoft's agent migration recipe: approve the rules, not the fixes, and let a differential test grade the result

The hard part of an agent-driven migration stopped being the translation a while ago. It’s the review. A migration that touches 400 files produces 400 diffs, and a human who reads them one at a time either gives up around file 60 or starts approving on rhythm. Microsoft’s Command Line publication published the first post of a series on this on 26 October 2026 , covering how its teams use agents to move very large codebases to new toolchains and from legacy C and C++ to Rust. Its answer to the review problem is worth copying at a much smaller scale.

Review the rules, not the edits

Microsoft describes an operating model with four moving parts. Agents first learn a candidate knowledge base from a sample of findings: for each kind of problem, what the fix looks like. A reviewer agent then scores every proposed fix on correctness, minimality, consistency with the conventions of the file it lands in, and maintainability, and those scores feed back into the knowledge base. Experts approve the knowledge base, not the individual fixes. Only then do agents apply it across the codebase, and similar fixes are grouped so a human sees one pattern instead of fifty copies.

That reorders where human attention goes. Reading one rule (“replace this deprecated call with that one, keep the null check”) is cheap and the decision is durable. Reading the fiftieth application of it is expensive and teaches nothing. The post says human feedback is reserved for approving the knowledge and for the novel or ambiguous cases the rules don’t cover.

None of this needs a research lab. A team migrating a framework version can write the rules down in a markdown file, have the agent fill in the before and after for each rule from a sample of real call sites, and approve that file once. The agent then works from the approved rules, and anything it can’t match to a rule goes on a separate list for a person.

The validation half

Rules approved by a person still get applied by an agent, so the other half of the recipe is how the output gets checked. Microsoft names three mechanisms.

First, migrate by feature, not by line. The team found that translating one feature at a time works better than translating a component line by line. A feature here is a unit of user-visible behaviour that can be validated end to end with scenario-driven tests, and each one has its own validation gate. The change a reviewer approves is “this behaviour still holds”, not “these lines look right”.

Second, differential testing: run the source and the target on the same inputs and compare what they do. Third, held-out tests that the agent never sees, which measure whether the translation is robust or merely tuned to the tests it was shown. That last one is the same idea as the holdout benchmark covered earlier on Tokenise, applied to a migration.

The Rust timeline in the post shows why the gates matter. In January 2025 the generated Rust didn’t compile. By June 2025 it compiled but didn’t run correctly or pass tests. By September 2025 the team had working translations of about 30,000 lines, with manual guidance, and from February 2026 it was modernising large codebases with significant automation. Each stage moved the bottleneck: compiling, then passing tests, then validated behaviour at scale. The last is where a human can’t keep up by reading.

A differential test you can run today

If the old and new implementations both exist, a differential harness is a short script. The one below runs two commands over a directory of input files and compares exit code and stdout. The twist borrowed from the grouping idea: mismatches aren’t printed one per file. They’re bucketed by a signature, so 26 failures show up as a handful of patterns.

#!/usr/bin/env python3
"""Run two implementations over the same inputs and group the mismatches."""
import re, subprocess, sys
from collections import defaultdict
from pathlib import Path

old_cmd, new_cmd, corpus = sys.argv[1].split(), sys.argv[2].split(), Path(sys.argv[3])

def run(cmd, data):
    p = subprocess.run(cmd, input=data, capture_output=True, text=True, timeout=10)
    return p.returncode, p.stdout.strip()

def signature(data, old, new):
    # Mask digits so "total=10 vs total=11" and "total=7 vs total=8" share a group,
    # then add the input features that matter for this migration.
    shape = re.sub(r"\d+", "N", f"{old} -> {new}")
    return f"negative input: {'-' in data} | new == 100: {new[1] == 'total=100'} | {shape}"

groups = defaultdict(list)
files = sorted(corpus.glob("*.txt"))
for f in files:
    data = f.read_text()
    old, new = run(old_cmd, data), run(new_cmd, data)
    if old != new:
        groups[signature(data, old, new)].append(f.name)

print(f"{len(files)} inputs, {sum(map(len, groups.values()))} mismatches, {len(groups)} groups")
for sig, names in sorted(groups.items(), key=lambda kv: -len(kv[1])):
    print(f"\n[{len(names)}] {sig}\n    e.g. {', '.join(names[:3])}")
sys.exit(1 if groups else 0)

To see it work, take a trivial “legacy” program that sums integers read from stdin, and a “migrated” version that has picked up two bugs: it skips lines starting with a minus sign, and it caps the total at 100. Run against 59 generated input files:

python3 difftest.py "python3 old.py" "python3 new.py" corpus

59 inputs, 26 mismatches, 3 groups

[23] negative input: False | new == 100: True | (N, 'total=N') -> (N, 'total=N')
    e.g. case00.txt, case02.txt, case12.txt

[2] negative input: True | new == 100: True | (N, 'total=N') -> (N, 'total=N')
    e.g. case08.txt, case32.txt

[1] negative input: True | new == 100: False | (N, 'total=N') -> (N, 'total=N')
    e.g. case52.txt

Twenty-six failing files, three groups, and each group points at a cause: the cap (23), the cap and the negatives together (2), and negatives alone (1). A reviewer reads three lines and goes looking for two bugs. Without the grouping that’s 26 diffs, mostly identical.

The signature function is the part to adapt. The digit masking keeps “total=10 against total=11” from becoming its own group, and the two extra fields in the signature are features of the input and output that matter for this migration. Pick them by looking at the first dozen mismatches, then tighten. For a real service the equivalent might be the HTTP status pair, the endpoint, or whether the payload contained a null.

Where it breaks

A differential test only proves the new code behaves like the old code. If the old code has a bug, the migration will faithfully reproduce it, and the test will call that a pass. Microsoft’s post raises the same worry about specifications: if an agent infers a specification from the code it’s supposed to check, how can anyone tell whether it captures what the system should do or only what it currently does? Treat a green differential run as “behaviour preserved”, never as “behaviour correct”.

The corpus is the other limit. Inputs the corpus doesn’t contain produce no mismatches, so a clean run on 59 hand-picked files says little about the 60th. Seed the corpus from production traffic or logs where you can, and keep a second set that the agent never sees, so it can’t tune the migration to the files it’s being graded on.

Grouping can also hide things. Two different bugs with the same signature look like one, which is why the signature should be tightened whenever a group turns out to contain unlike cases. And a reviewer agent scoring fixes is another model’s opinion, not a test, so its scores prioritise human review rather than replace it.

First ten minutes

Pick one migration that’s already planned and write three things before any agent touches code: a markdown file listing the two or three transformation rules you expect, a directory of 30 to 50 representative inputs, and a held-out directory of 10 more that stays out of the agent’s working tree. Copy the script above, point it at the old and new commands, and run it against the unmigrated code first. It should print zero mismatches. If it doesn’t, the harness is the problem and the agent hasn’t been involved yet.