Google let an agent optimise a 205,000-line repo overnight. Two of six changes shipped, and the holdout benchmark is why

Google’s Agent Development Kit (ADK) team pointed Gemini 4 Argon at the google/adk-python repository, about 205,000 lines across nearly 800 source files, and left it running for 11 hours. It proposed six optimisations across 32 files. Maintainers shipped two and rejected four, according to Google’s write-up .

The headline numbers are about 40% less pre-model overhead in long sessions and 21% faster cold imports. They’re less interesting than how the run was set up, because the same team ran the experiment twice and the first attempt shows what an unattended agent does when nothing surrounds it.

Round one: a paragraph and no rules

The first run got one paragraph asking for no noticeable overhead in startup, long sessions and multi-agent setups, with the model input and public API unchanged and a regression test for each change. No repository instructions. It ran for about two hours.

The model wrote a benchmark script seven minutes in, then spent 70 minutes measuring and reading before it touched ADK source. That part was good. The output wasn’t: it missed maintainer conventions, changed a public JSON schema through a hook on the authentication model, ran only 216 nearby unit tests and left the benchmark numbers out of its pull request description.

None of those are intelligence failures. They’re the failures of a new hire who was never shown the contributing guide, and they’re the ones you’ll see on your own repo.

Round two: three additions

The second run kept the goal and added three things.

First, the maintainers’ guidelines went into .agents/skills in the repository. Each skill has a short SKILL.md index that points to focused files under references/, loaded on demand so the context window isn’t flooded on every turn. Before editing anything, Argon read 12 of these files, including the rule for private underscore-prefixed modules and the checks for lazy loading.

Second, the goal was pinned. Antigravity’s /goal mode, Google says, keeps the goal alive across context compactions, which matters when a run outlives several of them. The prompt gained two sentences: open one pull request against main that maintainers would approve, with before and after numbers, and “Don’t merge it.”

Third, the team wrote an independent holdout benchmark, bench_turn.py, that the agent never saw. This is the piece worth stealing.

Why the holdout mattered

The agent had its own benchmarks: four workloads (package startup, a 200-step conversation, a 20-sub-agent handoff app and a 12-node workflow), each run 30 times before it edited anything. Optimising against your own measurements is how you get changes that win on the thing you measured and nowhere else.

Three of the four rejected changes show it. A multi-agent event cache gained 17% on a synthetic benchmark but relied on a weak reference that could return stale data across sessions if Python reused a memory address after garbage collection. A rewrite of how conversation history is formatted made a core path harder to read and didn’t move end-to-end latency on the holdout. A third change only helped synthetic replay tests.

The holdout is what separated a real gain from a gain the agent had tuned for. Google’s figures come from it: one session grows against an instant model and instant tools, each turn runs three rounds of four parallel tool calls, and each number is the median of 6 to 12 runs with byte-identical requests before and after.

The fourth rejection wasn’t a performance problem at all. It read a private attribute on OpenTelemetry’s tracer that could break silently on a minor upgrade. That is a review call, and no benchmark catches it. Google’s own summary of the split is that the agent explores and the maintainer subtracts.

What shipped, and a trap in it

The first shipped change filters session history before copying it instead of deep-copying everything on every read. The second makes heavy imports lazy. Deleting them outright passed all 16,892 open-source tests and still failed 25 internal Google test targets that patch those exact module attributes, so the shipped version uses lazy module accessors that keep the attributes patchable.

That’s a good example of why a passing test suite isn’t the finish line. The suite didn’t know what downstream code depended on.

The results come with honest limits from Google. The 40% applies to the in-memory session service, because database-backed stores deserialise on read and don’t change. Time to first request in a fresh process was flat (3.07 s to 3.09 s), because the deferred imports load on the first turn, so Google recommends a warm-up request before a server takes live traffic. A human won’t feel 28 ms or 78 ms on one call. Across a 158-turn run it adds up to 65.6 s falling to 56.1 s. The post gives no token or dollar cost for the run.

A holdout harness you can run

The holdout idea doesn’t need ADK. It needs a benchmark that lives outside the repository the agent works in, runs the same command at two git refs and reports a median. This script does that. Run the benchmark on the agent’s branch and on main, and keep the file somewhere the agent can’t read.

#!/usr/bin/env python3
"""usage: holdout_bench.py REPO BEFORE_REF AFTER_REF RUNS -- CMD [ARGS...]"""
import statistics, subprocess, sys, tempfile, time
from pathlib import Path

def median_ms(tree: Path, cmd: list[str], runs: int) -> float:
    subprocess.run(cmd, cwd=tree, check=True, capture_output=True)  # warm-up, not timed
    samples = []
    for _ in range(runs):
        start = time.perf_counter()
        subprocess.run(cmd, cwd=tree, check=True, capture_output=True)
        samples.append((time.perf_counter() - start) * 1000)
    return statistics.median(samples)

def main() -> None:
    repo, before, after, runs = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
    cmd = sys.argv[sys.argv.index("--") + 1:]
    with tempfile.TemporaryDirectory() as tmp:
        results = {}
        for label, ref in (("before", before), ("after", after)):
            tree = Path(tmp) / label
            subprocess.run(["git", "-C", repo, "worktree", "add", "--detach", str(tree), ref],
                           check=True, capture_output=True)
            results[label] = median_ms(tree, cmd, runs)
        for label in results:
            subprocess.run(["git", "-C", repo, "worktree", "remove", "--force", str(Path(tmp) / label)],
                           check=True, capture_output=True)
    b, a = results["before"], results["after"]
    print(f"before {b:8.1f} ms   after {a:8.1f} ms   change {(a - b) / b * 100:+.1f}%   (median of {runs})")

main()

Point it at a repo whose last commit is a “filter before copying” change like ADK’s first win, and it prints:

before    389.9 ms   after     60.7 ms   change -84.4%   (median of 9)

That toy repo deep-copies a 3,000-message history before filtering it, then filters first. The absolute numbers mean nothing for your code. The shape does: the same command, two refs, a median, and a verdict the agent didn’t write.

Copy the setup, not the scale

A reasonable version of round two for your repo:

1. Write your review rules down as a short index plus one file per rule, and say where they live in your agent’s context file. Include the boring ones: private module naming, import conventions, what counts as a public API.
2. State the goal as an outcome with constraints, and finish with the delivery format: one pull request, numbers in the description, not merged.
3. Keep a holdout benchmark outside the repo and compare before and after yourself.
4. Review the diff for what no benchmark sees: private attributes, weak references, anything that works until a minor version bump.

Expect to throw most of it away. Four of six is a normal discard rate for this kind of work, and a run that returns six accepted changes should make you suspicious of the review, not impressed by the agent.

Claude Code documents a /goal command too, and Codex has one (see Codex /goal only stops when you tell it how to check ), so the setup isn’t tied to Antigravity. Start with step three: write the holdout benchmark for the one path you’d most like faster, before the agent has seen the code.