Let a benchmark referee the agent's performance fix

A customer-reconciliation job takes 130 ms on 20,000 order lines, and the ticket says “make it faster”. Ask an agent and it comes back with a diff that looks plausible: the regex is hoisted, a list becomes a hash set, maybe there’s a Parallel in there for good measure. The tests pass. Whether it’s faster, and by how much, and at what cost in allocations, is left to the reviewer’s gut.

Gut is the wrong judge. Performance changes need two checks that a normal review doesn’t give: proof that the new code still does exactly what the old code did, and a measurement that says the new code is actually better. This Playbook sets up both as commands the agent can run itself, plus a small gate script that turns the numbers into ACCEPT or REJECT. By the end, the agent can try variants all day and only the ones that survive get to you.

Nothing here is specific to one agent. The prompt, the equivalence test and the benchmark gate all work with any coding agent that can read files and run dotnet. The code targets .NET 10 and BenchmarkDotNet 0.15.8.

The setup: one baseline, one referee

Keep the slow, correct code as a frozen baseline. The agent never edits it. Every candidate is a new method with the same signature, and two things judge it: an xUnit test that asserts the candidate returns what the baseline returns, and a BenchmarkDotNet run that measures it against the baseline.

src/Recon/Baseline.cs        frozen, slow, correct
src/Recon/Candidates.cs      where the agent works
tests/Recon.Tests/           equivalence tests
bench/Recon.Bench/           BenchmarkDotNet project
gate.py                      reads the benchmark JSON, accepts or rejects

The method under test sums order lines per SKU, skipping blocked SKUs and malformed lines. Lines look like SKU-00042,3,19.99. This is the baseline, and it has the usual suspects an agent finds quickly: a Regex constructed inside the loop, a List<string>.Contains on the blocked list, and decimal parsing for a value that is really whole cents.

using System.Globalization;
using System.Text.RegularExpressions;

namespace Recon;

public static class Baseline
{
    // Totals per SKU in cents. Lines look like "SKU-00042,3,19.99".
    // Malformed lines and blocked SKUs are skipped.
    public static Dictionary<string, long> Totals(IReadOnlyList<string> lines, IReadOnlyList<string> blockedSkus)
    {
        var blocked = blockedSkus.ToList();
        var totals = new Dictionary<string, long>();
        foreach (var line in lines)
        {
            var m = new Regex(@"^(?<sku>[A-Z]+-\d+),(?<qty>\d+),(?<price>\d+\.\d{2})$").Match(line);
            if (!m.Success) continue;
            var sku = m.Groups["sku"].Value;
            if (blocked.Contains(sku)) continue;
            var qty = int.Parse(m.Groups["qty"].Value, CultureInfo.InvariantCulture);
            var cents = (long)Math.Round(decimal.Parse(m.Groups["price"].Value, CultureInfo.InvariantCulture) * 100);
            totals[sku] = totals.GetValueOrDefault(sku) + qty * cents;
        }
        return totals;
    }
}

Creating a Regex per line is the expensive part. Microsoft’s regex guidance says the static Regex methods use a cache, while a freshly instantiated object pays the pattern-to-opcodes conversion again. The baseline pays that 20,000 times.

Gate one: the equivalence test

The test needs data and a place to put awkward lines. A seeded generator gives the same 20,000 lines on every run, with a malformed line every 97th row and a blocked list that actually hits some SKUs.

namespace Recon;

public static class SampleData
{
    public static (List<string> Lines, List<string> Blocked) Build(
        int lineCount = 20_000, int skuCount = 500, int blockedCount = 300, int seed = 42)
    {
        var rng = new Random(seed);
        var lines = new List<string>(lineCount);
        for (var i = 0; i < lineCount; i++)
        {
            if (i % 97 == 0) { lines.Add("not,a,line"); continue; }
            var sku = rng.Next(skuCount);
            var qty = rng.Next(1, 10);
            var price = rng.Next(100, 99_999) / 100m;
            lines.Add($"SKU-{sku:00000},{qty},{price:0.00}");
        }
        var blocked = Enumerable.Range(0, blockedCount).Select(i => $"SKU-{skuCount + 10 + i:00000}").ToList();
        blocked.AddRange(Enumerable.Range(0, 20).Select(i => $"SKU-{i * 7:00000}"));
        return (lines, blocked);
    }
}

Then the tests. Every candidate is registered once in a dictionary, and two theories run each one against the baseline: one on the generated data, one on a hand-written list of edge lines. The edge list is where candidates that “look equivalent” get caught, so make it nasty: empty strings, missing fields, a negative price, one decimal place, three decimal places, extra columns, leading zeros, and line endings.

using Recon;

namespace Recon.Tests;

public class EquivalenceTests
{
    private delegate Dictionary<string, long> Impl(IReadOnlyList<string> lines, IReadOnlyList<string> blocked);

    private static readonly Dictionary<string, Impl> All = new()
    {
        ["HoistedRegex"] = Candidates.HoistedRegex,
        ["RegexAndHashSet"] = Candidates.RegexAndHashSet,
        ["SpanParser"] = Candidates.SpanParser,
        ["ParallelBaseline"] = Candidates.ParallelBaseline,
    };

    public static TheoryData<string> Names => new(All.Keys);

    [Theory]
    [MemberData(nameof(Names))]
    public void Matches_baseline_on_generated_data(string name)
    {
        var (lines, blocked) = SampleData.Build();
        Assert.Equal(Baseline.Totals(lines, blocked).OrderBy(p => p.Key),
                     All[name](lines, blocked).OrderBy(p => p.Key));
    }

    [Theory]
    [MemberData(nameof(Names))]
    public void Matches_baseline_on_edge_lines(string name)
    {
        var lines = new[]
        {
            "", ",", ",,", "SKU-1,2,10.50", "SKU-1,2,10.5", "SKU-1,2,10.500", "SKU-1,2,.50", "SKU-1,2,-1.00",
            "-1,2,1.00", "SKU--1,2,1.00", "SKU-1,2,3.00,extra", "SKU-1,,1.00", "SKU-1,02,0.05",
            "SKU-4,1,1.00\n", "SKU-5,1,1.00\r\n",
        };
        var none = Array.Empty<string>();
        Assert.Equal(Baseline.Totals(lines, none).OrderBy(p => p.Key),
                     All[name](lines, none).OrderBy(p => p.Key));
    }
}

Comparing sorted sequences matters: dictionary enumeration order isn’t part of the contract, so a test that compares dictionaries by enumeration would fail on a correct candidate. Run it with dotnet test tests/Recon.Tests.

What the test catches that a review doesn’t

The fastest candidate parses with spans and no regex at all. A first version of it passes the generated data and fails the edge lines:

Failed Recon.Tests.EquivalenceTests.Matches_baseline_on_edge_lines(name: "SpanParser") [10 ms]
  Error Message:
   Assert.Equal() Failure: Collections differ
                             ↓ (pos 1)
Expected: [["SKU-1"] = 2110, ["SKU-4"] = 100]
Actual:   [["SKU-1"] = 2110]

SKU-4,1,1.00\n is the culprit. In .NET regular expressions, $ matches at the end of the string and also before a single final \n, so the baseline accepts a line with a trailing newline and the hand-written parser rejects it. Whether that behaviour is intended is a product question. Until someone answers it, the safe rule is that the optimised code keeps it, and the one-line fix is to strip a single trailing \n before validating the price. The \r\n line is rejected by both, which the test also confirms.

Here is the winning candidate with the fix applied. It uses HashSet<string> and Dictionary<string, long> alternate lookups, so the SKU is never turned into a string unless it is a key being added.

public static Dictionary<string, long> SpanParser(IReadOnlyList<string> lines, IReadOnlyList<string> blockedSkus)
{
    var blocked = blockedSkus.ToHashSet();
    var lookup = blocked.GetAlternateLookup<ReadOnlySpan<char>>();
    var totals = new Dictionary<string, long>();
    var totalsLookup = totals.GetAlternateLookup<ReadOnlySpan<char>>();
    foreach (var line in lines)
    {
        var s = line.AsSpan();
        var c1 = s.IndexOf(',');
        if (c1 <= 0) continue;
        var rest = s[(c1 + 1)..];
        var c2 = rest.IndexOf(',');
        if (c2 <= 0) continue;
        var sku = s[..c1];
        var qtySpan = rest[..c2];
        var priceSpan = rest[(c2 + 1)..];
        // Regex $ also matches before one final \n, so the baseline accepts it.
        if (priceSpan.EndsWith('\n')) priceSpan = priceSpan[..^1];
        if (!ValidSku(sku) || !AllDigits(qtySpan)) continue;
        if (!TryCents(priceSpan, out var cents)) continue;
        if (lookup.Contains(sku)) continue;
        var qty = int.Parse(qtySpan, CultureInfo.InvariantCulture);
        totalsLookup.TryGetValue(sku, out var current);
        totalsLookup[sku] = current + qty * cents;
    }
    return totals;
}

ValidSku checks [A-Z]+-\d+, AllDigits checks a non-empty digit run, and TryCents checks \d+\.\d{2} and returns whole cents without going through decimal. They are a dozen lines each, and the equivalence test is the reason it’s safe to trust them.

Gate two: the benchmark

The other three candidates are the changes an agent proposes first. HoistedRegex moves the pattern into a [GeneratedRegex] partial method, which the .NET SDK’s source generator turns into a compiled implementation. RegexAndHashSet adds a HashSet for the blocked list. ParallelBaseline is the one that always shows up: AsParallel().ForAll with a ConcurrentDictionary.

The benchmark project has one class, with the baseline marked so every other row is scaled against it, and [MemoryDiagnoser] so allocations are reported next to time.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using Recon;

BenchmarkRunner.Run<ReconBench>(args: args);

[MemoryDiagnoser]
public class ReconBench
{
    private List<string> _lines = null!;
    private List<string> _blocked = null!;

    [GlobalSetup]
    public void Setup() => (_lines, _blocked) = SampleData.Build();

    [Benchmark(Baseline = true)] public object Baseline_() => Baseline.Totals(_lines, _blocked);
    [Benchmark] public object HoistedRegex() => Candidates.HoistedRegex(_lines, _blocked);
    [Benchmark] public object RegexAndHashSet() => Candidates.RegexAndHashSet(_lines, _blocked);
    [Benchmark] public object SpanParser() => Candidates.SpanParser(_lines, _blocked);
    [Benchmark] public object ParallelBaseline() => Candidates.ParallelBaseline(_lines, _blocked);
}

Benchmarks must run in Release, so the command is dotnet run -c Release --project bench/Recon.Bench -- --filter '*'. On a 4-core Intel Xeon at 2.10GHz with .NET SDK 10.0.114, the default job prints:

| Method           | Mean       | Error     | StdDev     | Median     | Ratio | Gen0     | Gen1    | Allocated    | Alloc Ratio |
| Baseline_        | 133.163 ms | 8.7482 ms | 25.7942 ms | 123.072 ms |  1.03 |        - |       - | 118316.52 KB |       1.000 |
| HoistedRegex     |  33.145 ms | 1.7011 ms |  4.9080 ms |  31.792 ms |  0.26 |  76.9231 |       - |  13904.32 KB |       0.118 |
| RegexAndHashSet  |  11.210 ms | 0.3950 ms |  1.1460 ms |  10.952 ms |  0.09 |  93.7500 | 15.6250 |  13908.81 KB |       0.118 |
| SpanParser       |   2.205 ms | 0.0737 ms |  0.2161 ms |   2.156 ms |  0.02 |        - |       - |     72.8 KB |       0.001 |
| ParallelBaseline |   9.431 ms | 0.1852 ms |  0.4026 ms |   9.415 ms |  0.07 |  140.6250 | 15.6250 |  18756.4 KB |       0.159 |

(Columns trimmed. The baseline’s ratio shows 1.03 and not 1.00 because it is itself noisy, with a standard deviation of about 26 ms.)

Three things are worth reading off that table. Hoisting the regex alone is worth about four times, and the baseline’s 118 MB of allocation per call mostly disappears with it. The HashSet is worth another three times on top, because 320 blocked SKUs make List.Contains a linear scan on every line.

The third is the parallel version. It is faster than the regex-plus-hash-set version, 9.4 ms against 11.2 ms, and that’s the number a quick reviewer sees. It also keeps four cores busy and allocates 18 MB per call, and SpanParser finishes in a quarter of the time on one core. On a web server where those cores also serve requests, the parallel version is the one that makes the p99 worse. The benchmark won’t show that. It does show that the parallel candidate bought its speed with hardware and not with better code.

Gate three: accept or reject without reading a table

A table needs a reader. A gate needs a threshold. BenchmarkDotNet can export JSON, and a short script can compare medians and allocations for the candidate the agent names:

#!/usr/bin/env python3
"""Fail unless the chosen candidate beats the baseline by a margin, on time and on allocations."""
import glob, json, sys

candidate, min_speedup = sys.argv[1], float(sys.argv[2])
path = sorted(glob.glob("BenchmarkDotNet.Artifacts/results/*-report-full-compressed.json")
              or glob.glob("BenchmarkDotNet.Artifacts/results/*-report-full.json"))[-1]
rows = {b["Method"]: b for b in json.load(open(path))["Benchmarks"]}
base, cand = rows["Baseline_"], rows[candidate]

speedup = base["Statistics"]["Median"] / cand["Statistics"]["Median"]
alloc = cand["Memory"]["BytesAllocatedPerOperation"] / base["Memory"]["BytesAllocatedPerOperation"]
print(f"{candidate}: {speedup:.1f}x faster (median), {alloc:.1%} of baseline allocations")

if speedup < min_speedup or alloc > 1.0:
    sys.exit(f"REJECT: needs at least {min_speedup}x and no extra allocation")
print("ACCEPT")

The reported field names come from the full JSON export, which is why the command adds --exporters json. The --job short option cuts a run to under a minute, which suits the agent’s inner loop; use the default job before anything is merged. With the short job:

$ python3 gate.py SpanParser 5
SpanParser: 49.1x faster (median), 0.1% of baseline allocations
ACCEPT

$ python3 gate.py ParallelBaseline 20
ParallelBaseline: 15.2x faster (median), 16.0% of baseline allocations
REJECT: needs at least 20.0x and no extra allocation

The exit code is non-zero on a reject, so the agent sees a failed command and a message that says what to change. The numbers differ from the default-job table, since a short job takes fewer, noisier samples. That’s the reason to set the threshold well below what the best candidate achieves, and not at the edge of it.

The prompt

This is the instruction to give the agent. It is written so the agent can’t satisfy it by editing the referee.

Goal: make Baseline.Totals faster without changing what it returns.

Rules:
- Do not edit Baseline.cs, SampleData.cs, the tests, the benchmark project or gate.py.
- Add each attempt as a new static method in Candidates.cs with the same signature,
  and register it in the All dictionary in EquivalenceTests.cs and in ReconBench.
- A candidate must pass: dotnet test tests/Recon.Tests
- Then measure: dotnet run -c Release --project bench/Recon.Bench -- --filter '*' --job short --exporters json
- Then judge: python3 gate.py <MethodName> 5
- Report each attempt as: what changed, test result, gate output.
- If the equivalence test fails on an edge line, say whether the baseline's behaviour looks
  intended. Do not change the test to match the candidate. Keep the baseline's behaviour.
- Stop after one candidate is ACCEPTED, or after five rejected attempts.

The registration step is the one place the agent edits the test file, and a reviewer should check that the diff there is one added line. That’s easy to see, and it is the only way the referee can be weakened, so it’s worth looking at every time.

Making it yours

For a hot path in an ASP.NET Core service, move the data builder to something that resembles production: real line lengths, the real share of malformed rows, the real size of the blocked list. A benchmark that uses 20 blocked SKUs when production has 20,000 picks a different winner.

For an endpoint and not a pure function, keep the equivalence test at the HTTP boundary with WebApplicationFactory and compare response bodies, and benchmark the handler with the dependencies stubbed. The query-count technique in the EF Core query budget post covers the database side, which BenchmarkDotNet on a stubbed repository won’t.

For CI, run only the equivalence tests on every pull request, and run the benchmark gate on a schedule or on demand. Shared CI runners are noisy enough that a short job can flip a close result, so gate only on large margins there and keep the strict check for a dedicated machine.

What goes wrong

Micro-benchmarks flatter. The 49x above is the speedup of one function over 20,000 in-memory lines on one machine. It says nothing about the request that calls it, which may spend 98% of its time in the database. Before a change ships, confirm in a profiler or in production metrics that this function was the cost worth cutting.

The equivalence test is only as good as its edge list. The trailing-newline difference turned up because two of the edge lines were written to probe line endings. A difference nobody thought to include in the list will ship. Property-based tests that generate random strings widen that net, and a diff of outputs from a captured sample of real production lines is better still, minus anything sensitive.

The baseline can be wrong, and the test will faithfully preserve its bugs. If SKU-4,1,1.00\n should have been rejected all along, that’s a separate fix, made on purpose, in its own commit.

And don’t bother with any of this for code that runs once a day on 200 rows. The setup costs more than it saves.

The first 15 minutes

Pick one function you already suspect is slow, copy it to Baseline.cs unchanged, and write the generated-data equivalence test before anything else. Add the benchmark project, run it once in Release to see the baseline’s numbers, and only then hand the prompt to an agent.