ReviewBench scores AI code reviewers on 219 pull requests. Here's how to run the same test on your own

A reviewer that leaves ten comments on a pull request and gets one real bug has a precision problem. A reviewer that leaves two comments and misses the bug has a recall problem. Which one is worse depends on your team, and a leaderboard can’t know that.

GitHub published ReviewBench on 5 October 2026, an open benchmark for AI code review agents, in research preview at review-bench.ai. It’s worth understanding, mostly because it shows how to measure a reviewer properly. The same method works on a much smaller scale against your own repository, and that’s the version that tells you whether to keep paying for the reviewer you’ve got.

What ReviewBench measures

The dataset is 219 public pull requests from 187 open-source-licensed repositories, across 19 languages, according to the GitHub post . Each pull request has a golden set of findings, and a finding only counts as a true positive if it’s true, relevant and non-trivial, whatever produced it. Claude Sonnet 5 is the LLM grader, applying one rubric to every submission.

Scoring comes in two families. Grounded precision and recall use only the golden labels, which makes systems comparable. Augmented metrics also judge findings that match nothing in the golden set, so a reviewer gets credit for a valid issue nobody had labelled. GitHub names grounded recall as the headline comparison, and treats the augmented numbers as per-system diagnostics, because the denominator of augmented recall changes with each agent’s output.

There’s also an Fβ score that weights recall against precision, and the leaderboard re-ranks as β changes. That’s the useful part for a reader: the ranking isn’t one fact, it depends on how much you hate noise.

To enter, you register an agent with a container image, a configuration and your own model key. GitHub supplies the judge.

What the numbers do and don’t say

Two figures in the post deserve a second look. Senior engineers who didn’t build the dataset re-labelled the ground truth, and their true and false positive judgments agreed with ReviewBench 96.6% of the time. That’s evidence the labels are sound. It isn’t evidence that the labels are complete, which is why the augmented metrics exist.

The second is the link to production. In an experiment on the lighter tier of Copilot code review, ReviewBench predicted that a multi-model ensemble would raise precision, recall and comment volume while lowering cost per review. The online A/B test went the same way: addressed rate, GitHub’s proxy for precision, rose 8.0%, recall rose 13.6%, comment volume rose 61% and cost per review fell 8.0%.

Read that carefully. Direction of effect was predicted correctly, in the experiment GitHub chose to show. The addressed rate is itself a judged proxy rather than a count of bugs prevented, and the benchmark’s pull requests are open-source projects weighted towards the reviewable middle. Your private monolith with its in-house frameworks is not in there.

That’s an opinion, not a criticism: a public benchmark can tell you which reviewers are plausible. It can’t tell you which one works on a codebase it has never seen.

A 20-PR benchmark of your own

The expensive parts of ReviewBench are the dataset and the judge. At small scale you can replace both with human labels, because you only need enough pull requests to see a difference between two configurations.

Pick 20 merged pull requests from the last six months, a mix of sizes. For each, write down the findings a good reviewer should have raised, using three sources: review comments a human made that led to a change, follow-up commits that fixed something the PR introduced, and bugs you remember shipping. Give each a short id.

Then run the reviewer over the same diffs. Most reviewers can be pointed at a PR or a diff after the fact; the Claude Code Review and REVIEW.md posts cover two ways to run one. For each finding the reviewer raised, mark which golden id it matches, or null if it matches none. That marking is a minute of work per finding, and it’s the step to do by hand, since an unlabelled reviewer scoring itself is how you end up trusting a number.

Save it as JSON:

[
  {
    "pr": 101,
    "golden": ["null-deref-in-retry", "missing-auth-check"],
    "findings": [
      {"text": "retry loop can dereference null response", "matches": "null-deref-in-retry"},
      {"text": "rename variable for clarity", "matches": null}
    ]
  }
]

The scorer

This computes grounded precision, recall, F1 and an Fβ of your choosing. A golden finding counts once even if the reviewer raises it twice, and anything with matches: null is a false positive.

import json, sys

def fbeta(p, r, beta):
    if p == 0 and r == 0:
        return 0.0
    b2 = beta * beta
    return (1 + b2) * p * r / (b2 * p + r)

data = json.load(open(sys.argv[1]))
beta = float(sys.argv[2]) if len(sys.argv) > 2 else 1.0
tp = fp = fn = 0
for pr in data:
    golden = set(pr["golden"])
    matched = {f["matches"] for f in pr["findings"] if f["matches"]}
    tp += len(matched)
    fp += sum(1 for f in pr["findings"] if not f["matches"])
    fn += len(golden - matched)

precision = tp / (tp + fp) if tp + fp else 0.0
recall = tp / (tp + fn) if tp + fn else 0.0
print(f"PRs={len(data)} tp={tp} fp={fp} fn={fn}")
print(f"precision={precision:.2f} recall={recall:.2f} F1={fbeta(precision, recall, 1):.2f} F{beta:g}={fbeta(precision, recall, beta):.2f}")

Run against a three-PR illustration, where a reviewer finds two of four golden issues and adds three comments that match nothing, it prints:

$ python score.py sample.json
PRs=3 tp=2 fp=3 fn=2
precision=0.40 recall=0.50 F1=0.44 F1=0.44
$ python score.py sample.json 2
PRs=3 tp=2 fp=3 fn=2
precision=0.40 recall=0.50 F1=0.44 F2=0.48

The sample file is made-up input, there to show the arithmetic. Your own file is the one that matters.

Using it

Score the reviewer you run today. Then change one thing, such as the review instructions, the model, or a rule that tells it to skip style comments, and score again on the same 20 pull requests. If precision climbs and recall holds, the change is real. If recall drops, the reviewer got quieter, which feels like improvement in a pull request thread and isn’t.

Choose β before you look at the result. A team that merges AI-reviewed code with one human glance wants recall, so β of 2. A team that has stopped reading the bot’s comments wants precision, so β below 1, and probably should start by fixing that.

Twenty pull requests is small. A difference of one or two found bugs is noise, so treat the result as a screen for large effects and add pull requests as you hit new misses. Keep the file in the repository, and every bug that slips past review becomes a new entry.

If you’d rather compare against a public baseline, ReviewBench lets you register your own agent and run it on the 219 pull requests, at the cost of your own model usage. Start with the 20-PR file first. It takes an afternoon, and it’s the only one of the two that contains your code.