tokenise by Rosvetic
Development How-to

Write a REVIEW.md that quietens Claude Code Review, then gate merges on what it still finds

The review bot never fails its own check, so a repo that wants a hard stop needs a file that sets the bar and a 40-line CI job that reads the result. About 20 minutes.

10 min read

A brass-rimmed loupe on a ruled page in the dark, pink halftone dots filling the lens and fading outwards

A review bot that posts eleven comments on a three-line change gets muted by the end of the week. A bot that posts nothing on the one change that touches billing gets trusted, which is worse. Claude Code Review, the managed pull request reviewer for Team and Enterprise organisations, is tuned by a single Markdown file in the repo, and the defaults are written for production code in general, not yours.

This piece covers three things: a REVIEW.md that sets the bar for your repo, the branch rule that decides whether the file is read at all, and a small CI job that fails a pull request when Important findings are still open. The check run the service posts never fails by itself, so the gate is a few lines of your own.

Everything below assumes an Owner has already set the service up, as the Code Review docs describe.

What the reviewer reads, and what it ignores

According to the customisation docs , two files steer a review. CLAUDE.md is project context that Code Review reads, and a change that violates one of its rules becomes a finding. REVIEW.md is for review only: what to flag, at what severity, and what to skip. The agents that find issues and the agents that verify them both receive its contents, and the step that writes the final review consults it before settling severity.

A REVIEW.md at the repository root applies everywhere. One in a subdirectory, say services/billing/REVIEW.md, adds checks for changes under that path on top of the root file.

The local /code-review command is a different tool and reads CLAUDE.md only, which the earlier piece on reviewing locally before a push covers. So a rule you want in both places goes in CLAUDE.md, and REVIEW.md keeps the rules that only make sense on a pull request.

Start from the five problems, not from a template

REVIEW.md is freeform Markdown. It changes what a review finds and reports, not the format of the review. The docs map each common complaint to an instruction, and that mapping is the best way to build the file: write a section only when you’ve seen the problem.

  1. The wrong things are marked Important. Redefine what Important means for the repo. The default calibration targets production code, so a docs repo or a prototype wants a narrower definition.
  2. Too many nits. Cap them.
  3. Comments on generated files, or on what CI already checks. List the paths and categories to skip.
  4. A team rule isn’t being checked. State it.
  5. Findings that turn out to be wrong. Require evidence before a class of finding is posted.

Here’s a file for a C# API service that uses all five. It’s adapted from the shape of the example in the docs, with repo-specific details swapped in.

# Review instructions

## What Important means here

Reserve Important for findings that would break behaviour, leak data,
or block a rollback: incorrect logic, queries not scoped to the
caller's tenant, personal data in logs or exception messages, and EF
Core migrations that aren't backward compatible. Naming, style and
refactoring suggestions are Nit at most.

## Cap the nits

Report at most five Nits per review.

## Do not report

- Anything CI already enforces: formatting, analyser warnings, type errors
- Generated files under `src/Generated/` and `packages.lock.json`
- Test-only code that intentionally breaks production rules
- In `scripts/`, only report if near-certain and severe

## Always check

- New controller actions have an integration test
- Log statements don't include email addresses, user IDs or request bodies
- Every query against `Orders` filters by `TenantId`

## Evidence

A claim about runtime behaviour needs a `file:line` citation in the
source, not an inference from a method name.

A few choices in there are worth copying. The Important definition is a list of consequences (breaks behaviour, leaks data, blocks a rollback), which gives the reviewer a test it can apply to a finding instead of a mood. The skip list names what CI already enforces, because a duplicate comment costs the author a round trip for no information. And the evidence rule goes after the false positives that look plausible, which are the expensive kind.

Merge it before you test it

This is the step that catches people. The docs say Code Review reads REVIEW.md from the pull request’s base branch or from the default branch, never from the pull request’s own changes. A pull request that adds or edits REVIEW.md is reviewed under the old rules.

The practical loop is: merge the REVIEW.md change on its own, open a throwaway pull request that should trigger the new rules, and look at what comes back. Changing the file and the code it’s meant to govern in one pull request tests nothing.

The logic, in this author’s reading: a file the pull request could rewrite would let a change tell its own reviewer to skip it. The docs also say to protect the default branch. When the review reads the copy on a protected default branch, it honours three kinds of instruction from the root file: lowering a finding’s severity, excluding paths, files or classes of issue, and capping nit comments.

What counts as protected is specific. On the default branch, a classic branch protection rule or a ruleset both count. A ruleset counts when it restricts updates, or requires pull requests with at least one approval that is dismissed on a new push (or must come after the last push). If Code Review can’t confirm the protection settings, the table in the docs says it reads no REVIEW.md at all, so a file that seems to be ignored is a branch settings question first and a wording question second. The docs also have a troubleshooting entry for when the check run mentions REVIEW.md.

Subdirectory files add one more condition: the table shows them being read when the default branch is protected, or when the base branch is protected by a ruleset, and not in the unprotected cases. If a subdirectory file matters to you, protect the branch first.

The check run won’t stop a merge

Each review leaves a check run named Claude Code Review. Its conclusion is neutral, or skipped when an automatic review had nothing to look at. It never concludes as a failure, so adding it to required status checks gates nothing. The docs state this plainly: a review never approves or blocks the pull request.

Two built-in levers exist. Branch protection that requires conversation resolution makes each finding’s thread something the author has to resolve before merging. To dismiss a finding without a code change, resolve the thread, because replying alone doesn’t dismiss it.

The other lever is your own job. The check run’s Details text ends with a machine-readable line, bughunter-severity: {...}, holding a count per severity. Per the docs, the normal key is the count of Important findings, and the counts include findings from earlier reviews of the pull request whose threads are still open. That last part is useful: a fixed finding that the next review resolved drops out, and an ignored one keeps the gate red.

The gate

The script below waits for a completed Claude Code Review check run on the commit, reads the severity line and fails when Important findings are open. It shells out to the gh CLI, so the runner needs it and a GH_TOKEN.

#!/usr/bin/env python3
"""Fail the job when Claude Code Review reports Important findings."""
import json, os, subprocess, sys, time

REPO, SHA = os.environ["REPO"], os.environ["SHA"]
NAME = "Claude Code Review"
MARKER, END = "bughunter-severity: ", " -->"


def gh(path):
    out = subprocess.run(["gh", "api", path], capture_output=True, text=True, check=True)
    return json.loads(out.stdout)


def find_run():
    for _ in range(int(os.environ.get("TRIES", "40"))):
        runs = gh(f"repos/{REPO}/commits/{SHA}/check-runs")["check_runs"]
        done = [r for r in runs if r["name"] == NAME and r["status"] == "completed"]
        if done:
            return done[-1]
        time.sleep(float(os.environ.get("WAIT", "15")))
    return None


run = find_run()
if run is None:
    print("no completed review on this commit: no verdict")
    sys.exit(2)

text = gh(f"repos/{REPO}/check-runs/{run['id']}")["output"]["text"] or ""
if MARKER not in text:
    print("review finished without a severity line: no verdict")
    sys.exit(2)

counts = json.loads(text.split(MARKER)[1].split(END)[0])
print("severity counts:", json.dumps(counts, sort_keys=True))
if counts.get("normal", 0) > 0:
    print(f"{counts['normal']} Important finding(s) still open")
    sys.exit(1)
print("no open Important findings")

The exit codes are deliberate. A missing severity line is not zero findings: the docs say a review that failed, timed out, was cancelled or was superseded has no such line, and some completed reviews have none either. Exit 2 means no verdict, and what your workflow does with that is a policy choice. Treating it as a failure blocks merges whenever the service hiccups, and treating it as a pass lets an unreviewed change through.

To exercise the logic without a live review, point a stand-in gh at it that serves payloads shaped like the documented check run. These are constructed payloads, so the run proves the script’s branches and nothing about the service. Run it with four cases and it prints:

== completed {"nit": 1, "normal": 2, "pre_existing": 0}
severity counts: {"nit": 1, "normal": 2, "pre_existing": 0}
2 Important finding(s) still open
exit 1
== completed {"nit": 3, "normal": 0, "pre_existing": 1}
severity counts: {"nit": 3, "normal": 0, "pre_existing": 1}
no open Important findings
exit 0
== in_progress {}
no completed review on this commit: no verdict
exit 2
== completed <no line>
review finished without a severity line: no verdict
exit 2

Three nits and a pre-existing bug pass. Two Important findings fail. That split is the whole point of caring about the severity calibration in REVIEW.md: the gate is only as sensible as the definition of Important behind it.

Wiring it into a workflow

A GitHub Actions job that runs it looks like this. The checks: read permission is one of the documented GITHUB_TOKEN permission keys, and it’s what lets the token list check runs.

name: review-gate
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]

permissions:
  checks: read

jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - run: python3 .github/scripts/review_gate.py
        env:
          GH_TOKEN: ${{ github.token }}
          REPO: ${{ github.repository }}
          SHA: ${{ github.event.pull_request.head.sha }}

Two details matter. The job reads the head commit of the pull request, so a new push produces a new wait for a new review. And the timeout-minutes is a ceiling on how long a slow review can hold the job, which is the reason the script polls a fixed number of times instead of forever. Add the job to required status checks and the neutral review check becomes a real gate.

Note that this workflow is yours, not the service’s, and Code Review’s own behaviour still applies. Automatic reviews skip draft pull requests and pull requests from forks, so the gate will report no verdict on those until someone comments @claude review. On a draft that command runs a review, but pushes to a draft don’t start new ones.

What goes wrong

Re-reviews are narrower than the first. A later automatic review focuses on what changed since the last one and posts fewer nits and pre-existing findings. To get a review that covers the whole pull request again, comment @claude review. Anyone with write access can do it, and it has to be a new top-level comment that starts with the command.

If the repo also runs the Claude Code GitHub Action with its default @claude trigger phrase, an @claude review comment starts the Action as well. Set the Action’s trigger_phrase to something else to keep them apart.

Review depth is an Owner setting per repository (Quick, Standard or Deep), and a more thorough review costs more. Pick it with the gate in mind.

Don’t bother with the gate on a repo where reviews are still noisy. Tune REVIEW.md for a couple of weeks first, because a gate on an uncalibrated Important level teaches the team to resolve threads reflexively. And whatever the gate says, the docs are explicit that Code Review can miss bugs and a review with no findings doesn’t mean the code is error-free. It’s one check among several.

The first 20 minutes

Write the REVIEW.md from the problems you’ve actually seen on recent pull requests, merge it to the default branch, and confirm the branch is protected. Open a small pull request that should produce one nit and one Important finding and see whether the file did what you meant. Only then add the gate job, and run it advisory-only for a week before it becomes a required check.