tokenise by Rosvetic
Development How-to

A new hire's first week with an agent: a repo tour you can check against the code

An agent will happily tell a new starter where everything lives, including files that don't exist. A 20-line script and four questions catch the wrong answers. About 20 minutes to set up.

8 min read

A small clay town map of rounded charcoal and cream blocks with a teal path winding to a flag, and one peach block off the path

A new starter opens the repo on Monday, asks the agent for a tour, and gets a confident page of Markdown. It names the service that places orders, the class that retries failed payments and the folder where the tests live. It reads like it was written by someone who has worked there for years.

Two of those three claims are probably right. The third is a class that has never existed in this codebase, and the new starter has no way to know which one it is. They’ll spend a morning searching for RetryPolicy, conclude the problem is them, and stop asking.

The fix is cheap: treat the tour as a set of claims and check the claims mechanically before anyone reads it. This piece builds a small checker, shows what it catches and what it can’t, and gives four questions for the things a script can’t see. It works with any coding agent that can read files and run a shell, and with no agent at all, because the check is a script and the questions are plain English.

What the tour is for

The tour isn’t documentation. It’s a map for the first two days, so a new hire can find the order flow, the payment boundary and the tests without asking someone who’s busy.

That use has one requirement above all others: every concrete thing in it must exist. A vague tour wastes an hour. A tour with a made-up file name wastes a morning and teaches the new starter to distrust the repo, the agent or themselves.

Agents get this wrong in a predictable way. They infer from names. A payments folder with a gateway class suggests a retry policy, because most payment code has one, and the model fills the gap with something plausible. The same inference produces invented config keys, invented base classes and invented test helpers.

The demo repo

Here’s a deliberately small C# project. It’s enough to show both kinds of error.

shop/
  src/Orders/OrderService.cs
  src/Billing/StripeGateway.cs
  tests/OrderServiceTests.cs

The service places an order. Read it carefully, because the order of the two awaits matters later.

namespace Shop.Orders;

public class OrderService(IOrderRepository repo, IPaymentGateway payments)
{
    public async Task<Order> PlaceAsync(NewOrder request)
    {
        var order = Order.From(request);
        await payments.ChargeAsync(order.Total);
        await repo.SaveAsync(order);
        return order;
    }
}

Now the tour an agent might write, saved as TOUR.md:

# Tour

Orders are placed in `src/Orders/OrderService.cs` by `PlaceAsync`, which saves the order and then charges the card.
Payments go through `StripeGateway` in `src/Billing/StripeGateway.cs`.
Retries are handled by `RetryPolicy` in `src/Billing/RetryPolicy.cs`.
Order tests live in `tests/OrderServiceTests.cs`.

It has two errors. RetryPolicy doesn’t exist. And the first sentence says the order is saved and then the card is charged, when the code charges first and saves second. One error is about existence and one is about behaviour. They need different checks.

Check one: does every named thing exist

The first check is a script. It reads the tour sentence by sentence, takes everything in backticks, and treats anything containing a slash as a path and anything else as an identifier. Paths must be files. Identifiers must appear somewhere in the source.

#!/usr/bin/env python3
"""Check a repo tour: every backticked path must exist, every backticked
identifier must appear in the repo's source files."""
import re, sys, pathlib

root = pathlib.Path(sys.argv[1])
tour = pathlib.Path(sys.argv[2]).read_text()
source = {p: p.read_text() for p in root.rglob("*") if p.suffix in {".cs", ".ts"}}
failures = 0
for sentence in re.split(r"(?<=\.)\s+", tour):
    ticks = re.findall(r"`([^`]+)`", sentence)
    paths = [t for t in ticks if "/" in t]
    names = [t for t in ticks if "/" not in t]
    for p in paths:
        if not (root / p).is_file():
            print(f"MISSING FILE  {p}"); failures += 1
    for n in names:
        if not any(n in text for text in source.values()):
            print(f"MISSING NAME  {n}"); failures += 1
print(f"{failures} unverified claim(s)")
sys.exit(1 if failures else 0)

Save it as check_tour.py, then run it against the demo repo:

python3 check_tour.py . TOUR.md

It prints:

MISSING FILE  src/Billing/RetryPolicy.cs
MISSING NAME  RetryPolicy
2 unverified claim(s)

The exit code is 1, so it works as a gate. Extend the suffix set for your languages. For a TypeScript repo, add .tsx; for a monorepo with a lot of generated code, exclude the generated folders, or every identifier will “exist” in a bundle somewhere and the check will pass for the wrong reason.

The loop with the agent is then mechanical. Give it the script’s output and ask for a corrected tour, with one rule:

The tour you wrote has these unverified claims:

MISSING FILE  src/Billing/RetryPolicy.cs
MISSING NAME  RetryPolicy

For each one, search the repo and either correct the claim or delete the sentence.
Do not replace a missing name with a guess. If retries are not implemented, say
"No retry logic exists in this repo." Then run `python3 check_tour.py . TOUR.md`
and show me the output.

The last sentence of that prompt matters. The agent has to run the check itself and show the result, so the check is evidence and not a promise.

What the script can’t see

Delete the retry sentence and run it again:

0 unverified claim(s)

Clean. And the tour still tells the new hire that orders are saved before the card is charged, which is backwards. In a payments flow that’s the difference between “a failed charge leaves no order” and “a failed save leaves a customer charged for nothing”. It’s exactly the kind of thing a new starter will build a mental model on.

Existence checks verify nouns. They can’t verify verbs, order, conditions or relationships, and those are most of what makes a tour useful. Closing that gap takes a person, but a person with the right questions can do it in minutes.

Four questions that expose a wrong answer

These are for the new starter to ask, one at a time, in a fresh session, after the tour exists. Each has an answer they can verify in the repo without trusting the agent.

  1. Show me the exact lines. “Quote the lines of PlaceAsync that save the order and that charge the card, with line numbers.” A model that’s summarising from memory will paraphrase. A model that has read the file will quote, and the line numbers can be checked in the editor in ten seconds. In the demo, lines 8 and 9 settle the ordering argument immediately.

  2. What happens if it fails? “If ChargeAsync throws, what state is the database in?” This forces the model to trace the behaviour instead of describing the structure. The answer is checkable: set a breakpoint, or write a three-line test.

  3. Who calls this? “List every caller of PlaceAsync.” The new hire then runs a find-usages in the IDE and compares. A missing caller means the agent didn’t search, and an extra one means it invented a path.

  4. What would you check if you weren’t sure? This one is for the agent’s confidence. A good answer names files and commands. A weak answer restates the claim more firmly. If the agent can’t say how to verify its own tour sentence, treat that sentence as unverified.

None of the four needs a particular agent. They need the ability to read the repo and nothing else.

Making it a first-week exercise

The best use of this is not for the lead to run it once and hand over a polished tour. It’s for the new starter to do it, because checking the tour is a guided walk through the code with a reason to read each file.

A workable first-week shape:

  • Day one: the new hire asks the agent for the tour, saves it as TOUR.md, and runs the checker. Every failure is a small puzzle (“where is retry logic really handled?”) that sends them into the code.
  • Day two: they pick the three sentences about behaviour they care about most and run the four questions on each. Any disagreement between the agent and the code goes in a list.
  • End of week: they send the list to the team. Each item is either a bug in the tour, a surprise in the code, or a missing test. All three are useful to the team, and the third is the best onboarding output there is.

The corrected tour can be committed, but treat it as dated. Put the commit hash it was checked against at the top, and re-run the checker in CI so a rename breaks the build instead of silently making the tour wrong. A single step is enough:

- name: Check repo tour
  run: python3 scripts/check_tour.py . docs/TOUR.md

What goes wrong

The checker passes common words. If the tour backticks Order and any file in the repo contains the word “Order”, the check is satisfied. Prefer specific identifiers such as PlaceAsync over general ones, and tighten the script to match whole words if the repo is large.

The agent can game it. If told only “make the check pass”, it may delete every concrete claim and leave a tour that says nothing. Read the corrected tour. A tour that’s shorter and vaguer isn’t more accurate.

A passing checker also says nothing about omissions. The tour may skip the part of the system where the surprises are, such as a background job nobody mentions. Ask the new hire to find one thing the tour doesn’t cover by the end of the week.

Don’t bother with any of this for a repo small enough to read in an afternoon. The method pays off when there are enough files that a new hire can’t hold the map in their head and an agent can plausibly invent a corner of it.

First 20 minutes

Ask your agent for a tour of the repo you know best, with every file and class in backticks. Save it, run the 20 lines of Python above against it, and see how many unverified claims come back before you read a word of the prose.