tokenise by Rosvetic
Security Playbook

Turn every defect you find in agent code into a Semgrep rule, so the same mistake can't come back

Review comments evaporate; a failing check doesn't. A loop for converting each bug an agent ships into a tested rule that runs on every change. About 20 minutes to try.

13 min read

A matte white cube sheared along one band, with gold light spilling from the break against a dark ground

A reviewer leaves the same comment on the third agent-written pull request this month: “this builds the query from a string”. The agent fixes it, the comment gets resolved, and the lesson lives in a closed thread. Next sprint a different session, with no memory of the thread, writes the same query the same way.

Review is good at finding a defect once. It’s bad at making sure the defect never returns, because the knowledge sits in a person’s head and a resolved comment. A static-analysis rule holds that knowledge in a file, runs in seconds, and doesn’t get tired on the fortieth diff of the week.

This Playbook is a loop: every time review finds a defect in agent-written code, write a Semgrep rule for it, write the rule’s test cases first, and run the rules on every change. The agent can write the rules too, and the test file is what stops it writing a rule that doesn’t work. The setup is TypeScript, runs on a laptop with no account, and takes about 20 minutes. Nothing in it is specific to one agent. It works with any coding agent that can read files and run a shell, and the checks that prove it worked are plain command-line output.

Why a rule beats a rule in the prompt

The usual response to a recurring defect is a line in the agent’s instructions file: “never build SQL from strings”. That helps, and it costs context on every session. It also has no check behind it. If the agent ignores it, or a different agent reads a different file, nothing fails.

A rule moves the enforcement out of the prompt and into the build. The agent can still write the bad code. It just can’t land it, and the failure message tells it what to do instead. The message is the part that matters most, because the next thing that reads it is usually the agent that wrote the problem, working from the CI log or the terminal.

Semgrep suits this because its rules look like the code they match. A pattern for db.query(…${$X}…) is a line you can read. There’s no AST API to learn, which makes rules cheap enough to write for a defect you’ve only seen once.

Install and the three defects

Semgrep is third-party, so some caution first. It’s an open-source command-line tool (the package metadata lists LGPL-2.1-or-later) that runs on your machine. It sends usage metrics by default when you pull rules from its registry or log in. This setup uses only local rule files and passes --metrics=off, which the Semgrep docs say means metrics are never sent. Read the install instructions and the licence before adding it to a company laptop.

python3 -m venv .venv-semgrep
.venv-semgrep/bin/pip install semgrep
.venv-semgrep/bin/semgrep --version

1.184.0

The demo is one TypeScript file with three defects that agents produce regularly, each with a safe twin beside it:

// src/users.ts
import { randomBytes } from "node:crypto";
import { exec, execFile } from "node:child_process";

export interface Db {
  query(sql: string, params?: unknown[]): Promise<unknown[]>;
}

export async function findUser(db: Db, email: string) {
  return db.query(`SELECT id, email FROM users WHERE email = '${email}'`);
}

export async function findUserSafe(db: Db, email: string) {
  return db.query("SELECT id, email FROM users WHERE email = $1", [email]);
}

export function newResetToken(): string {
  return Math.random().toString(36).slice(2);
}

export function newResetTokenSafe(): string {
  return randomBytes(24).toString("base64url");
}

export function archiveLogs(dir: string) {
  return exec(`tar -czf logs.tgz ${dir}`);
}

The first is SQL injection through a template literal. The second is a password reset token drawn from Math.random(), which is predictable and isn’t meant for secrets. The third passes a caller-supplied directory into a shell.

All three compile. All three pass a test that checks the happy path. A reviewer skimming a 600-line diff can miss any of them, and so can an agent asked to review its own work, because the code looks like ordinary code.

Step 1: write the test cases before the rule

The order matters. Write the file of examples first, marking the lines the rule must flag with ruleid: and the lines it must leave alone with ok:. Semgrep runs these annotations with --test, and the annotation goes on the line above the code it describes. The rule id after the colon is the rule being tested.

// rules/agent-defects.ts
declare const db: { query(sql: string, p?: unknown[]): Promise<unknown[]> };
declare const email: string;
declare function exec(cmd: string): void;
declare function execFile(cmd: string, args: string[]): void;

// ruleid: sql-built-from-string
db.query(`SELECT * FROM users WHERE email = '${email}'`);
// ruleid: sql-built-from-string
db.query("SELECT * FROM users WHERE email = '" + email + "'");
// ok: sql-built-from-string
db.query("SELECT * FROM users WHERE email = $1", [email]);
// ok: sql-built-from-string
db.query("SELECT * FROM users " + "WHERE active");

// ruleid: math-random-for-secret
const t = Math.random().toString(36);

// ruleid: shell-command-from-template
exec(`tar -czf logs.tgz ${email}`);
// ok: shell-command-from-template
execFile("tar", ["-czf", "logs.tgz", email]);

The ok: cases carry as much weight as the ruleid: ones. A rule that flags everything passes every positive test and is useless in a week, because people start ignoring it. The concatenation of two string literals is there to prove the rule doesn’t punish a harmless line break in a long query.

The test file sits next to the rule file and shares its base name, so agent-defects.yml is tested by agent-defects.ts. Semgrep pairs them by name.

Step 2: write the rules

# rules/agent-defects.yml
rules:
  - id: sql-built-from-string
    languages: [typescript]
    severity: ERROR
    message: >-
      SQL assembled from a template literal or concatenation. Pass values as
      parameters: db.query("... $1", [value]).
    patterns:
      - pattern-either:
          - pattern: $DB.query(`...${$X}...`)
          - pattern: $DB.query($A + $B)
      - pattern-not: $DB.query("..." + "...")

  - id: math-random-for-secret
    languages: [typescript]
    severity: ERROR
    message: >-
      Math.random() is not a secret source. Use randomBytes() or
      crypto.randomUUID() from node:crypto for tokens, ids and nonces.
    pattern: Math.random()
    paths:
      exclude:
        - "*.test.ts"

  - id: shell-command-from-template
    languages: [typescript]
    severity: ERROR
    message: >-
      Shell command built from a template literal. Use execFile("tar", [args])
      so the input is never parsed by a shell.
    pattern: exec(`...${$X}...`)

Read the patterns as code with holes in them. $DB and $X are metavariables that match any expression, and ... matches any run of characters or arguments. The pattern-not line carves the literal-plus-literal case out of the concatenation pattern.

Each message names the fix and says what to call instead. That’s deliberate: it’s the sentence the agent reads when the check fails, so it should be an instruction and not a diagnosis.

Now run the tests:

.venv-semgrep/bin/semgrep --metrics=off --test rules

3/3: ✓ All tests passed
No tests for fixes found.

Every annotation held. If an ok: line gets flagged or a ruleid: line doesn’t, the run says so and names the line.

Step 3: run the rules on the code

.venv-semgrep/bin/semgrep --metrics=off --config rules/agent-defects.yml --error src

Trimmed output (the box-drawing banner and summary are removed):

src/users.ts
   rules.sql-built-from-string
          SQL assembled from a template literal or concatenation. Pass values as
          parameters: db.query("... $1", [value]).
            9┆ return db.query(`SELECT id, email FROM users WHERE email = '${email}'`);
   rules.math-random-for-secret
          Math.random() is not a secret source. Use randomBytes() or
          crypto.randomUUID() from node:crypto for tokens, ids and nonces.
           17┆ return Math.random().toString(36).slice(2);
   rules.shell-command-from-template
          Shell command built from a template literal. Use execFile("tar", [args])
          so the input is never parsed by a shell.
           27┆ return exec(`tar -czf logs.tgz ${dir}`);

Three findings, one per defect, and none on the safe twins. With --error the process exits with code 1 when there’s any finding, which is what a pre-push check or a CI step needs. After the three fixes (a parameter, randomBytes, execFile), the same command prints nothing and exits 0.

Step 4: the rule that misses, and what to do about it

A pattern rule matches shapes of code, so it has blind spots. This is where most rule collections go wrong: they pass their tests, ship, and then the agent writes the same defect in a different shape.

// src/miss.ts
import type { Db } from "./users.js";

export async function findByName(db: Db, name: string) {
  const sql = `SELECT id FROM users WHERE name = '${name}'`;
  return db.query(sql);
}

This is the same injection, but the string is built on one line and passed on the next. Run the rules and miss.ts produces zero findings. Naming an intermediate sql variable reads as tidy code, so it’s an easy shape for generated code to take.

The fix is the discipline the whole Playbook rests on: add the miss to the test file first, watch it fail, then widen the rule. Add a ruleid: case for the two-line form, and an ok: case for the parameterised version of the same shape:

function byName(name: string) {
  // ruleid: sql-built-from-string
  const sql = `SELECT id FROM users WHERE name = '${name}'`;
  return db.query(sql);
}

function byNameSafe(name: string) {
  // ok: sql-built-from-string
  const sql = "SELECT id FROM users WHERE name = $1";
  return db.query(sql, [name]);
}

Run --test and the ruleid: case fails, because the rule can’t see it yet. Then add a multi-statement pattern to the pattern-either list:

          - pattern: |
              $SQL = `...${$X}...`;
              ...
              $DB.query($SQL)

The ... between the statements matches whatever sits between the assignment and the call. Run the tests and then the scan again:

3/3: ✓ All tests passed

src/miss.ts
   rules.sql-built-from-string
            4┆ const sql = `SELECT id FROM users WHERE name = '${name}'`;
            5┆ return db.query(sql);

Semgrep also has a taint mode, which follows a value from a source to a sink across statements, and it’s the tool for flows this pattern can’t express: a value passed through a helper function or built in a loop. It’s also fiddlier. A first attempt at a taint rule for this same file flagged the safe parameterised call as well, and fixing that takes more tuning than the pattern did. Start with patterns, keep the defects you can name, and move to taint mode for the one that keeps slipping through.

Step 5: let the agent write the rules, with the tests as its brief

Writing rules by hand for every review finding is the part people skip. Hand it to the agent, with a prompt that makes the test file the contract:

Review found this defect in src/users.ts:

  db.query(`SELECT ... '${email}'`)

Write a Semgrep rule for it in rules/agent-defects.yml and the test cases in
rules/agent-defects.ts. Rules:

1. Write the test cases first. At least two `ruleid:` cases (including one
   where the string is built on a separate line from the call) and two `ok:`
   cases that look similar but are safe.
2. Run `semgrep --metrics=off --test rules` and show it failing for the
   right reason before you write the rule.
3. Write the rule. The message must say what to call instead.
4. Run the tests again and run the scan against src/. Report every finding
   and any file where you expected one and got none.
5. Do not edit src/ in this change. The rule lands first, the fixes in a
   separate commit.

Step 2 is the guard. An agent that writes a rule and its tests together will make them agree with each other, which proves nothing. Forcing the failing run first shows that the test cases can fail, and the ok: cases protect against a rule that matches too much.

Step 5 keeps the rule change reviewable. A diff containing both a new rule and every fix it triggers is hard to read, and the reviewer ends up approving the rule on faith. Two commits means the first one can be judged on whether the examples are right.

Step 6: run it where the code lands

A rule nobody runs is a comment. Two places cover most teams.

Before push, as a one-liner in whatever mechanism you already use for local checks (a git pre-push hook, Husky, a task runner):

.venv-semgrep/bin/semgrep --metrics=off --config rules --error --quiet src

And in CI, so a change that skipped the local check still fails. This workflow is documented behaviour and wasn’t run here:

name: agent-defect-rules
on: pull_request
jobs:
  semgrep:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: "3.13"
      - run: pip install semgrep
      - run: semgrep --metrics=off --test rules
      - run: semgrep --metrics=off --config rules --error src

The two steps are deliberately separate. The first tests the rules themselves, so a bad edit to a rule fails the build before it can silently stop catching anything. The second applies them. Pin the Semgrep version in a requirements file if you want identical results week to week.

Suppressions: the escape hatch, and its cost

Some matches are legitimate. Math.random() is fine for jitter on a retry delay. Semgrep takes a nosemgrep comment on the same line or the line above, optionally followed by a rule id:

// nosemgrep: math-random-for-secret
const jitter = Math.random() * 100;

That file scans clean and exits 0. The danger is that an agent facing a failing check may reach for the suppression comment as the quickest way to green. Two defences: grep for it in review (git diff | grep nosemgrep is a ten-second check), and add a rule that flags nosemgrep itself outside an approved folder if the problem actually shows up. Treat a new suppression like a new any cast: allowed, but someone should have to look at it.

Make it yours

  • C# and .NET: Semgrep lists C# among its supported languages, so the same loop applies in principle to string.Format into SqlCommand, Random used for tokens, or Process.Start with a concatenated command (write the ruleid: cases first and see what matches, since this setup was only run on TypeScript). If the team already has Roslyn analyzers, put the type-aware checks there and keep Semgrep for fast syntactic ones.
  • Team rules, not personal ones: keep rules/ in the repo root and review changes to it like production code. A rule is policy, and a silent edit to one is a silent policy change.
  • Severity: start new rules at WARNING and promote to ERROR after a week without false positives, so a bad pattern costs noise and not a blocked release.

What goes wrong

Patterns age. A rule written for db.query stops matching the day the team switches to a query builder, and nothing tells you. The positive test cases pass, because they still contain the old call. Add a case for each new data-access style when it arrives, and keep the rule file short enough that someone reads it.

False positives destroy trust faster than misses cost you. Ten spurious findings in the first week and people will suppress by reflex. Write ok: cases for every near miss you think of, and delete a rule that can’t be made precise.

Static rules catch what you’ve already seen. They do nothing for the defect nobody has met yet. This is a ratchet that stops regression, and it’s a complement to review and to tests, not a replacement for either.

Don’t bother if you have two services and one reviewer who remembers everything. The loop pays off when the knowledge outgrows one head: several reviewers, several agent sessions a day, or defects that keep returning.

The first 20 minutes

Pick the last defect a reviewer flagged in agent-written code that wasn’t a one-off. Install Semgrep into a virtualenv, write two ruleid: and two ok: cases for it, run semgrep --test and watch it fail, then write the rule and watch it pass. Run it over src/ with --error, and wire the same command into the check you already run before pushing.