675 security pull requests from coding agents: the flaws were regexes, shell strings and file paths

A study of 675 security-related pull requests written by coding agents found that most of the flaws a static scanner could see came from a short list: regular expressions that can blow up, shell commands built from strings, and file paths built from user input. Those three account for roughly 60% of all findings. They’re also the three things a reviewer skimming an agent’s diff is least likely to stop on, because each one looks like ordinary code.

The paper is “Insights into Security-Related AI-Generated Pull Requests” (Rabbi, Turzo, Champa and Zibran, arXiv 2604.19965 , submitted 21 April 2026 and accepted at EASE 2026). It’s one study with real limits, which are covered below. But the shape of its results matches what’s cheap to check, so it’s a good prompt for rearranging where review time goes.

What the authors did

They started from the AIDev dataset, which collects more than 456,000 pull requests opened by Codex, Devin, GitHub Copilot, Cursor and Claude Code. They used the version last updated on 1 August 2025, kept repositories with more than 100 stars, and ended up with 33,596 agent PRs across 2,807 repositories.

A keyword filter for security terms (injection, path traversal, CVE, sanitize and so on) found 1,047 candidates, 3.1% of the total. gemini-2.0-flash then judged each PR’s title and body, which cut the set to 675. Two authors labelled a sample by hand to check the model. Then Semgrep, with default rules, scanned the code those 675 PRs introduced.

Of the 675, 354 were merged (52.4%), 219 were closed without merging (32.4%) and 102 were still open (15.1%).

What Semgrep found

104 of the 675 PRs (15.4%) introduced at least one finding, and the scanner reported 853 findings in total. The median among affected PRs was three. One PR had 109, and it was rejected.

Six weaknesses made up over 80% of the findings (Table 3 in the paper):

Two things stand out. First, nothing exotic: these are the bugs that appear in every secure-coding checklist. Second, the agents that were studied differed in which ones they produced, and flagged PRs showed up among the merged ones for every agent in the paper’s Table 4. Several of those merged PRs carried command injection or path traversal findings. The authors read this as review effort not matching security risk: the same dataset shows PRs rejected for style or missing tests.

What it can’t tell you

Read the 15.4% carefully. It’s the share of security-tagged agent PRs that tripped a default Semgrep rule. It isn’t the rate for agent code in general, and it says nothing about how humans compare, because the study has no human baseline.

The authors list their own caveats. Semgrep default rules on a single tool miss things and raise false alarms, and an alert isn’t a proven exploit. The manual check covered only the 675 PRs the model kept, so it can catch false positives but not security PRs the filter or the model missed. The data stops at August 2025 and covers five agents, so it says little about current models.

One inconsistency is worth knowing about: the paper gives Cohen’s kappa for the manual labelling as 0.79 in the methods section and 0.62 in the threats section. Treat the model-labelled set as moderately validated.

The regex count is also lumpy. A single rule family can generate hundreds of findings from a handful of PRs, and the paper’s Table 4 shows 289 of the CWE-1333 hits sitting in one outcome group. The ranking is a prompt to look, not a measurement of how often an agent writes a bad regex.

Three checks worth running

Each of these takes a minute on a diff and works the same whichever agent wrote it. The examples are Node 22 and print what’s shown when run.

Nested quantifiers in a regex

An email-style pattern with a quantifier inside a quantified group looks fine in review and fails on a near-miss input:

const risky = /^([a-zA-Z0-9]+\.?)+@example\.com$/;
const safe = /^[a-zA-Z0-9]+(?:\.[a-zA-Z0-9]+)*@example\.com$/;

for (const n of [24, 28, 32, 34]) {
  const input = "a".repeat(n) + "!";
  for (const [name, re] of [["risky", risky], ["safe", safe]] as const) {
    const t = process.hrtime.bigint();
    re.test(input);
    const ms = Number(process.hrtime.bigint() - t) / 1e6;
    console.log(`${name.padEnd(5)} n=${n} ${ms.toFixed(1)} ms`);
  }
}

Run it and it prints (timings vary by machine):

risky n=24 1.4 ms
safe  n=24 0.0 ms
risky n=28 3.9 ms
safe  n=28 0.0 ms
risky n=32 66.9 ms
safe  n=32 0.0 ms
risky n=34 245.6 ms
safe  n=34 0.0 ms

In this run the cost roughly doubles with every extra character, while the safe pattern stays flat. The review rule: any regex applied to input a user controls gets a test with a long near-match string, and a regex with a group inside a + or * needs a reason.

A shell command built from a string

If the diff contains exec, execSync, system or Process.Start with an interpolated string, the question is what happens when the value contains a semicolon:

import { execSync, execFileSync } from "node:child_process";

const branch = "main; echo INJECTED";
console.log("string :", execSync(`echo checking ${branch}`).toString().trim().replace(/\n/g, " | "));
console.log("argv   :", execFileSync("echo", ["checking", branch]).toString().trim());

string : checking main | INJECTED
argv   : checking main; echo INJECTED

The string version ran a second command. The argument-array version treated the whole value as one argument. In C#, the equivalent is ProcessStartInfo.ArgumentList instead of a concatenated Arguments string.

A path joined from user input

path.join(root, name) doesn’t stop name from climbing out of root. Resolve first, then check the result is still inside:

import path from "node:path";

const root = path.resolve("/srv/uploads");
const naive = (name: string) => path.join(root, name);
function guarded(name: string): string {
  const full = path.resolve(root, name);
  if (full !== root && !full.startsWith(root + path.sep)) throw new Error("outside root");
  return full;
}

"report.pdf"                   naive -> /srv/uploads/report.pdf      guarded -> /srv/uploads/report.pdf
"../../etc/passwd"             naive -> /etc/passwd                  guarded -> REJECTED: outside root
"../uploads-old/secret.txt"    naive -> /srv/uploads-old/secret.txt  guarded -> REJECTED: outside root

The third row is the one reviewers miss. A bare startsWith(root) check would let /srv/uploads-old through, which is why the separator is part of the test. In ASP.NET Core the same pattern is Path.GetFullPath followed by a prefix check that includes the trailing separator.

Make the scanner do the first pass

The paper’s own suggestion is lightweight static checks or automated severity scoring to triage agent PRs, so reviewers spend attention where the risk is. The pieces already exist. Semgrep’s default rules were enough to find the 853 findings in the study, so running them on every agent PR in CI is the lowest-effort version. For the defects your own team keeps seeing, turn each one into a custom rule .

Another finding is about feedback: 38.8% of the security PRs that were closed got no explanation. If you reject an agent’s change, a one-line reason in the PR (or in the instruction file the agent reads) is the only way the next attempt improves.

Try it this week: take the last three agent PRs your team merged, search each diff for exec, system, Process.Start, new RegExp, Path.Combine and path.join, and read just those lines first.