Test a Claude Code skill like code: claude plugin eval, a no-plugin baseline and a CI gate

A skill is a Markdown file that changes how an agent behaves, and it has no compiler and no test runner. Reword one line of its instructions and the next session behaves a little differently, or a model update changes how readily Claude picks the skill at all. Nothing goes red.

claude plugin eval is the missing test runner. It ships in Claude Code (the output below is from 2.1.304, and the docs set 2.1.269 as the minimum). Each case is a prompt plus pass/fail graders. Every case runs several times with the plugin and several times without it, so the report separates “the answer was right” from “the plugin is why”.

This piece builds a two-case suite for a small commit-message skill, shows what a regression looks like when it’s caught, and ends with the CI command.

What it costs and what it needs

Every run is a real model call on your account, and so is every llm grader. The docs describe the cost figure the command prints as a list-price estimate, and say it counts against plan usage or the API bill. A suite makes roughly cases × runs agent runs with the plugin, the same number again for the baseline, plus three short judge calls per llm grader per run.

You need git 2.31 or later if git is installed (older versions make the command stop before running any case), and a plugin directory with a plugin.json or .claude-plugin/plugin.json.

The plugin under test

The plugin is one skill. Here’s the layout:

commit-helper/
├── .claude-plugin/plugin.json
└── skills/commit-message/SKILL.md

---
name: commit-message
description: Draft a commit message from a described change. Use when the user asks for a commit message.
---

Write a Conventional Commits message: type(scope): summary, imperative mood, 72 characters or fewer.

plugin.json needs only a name, a version and a description. claude plugin validate . passes on it, with a warning that no author is set.

Scaffold a case

You can let claude plugin eval init interview you and write the suite. To see exactly what a case is made of, scaffold a blank one instead:

claude plugin eval init --bare rename-change

Created evals/rename-change/prompt.md and evals/rename-change/graders/criteria.md

A case is a directory. prompt.md holds the message Claude receives, plus run limits in its frontmatter. Each file in graders/ is one check. The blank template ships a TODO prompt and an llm grader, so it isn’t runnable until you fill it in.

Write the prompt the way a person would type it, without naming the skill. The point is to find out whether the skill gets picked on natural phrasing.

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---

Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.

allowed_tools matters. Runs never stop to ask permission, and any tool that needs a grant you didn’t give is removed from the session. Skill has to be listed or the skill can’t fire. Each run also starts in an empty working directory with only the plugin loaded: your CLAUDE.md, user settings, hooks and other plugins are absent, which is what makes the results repeatable.

Two graders per case: the result and the route

The docs recommend one grader on the output and one on the steps taken. The output check here is a regex, which is deterministic and free:

---
type: regex
pattern: '^(feat|fix|refactor|chore|docs|test)(\([a-z0-9-]+\))?: [a-z].{0,70}$'
flags: m
---

It checks the final message (last_message, the default target) for a line that looks like a Conventional Commits subject. Regexes use JavaScript syntax, so case-insensitivity goes in flags, not an inline (?i).

The route check asserts the skill was actually invoked:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit-message"'
---

The optional prefix in the pattern lets it match both commit-message and the namespaced commit-helper:commit-message form.

There are only six grader types. regex, tool_used, tool_order and file_exists are computed from the transcript and files. llm asks a judge model to vote on a rubric, and baseline compares against a reference transcript. There are no custom-code graders, so anything you’d want to assert with a script has to be expressed as a regex over a file the run produced. The docs suggest asking Claude to run the build or tests inside the run, write the outcome to a file, and grading that file.

A case that should not fire

A skill that triggers on everything is as broken as one that never triggers. The second case sends an unrelated question and asserts the skill stays quiet:

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---

What's the difference between a mutex and a semaphore?

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit-message"'
min: 0
max: 0
arm: both
---

min: 0 and max: 0 is how you assert a tool was never called. The arm: both line needs explaining, which comes next.

Reading the baseline

Run the suite from the plugin root:

claude plugin eval . --trust-plugin --no-publish

The first run in a directory asks Trust this plugin directory?, because the command loads the plugin’s skills, hooks and agents and runs them as you. --trust-plugin answers that in advance, so only pass it for a plugin you’d run on your own machine. --no-publish keeps the HTML report local.

Six runs per case (three with, three without) finished in 76 seconds and printed this summary:

CASE               WITH  W/OUT Δ      RUNS COST    NOTES
rename-change      1.00  0.00  +1.00  6    $0.15   
unrelated-request  1.00  1.00  0.00   6    $0.17   

2 case(s) · mean Δ +0.50 · 76s · $0.32
Report: evals/results/2026-10-20T11-05-44-494Z/report.html

WITH is the score with the plugin, W/OUT the score with no plugin loaded, and Δ the difference. Read the first row as: with the skill Claude produced a conforming subject line every time, and without it never did, so the skill is responsible for the format. Read the second as: the skill stayed out of an unrelated question, but then so did plain Claude, so that case can only ever catch over-triggering. It can’t prove the plugin helped, and Δ 0.00 says so.

One detail trips people up. In a two-arm run, tool_used: Skill graders are reported as indicators and excluded from the score, because a “skill was invoked” check can never pass without the plugin and would drag the baseline to zero. That is why the no-skill grader above sets arm: both: a “must not fire” check is meaningful in both arms. Pass --ablation none to run one arm and score every grader.

What a regression looks like

Now weaken the skill. Replace its body with a looser instruction, leaving the description alone:

---
name: commit-message
description: Draft a commit message from a described change. Use when the user asks for a commit message.
---

Write a friendly commit message that explains the change.

Run just the one case, single arm, with a threshold:

claude plugin eval . --trust-plugin --no-publish --threshold 0.8 --ablation none --case rename-change
echo $?

  rename-change run 3/3: score 0.50  $0.03
    ✗ format (weight 1): pattern not found in last_message
    ✓ skill-fired (weight 1): Skill called 1x (expected 1..∞)
✗ rename-change  score 0.50  (3 runs)  $0.08

CASE           SCORE PASS% RUNS COST    NOTES
rename-change  0.50  0%    3    $0.08   format: pattern not found in last_message

1 case(s) · 54s · $0.08
1

(The output is trimmed to the final run and the summary.) The skill still fires, so a check on invocation alone would have stayed green. The format grader catches that the skill’s instructions no longer produce the shape the team relies on, and the exit code is 1.

Without the baseline the score is 0.50 because half the graders pass. A case passes when its score meets --threshold, which defaults to 1.0. Set it deliberately: at the default, any single flaky run fails the build.

Description changes are the other regression

The docs flag one finding as common: a Δ near zero with the tool_used: Skill grader failing means Claude isn’t choosing the skill on natural phrasing, and the fix is the skill’s description. That is a frontmatter edit, which no linter will flag.

It doesn’t always bite in a small plugin. Replacing the description with “Helps with git text.” still produced a fully passing suite with the same Δ +1.00, because with one skill loaded there was nothing else to pick. The failure shows up when the plugin has several skills with overlapping descriptions, so write a case per skill using phrasing that could plausibly match a neighbour, and add a min: 0, max: 0 case for each neighbour.

Run it in CI

The documented CI invocation pins both models, so a model rollout isn’t mistaken for a plugin regression, and caps spend:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

Substitute model IDs your account can use. The exit codes carry the verdict:

A few details worth knowing before wiring it up:

• --json takes an optional path, so put the target before it. claude plugin eval --json . reads . as the output path and errors that it must end in .json.
• A run that ends on an API error, such as a rate limit, scores 0 without grading and makes the command exit non-zero even if every case passed. Since v2.1.299 that is the behaviour; earlier versions graded whatever the run had produced.
• results.json has partial: true and a partialReason when the ceiling or an interrupt cut the suite short. Keep those out of any trend chart.
• The Δ is reported but never changes the exit code. A plugin that adds nothing still passes.
• claude plugin eval init needs a terminal. In CI, use init --bare <name> to get the blank template.

To keep per-commit cost down, the docs advise giving every-change suites only graders that don’t call a judge, and --ablation none where Δ isn’t needed. That halves the runs. Run the full two-arm suite with judge graders on a schedule or before a release.

What goes wrong

Three runs per case is a small sample. The docs call a single run noisy and say to confirm any change at the default three before trusting it. A skill that passes two of three runs scores 0.67 and fails an 0.8 threshold, which is correct behaviour for a flaky skill and annoying for a flaky grader.

llm graders are the least stable part. The docs advise grading long output with a regex over the file instead, keeping rubrics to concrete PASS and FAIL conditions, and, when the skill fired but Δ is negative, suspecting the judge before the plugin. A small judge model can mark a correct answer wrong because of formatting. --judge-model sonnet is the suggested next try.

Isolation cuts both ways. Because no CLAUDE.md or project config loads, a skill that depends on your repo’s conventions will look worse in an eval than it does at your desk. Ship whatever the case needs inside the plugin, or seed files with a scaffold_script, which only runs when you pass --scaffold, so pass that flag only for suites you wrote.

Don’t bother for a personal skill you edit weekly and use daily. The break-even is a skill shared across a team, or one that other skills sit near, where a quiet drop in behaviour costs someone else an afternoon.

First 20 minutes

1. Update to a Claude Code that has the command (claude --version, then claude update) and check git --version is 2.31 or later.
2. From the root of a plugin or skills directory, run claude plugin eval init --bare first-case and replace the placeholders with one natural-phrasing prompt, one regex grader on the output and one tool_used: Skill grader.
3. Run claude plugin eval . --trust-plugin --no-publish and read the Δ. Then break the skill on purpose and confirm the run goes red.
4. Copy the CI command above into a job, and add evals/results/ to .gitignore.