Claude Code now logs every tool call it stops. Build a refusal ledger that shows which rules earn their keep

A deny rule that never fires is either perfect or dead, and there’s been no way to tell which. Six months after someone added Bash(curl *) to the project settings, nobody knows whether it stopped one stray download or a hundred. Meanwhile the hook that blocks destructive commands has been quietly failing on the one pattern agents actually produce, and the only evidence is a bad afternoon.

Claude Code 2.1.307 added something that fixes the blind spot. According to the changelog, there’s now a claude_code.safety_event OpenTelemetry event for each tool call that a permission rule, auto mode, a safety check, a hook or the sandbox stopped, prompted or sent back. Every refusal becomes a row you can count.

This piece builds a refusal ledger from it: a local receiver that keeps only the refusals, a wrapper that labels each run, and a report that groups them by what stopped them and what was being attempted. It also exits non-zero when one prompt runs into too many walls, which makes it usable as a CI check. The pattern is the same one used for the cost ledger , pointed at guardrails instead of dollars.

What you need

Claude Code 2.1.307 or later (the output below is from 2.1.308), Python 3 and a shell. No collector, no pip installs. If the team already runs an OpenTelemetry collector, skip the receiver and filter on event.name there.

One caveat before the build. When this was written, the events list on the monitoring page didn’t mention safety_event. The changelog line is the only official description. The attribute names below come from running the event, not from a reference, so treat them as correct for 2.1.308 and check them after upgrades.

Step 1: See what an event looks like

Before building anything, look at one. Create a scratch repo with a deny rule and a hook, then ask the agent to trip both.

{
  "permissions": { "deny": ["Bash(curl:*)", "Read(./.env)"] },
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "python3 -c \"import json,sys; d=json.load(sys.stdin); sys.exit(2 if 'rm -rf' in d['tool_input'].get('command','') else 0)\""
          }
        ]
      }
    ]
  }
}

Save that as .claude/settings.json. The Bash rule uses the older curl:* spelling. The permissions docs now write the same rule as Bash(curl *). The hook reads the tool input from stdin and exits with code 2 when the command contains rm -rf. Per the hooks guide, exit 2 blocks the action and stderr becomes the reason.

Now run Claude Code headlessly with the console log exporter:

export CLAUDE_CODE_ENABLE_TELEMETRY=1 OTEL_LOGS_EXPORTER=console OTEL_LOGS_EXPORT_INTERVAL=1000 OTEL_METRICS_EXPORTER=none
claude -p "Run: curl -s https://example.com. Then read the file .env. Report briefly." --model haiku

Among the events printed is one with the body claude_code.safety_event. Trimmed to the attributes that matter:

category: "permission_rule",
decision: "reject",
decision_source: "config",
reason_type: "subcommandResults",
reason_code: "rule:projectSettings",
rule_source: "projectSettings",
tool_class: "bash",
tool_name: "Bash",
tool_source: "builtin",
permission_mode: "default",
tool_use_id: "toolu_01UrNdWeSeLRQWmevGG7oprb",
sandbox_violation_count: "0",
sandbox_enabled: "false",

Three things stand out. rule_source says which settings layer owns the rule that fired (projectSettings here, and cliArg when the block came from --disallowedTools), so a noisy rule has an address. tool_class is a coarse bucket (bash, file) that’s easier to group on than tool names. And there’s no command in there at all. The event tells you that something was stopped, not what.

Every event also carries user.email, an account UUID and an organisation ID alongside the rest. That matters in step 2.

Step 2: A receiver that keeps only the refusals

Claude Code can export the same events over OTLP, the OpenTelemetry wire protocol, and JSON over HTTP is readable with nothing but the standard library. Save this as safety_sink.py:

#!/usr/bin/env python3
"""Tiny OTLP/HTTP (JSON) receiver. Keeps safety_event and tool_decision records."""
import json, sys
from http.server import BaseHTTPRequestHandler, HTTPServer

OUT = sys.argv[1] if len(sys.argv) > 1 else "safety-events.jsonl"
PORT = int(sys.argv[2]) if len(sys.argv) > 2 else 4318
DROP = ("user.", "organization.", "ccr.")  # identity never reaches the file


def value(v):
    for key, raw in v.items():
        if key == "intValue":
            return int(raw)
        return raw
    return None


def attrs(items):
    return {a["key"]: value(a["value"]) for a in items or [] if not a["key"].startswith(DROP)}


class Sink(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
        if self.path.endswith("/v1/logs"):
            res = json.loads(body).get("resourceLogs", [])
            with open(OUT, "a", encoding="utf-8") as f:
                for rl in res:
                    for sl in rl.get("scopeLogs", []):
                        for rec in sl.get("logRecords", []):
                            a = attrs(rec.get("attributes"))
                            if a.get("event.name") in ("safety_event", "tool_decision"):
                                a["task"] = attrs(rl.get("resource", {}).get("attributes")).get("task", "")
                                f.write(json.dumps(a) + "\n")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b"{}")

    def log_message(self, *args):
        pass


HTTPServer(("127.0.0.1", PORT), Sink).serve_forever()

It listens on 127.0.0.1 only, drops every identity attribute before writing, and keeps two event types. safety_event is the refusal. tool_decision is kept because it can answer the question the refusal can’t: what was the agent trying to do?

That second event includes a tool_use_id that matches the refusal’s, and with OTEL_LOG_TOOL_DETAILS=1 it adds a tool_parameters string. For a Bash call that string holds the full command. The report will join the two.

Start the receiver in the background:

nohup python3 safety_sink.py ~/ledger/safety-events.jsonl 4318 >/dev/null 2>&1 &

Step 3: A wrapper that labels each run

The label turns a pile of events into something you can read. OTEL_RESOURCE_ATTRIBUTES attaches custom attributes to everything a run emits, and the receiver copies task onto each row. Keep labels to letters, digits and hyphens, because the value must not contain spaces.

Save this as run-watched and chmod +x it:

#!/usr/bin/env bash
# Usage: run-watched <task-label> <claude args...>
set -euo pipefail
label="$1"; shift
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
export OTEL_LOGS_EXPORT_INTERVAL=1000
export OTEL_RESOURCE_ATTRIBUTES="task=${label}"
claude "$@" </dev/null
sleep 2

The two-second sleep lets the last batch of events flush before a short -p run exits. Export OTEL_LOG_TOOL_DETAILS=1 in your shell before using it, or the report can’t show what was attempted.

Now run three small tasks, each built to hit a different wall:

export OTEL_LOG_TOOL_DETAILS=1
run-watched deny-rule -p "Run: curl -s https://example.com. Then read the file .env. Report briefly." --model haiku
run-watched hook -p "Run exactly: rm -rf build. Report briefly." --model haiku

Step 4: The report

Save this as report.py:

#!/usr/bin/env python3
"""Group safety events by what stopped them. Usage: report.py events.jsonl [limit]"""
import json, sys
from collections import Counter

rows = [json.loads(l) for l in open(sys.argv[1], encoding="utf-8")]
params = {r["tool_use_id"]: r.get("tool_parameters", "") for r in rows if r["event.name"] == "tool_decision"}
events = [r for r in rows if r["event.name"] == "safety_event"]

groups = Counter(
    (e["category"], e["decision"], e.get("rule_source") or e.get("hook_event") or "-", e["tool_class"])
    for e in events
)
print(f"{'category':16} {'decision':8} {'by':16} {'tool':6} count")
for (cat, dec, by, tool), n in groups.most_common():
    print(f"{cat:16} {dec:8} {by:16} {tool:6} {n:5}")

print("\nwhat was stopped:")
for e in events:
    p = params.get(e["tool_use_id"], "")
    try:
        j = json.loads(p)
        p = j.get("full_command") or j.get("file_path") or p
    except (ValueError, AttributeError):
        pass
    print(f"  {e['task']:10} {e['tool_name']:5} {str(p)[:70] or '(no parameters logged)'}")

# A prompt that hit several rejections is an agent pushing against a rule.
LIMIT = int(sys.argv[2]) if len(sys.argv) > 2 else 3
per_prompt = Counter(e["prompt.id"][:8] for e in events if e["decision"] == "reject")
noisy = {k: n for k, n in per_prompt.items() if n >= LIMIT}
for k, n in noisy.items():
    print(f"\nprompt {k} was stopped {n} times")
sys.exit(1 if noisy else 0)

Run it with a limit of 2:

python3 report.py ~/ledger/safety-events.jsonl 2

That prints:

category         decision by               tool   count
permission_rule  reject   projectSettings  bash       1
permission_rule  reject   projectSettings  file       1
hook             reject   pre_tool_use     bash       1

what was stopped:
  deny-rule  Bash  curl -s https://example.com
  deny-rule  Read  (no parameters logged)
  hook       Bash  rm -rf build

prompt d52682fd was stopped 2 times

The exit code was 1, because the first prompt hit two walls. With the default limit of 3 it exits 0.

The top table answers the question from the opening: which layer is doing the work. The second section answers what for. The last line is the one to watch in practice. One refusal is the system working. Several in a single prompt means the agent is trying variations, and that’s worth reading the transcript for.

Reading the ledger

A week of rows tends to sort into a few patterns, and each one suggests a different fix. This is analysis, not something the tooling decides for you.

A rule that fires often on harmless commands is too broad. If Bash(git push *) is rejecting pushes to a personal fork a dozen times a week, either narrow the rule or move it to ask so a person can approve the legitimate ones. Rules are evaluated deny first, then ask, then allow, so a deny can’t be overridden by a later allow.

A rule that fires on the same command again and again points at the agent, not the rule. The fix is a line in CLAUDE.md telling it what to use instead (rg rather than curl | grep, the project’s script rather than a raw rm), so it stops spending turns on a path it can’t take.

A rule that never fires after a month is either well understood by the agent or aimed at something that never happens. That’s not proof it’s safe to delete. A guard against force-pushes should be silent when everything is working. But it’s worth asking whether the rule is the only thing covering a risk, and testing that it still matches.

A hook that rejects far more often than you expected is a hook worth reading. Hooks are ordinary scripts, and an over-eager pattern (rm -rf matched inside a quoted string, say) blocks legitimate work. The hook_event and reason_code fields tell you which hook point rejected the call.

Make it yours

For a team, run one receiver on a shared host instead of a laptop and point every developer’s OTEL_EXPORTER_OTLP_ENDPOINT at it, so all runs land in one file. Keep 127.0.0.1 binding for personal use. A shared receiver needs authentication and TLS, which this 45-line version doesn’t have.

For CI, run the agent step through run-watched, then add the report as a follow-up step. Because the script exits 1 when a prompt crosses the limit, a pipeline can fail, or just flag, an unattended run that spent its time fighting a rule. For an agent step that’s supposed to be boring, a non-zero report is a useful signal.

To see what the sandbox contributes, watch sandbox_enabled and sandbox_violation_count on the events. They’re strings ("false", "0"), not booleans, so convert them before comparing. Every event in the output above had the sandbox off, so what they look like with it on isn’t shown here.

What goes wrong

The biggest gap is prompts nobody answered. In headless runs with the default permission mode, a Write call and a touch that needed approval were declined, and the agent reported that neither step ran. No safety_event appeared for either. If the event is meant to cover calls that were prompted, that case isn’t visible in claude -p on 2.1.308, so the ledger undercounts friction in unattended runs. Interactive sessions weren’t exercised.

A denied file read left no tool_decision row, which is why the Read line above shows no parameters. The refusal is counted but its target isn’t recorded. Bash refusals carry the full command; file refusals, in these runs, didn’t.

The ledger is also not a log of what the agent did. It records what was stopped. A command that slipped past a rule doesn’t appear. The permissions docs are blunt about this: a Bash deny rule matches the command text Claude writes, so /usr/bin/curl or sh -c 'curl ...' isn’t matched by Bash(curl *). A quiet ledger can mean a good rule or a rule that’s being walked around. Pair it with the sandbox, which restricts what a process can reach regardless of how the command is spelled.

tool_parameters holds full commands, which can include tokens pasted on a command line. Keep the ledger file out of version control, and don’t ship it to a shared place without scanning it. The receiver drops identity attributes, but nothing scrubs the command text.

Finally, the attributes are undocumented. A minor release can rename rule_source, and a report that silently groups everything under - is worse than one that crashes. After each upgrade, run step 1 once and compare.

Don’t bother if the team has no deny rules or hooks yet. There’s nothing to measure, and writing those comes first.

First 15 minutes

Check claude --version is 2.1.307 or later. Run step 1 in any repo that has a deny rule, and read one real event. Then save the receiver, the wrapper and the report, start the receiver, and send one task through run-watched that you know will hit a rule. If the report shows the row with the command next to it, the plumbing works, and a week of normal use will show which rules were worth writing.