Your overnight Claude Code run died at 2am. A StopFailure hook and a 30-line wrapper decide whether to wait or wake you

A nightly job asks Claude Code to migrate forty files. At 2:14am the API returns a server error, the run ends mid-migration, and the cron job exits non-zero into a log nobody reads. At 8am the working tree is half changed, the log says API Error, and the only way to find out whether that was a thirty-second blip or an expired key is to read it and guess.

Claude Code already retries a failed API request ten times by default, and an environment variable can stretch that to forever for certain errors. What it doesn’t do is tell you what happened once it gives up, and it doesn’t distinguish between an error that waiting fixes and an error that needs a person.

There’s a hook for exactly this moment. StopFailure fires when a turn ends because of an API error, and it hands over the error type. This Playbook turns it into two pieces: a hook that writes down why the run stopped and whether waiting can help, and a wrapper script that reads that record and either sleeps and resumes the same session or exits with a code your scheduler can alert on. Both are short, and the output below is from Claude Code 2.1.312.

What the hook gives you

The hooks reference describes StopFailure as running instead of Stop when the turn ends due to an API error. It receives the usual common fields plus three of its own: error, an optional error_details, and an optional last_assistant_message, which for this event holds the rendered API error string rather than anything Claude said.

The error field takes one of twelve values: rate_limit, overloaded, authentication_failed, oauth_org_not_allowed, account_on_hold, billing_error, invalid_request, model_not_found, server_error, max_output_tokens, cloud_credential_error or unknown. That list is the whole point. Four of those are things time can fix. The rest are a person’s job.

Two limits shape the design. The hook has no decision control: Claude Code ignores its output and exit code, apart from a terminalSequence field that only works in an interactive session and is ignored under -p. So the hook can’t retry anything itself. It can only leave a note. And a note is enough, because the process that launched claude -p is still alive and can read it.

The hook

Put this at .claude/hooks/stop-failure.py. It sorts the error into “wait” or “fix”, appends a line to a ledger, overwrites a last-failure.json the wrapper will read, and optionally posts to a webhook when a human is needed.

#!/usr/bin/env python3
"""StopFailure hook: record why the turn died and whether waiting can fix it."""
import json, os, sys, time, urllib.request

WAIT = {"rate_limit", "overloaded", "server_error", "unknown"}
FIX = {"authentication_failed", "oauth_org_not_allowed", "account_on_hold",
       "billing_error", "model_not_found", "invalid_request", "max_output_tokens",
       "cloud_credential_error"}

event = json.load(sys.stdin)
error = event.get("error", "unknown")
record = {
    "at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    "session_id": event.get("session_id"),
    "transcript": event.get("transcript_path"),
    "error": error,
    "action": "wait" if error in WAIT else "fix",
    "message": (event.get("last_assistant_message") or "")[:300],
}

state_dir = os.path.join(event.get("cwd", "."), ".claude", "state")
os.makedirs(state_dir, exist_ok=True)
with open(os.path.join(state_dir, "failures.jsonl"), "a") as f:
    f.write(json.dumps(record) + "\n")
with open(os.path.join(state_dir, "last-failure.json"), "w") as f:
    json.dump(record, f)

url = os.environ.get("FAILURE_WEBHOOK")
if url and record["action"] == "fix":
    req = urllib.request.Request(
        url,
        json.dumps({"text": f"Claude run stopped: {error}. {record['message']}"}).encode(),
        {"content-type": "application/json"},
    )
    try:
        urllib.request.urlopen(req, timeout=5)
    except OSError:
        pass

The FIX set is documentation more than logic: anything not in WAIT is treated as “fix”. That’s deliberate. A new error type added in a later release defaults to waking you, not to retrying a billing failure all night.

The webhook is optional and fires only for the “fix” class. A retryable error that resolves itself shouldn’t page anyone. The except OSError: pass is there because a hook that raises while Claude Code is already in trouble helps nobody.

Register it

In .claude/settings.json:

{
  "hooks": {
    "StopFailure": [
      {
        "hooks": [
          { "type": "command", "command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop-failure.py\"" }
        ]
      }
    ]
  }
}

There’s no matcher, so it fires for every error type and the script does the sorting. If you’d rather split the logic in config, StopFailure accepts a matcher on the error type, such as "matcher": "rate_limit|overloaded". Matchers for this event accept only letters, digits, underscores and |; a hyphen or comma pushes it onto the regular-expression path.

Add .claude/state/ to .gitignore. The ledger is operational data, and an unattended agent that runs git add -A shouldn’t commit it.

The wrapper

The wrapper owns the session ID so it never has to dig it out of a payload. It starts the run with --session-id, and every retry uses --resume on that same ID, so a retry continues the conversation instead of starting the migration again from the prompt.

Save as overnight.sh and chmod +x it:

#!/usr/bin/env bash
# Usage: ./overnight.sh "task prompt"
# Retries a headless Claude Code run when the API says "wait", stops when it says "fix".
set -u
TASK="$1"
MAX_ATTEMPTS="${MAX_ATTEMPTS:-5}"
BASE_WAIT="${BASE_WAIT:-60}"
STATE=.claude/state
SESSION="$(python3 -c 'import uuid;print(uuid.uuid4())')"
rm -f "$STATE/last-failure.json"

for attempt in $(seq 1 "$MAX_ATTEMPTS"); do
  if [ "$attempt" -eq 1 ]; then
    claude -p "$TASK" --session-id "$SESSION" </dev/null && exit 0
  else
    claude -p "Continue the task from where you stopped. Re-check the working tree first." \
      --resume "$SESSION" </dev/null && exit 0
  fi

  [ -f "$STATE/last-failure.json" ] || { echo "failed without a StopFailure record" >&2; exit 1; }
  action="$(python3 -c "import json;print(json.load(open('$STATE/last-failure.json'))['action'])")"
  error="$(python3 -c "import json;print(json.load(open('$STATE/last-failure.json'))['error'])")"
  if [ "$action" = "fix" ]; then
    echo "stopped on $error: needs a human, not a retry" >&2
    exit 3
  fi
  delay=$(( BASE_WAIT * 2 ** (attempt - 1) ))
  echo "attempt $attempt hit $error, waiting ${delay}s" >&2
  sleep "$delay"
done
echo "gave up after $MAX_ATTEMPTS attempts" >&2
exit 4

The exit codes are the interface for whatever schedules this. Zero is success. One means claude failed without the hook writing a record, which points at something other than an API error: a crash, a bad flag, a hook that didn’t run. Three means a human is needed. Four means the API kept failing for the whole budget.

Two details matter. The resume prompt tells Claude to re-check the working tree first, because a run that died mid-edit may have left a file half written and the transcript can’t know what the filesystem looks like now. And </dev/null stops claude -p waiting three seconds for stdin that will never arrive.

With the defaults, the waits are 60, 120, 240 and 480 seconds, a bit under fifteen minutes in total. Stretch BASE_WAIT if your errors tend to be longer outages.

Watch the three outcomes

To see each path without waiting for a real outage, point ANTHROPIC_BASE_URL at a local stub that returns whatever status you choose, and set CLAUDE_CODE_MAX_RETRIES=1 so Claude Code’s own retries don’t swallow the minute you’re measuring. Leave the retry setting alone in production. Its ten retries are the cheap first line of defence, and the wrapper is what’s left when they run out.

First, a rate limit that clears after about sixteen seconds. With BASE_WAIT=3 for the demo, the wrapper prints:

API Error: Server is temporarily limiting requests (not your usage limit) · fake 429
attempt 1 hit rate_limit, waiting 3s
API Error: Server is temporarily limiting requests (not your usage limit) · fake 429
attempt 2 hit rate_limit, waiting 6s
Done: migrated 3 of 3 files.

The exit status is 0, and failures.jsonl holds two lines, both with the same session_id. That repeated ID is the proof that attempts two and three resumed one session. The fake 429 text is the stub’s message; a real API supplies its own.

Second, an authentication failure, which waiting can’t fix:

Authentication error · This may be a temporary network issue, please try again
stopped on authentication_failed: needs a human, not a retry

The exit status is 3 and last-failure.json reads:

{"at": "2026-10-29T07:10:15Z", "session_id": "19ade338-5938-4fe1-bc85-2060c6d8f2ed", "error": "authentication_failed", "action": "fix", "message": "Authentication error · This may be a temporary network issue, please try again"}

(The transcript field is trimmed from that line.) Note what the message says: “may be a temporary network issue”. The human-readable text is vague. The error field is not, and that’s why the wrapper reads the field.

Third, a server that never recovers, with MAX_ATTEMPTS=2:

API Error: 500 fake 500. This is a server-side issue, usually temporary (message trimmed)
attempt 1 hit server_error, waiting 1s
API Error: 500 fake 500. This is a server-side issue, usually temporary (message trimmed)
attempt 2 hit server_error, waiting 2s
gave up after 2 attempts

Exit status 4, two ledger lines, both server_error. That’s the morning-after view you wanted: a file that says what happened, how many times, and when.

Wire it to a scheduler

A crontab entry that runs the job at 1am and leaves a log:

0 1 * * * cd /srv/app && ./overnight.sh "$(cat .claude/prompts/migrate.md)" >> .claude/state/cron.log 2>&1

Cron won’t alert on exit codes by itself, so add a line that does. A tiny || curl ... after the script works, as does running the wrapper from a CI job that fails on non-zero. In a GitHub Actions job, exit 3 and exit 4 are worth different notifications: three is “check the secret”, four is “the API had a bad night”.

The same wrapper works unchanged inside a CI step. If you already bound a job with CLAUDE_CODE_RETRY_WATCHDOG, read the piece on putting a ceiling on it first, because the two overlap.

Where this fits next to the retry watchdog

CLAUDE_CODE_RETRY_WATCHDOG=1 makes Claude Code retry 429 and 529 capacity errors indefinitely, with backoff up to five minutes between attempts. For those two errors it’s simpler than this wrapper: one variable, nothing to maintain.

What it doesn’t cover is the rest of the list. A server_error isn’t a capacity error. An expired key, a billing problem, an organisation that disallowed your OAuth login and a model name that no longer exists all end the run. The docs also say the watchdog fails at once on a 429 that reports a spend limit or exhausted usage credits. That run still dies, and with nothing recorded.

So the two combine. Set the watchdog for capacity errors and keep the hook and wrapper for everything that gets through. The hook’s ledger is useful even if you never write the wrapper, because it turns “the job failed” into a line you can grep.

Make it yours

Three variations worth considering.

Slack or Teams instead of a webhook URL: the FAILURE_WEBHOOK post is a plain JSON body with a text field, which is what Slack incoming webhooks expect. Check what your chat tool wants before pointing it at the variable.

A stricter “wait” set: if a nightly job must never retry on unknown, remove it from WAIT. An unclassified error is the one case where the script is guessing.

A different agent: the wrapper’s shape suits any CLI that can run headless and resume a session, but StopFailure is a Claude Code hook. For another tool, check what failure signal it exposes before copying the design.

What goes wrong

The hook can’t see usage-limit resets. Subscription plans show messages such as “You’ve hit your session limit” with a reset time, and the errors page describes Claude Code blocking further requests until that time. The reference doesn’t say which error value those map to, and that mapping wasn’t something that could be checked here. If your nightly job runs on a subscription, find out by letting it hit the limit once, then adjust the WAIT and FIX sets. The cap on attempts and the doubling delay bound the damage either way: five attempts, a bit under fifteen minutes, then exit 4.

A resumed session carries its whole transcript. After a long run, the first resumed turn re-reads all of it, and if the pause outlasts the prompt cache lifetime that read is uncached. A job that failed early loses almost nothing by resuming. A job that failed at hour five pays for the context again. For very long runs, resuming the same session may cost more than it saves, and a fresh run that reads a progress file may be cheaper.

Resume continues the conversation, not the filesystem. If an error struck between a file being opened and written, the working tree can be in a state the transcript doesn’t describe. The “re-check the working tree first” line in the resume prompt helps, but a git commit after each unit of work helps more, because the next attempt can read git status and git log and see exactly where the last one got to.

Hook failures are silent. The hook’s own output and exit code are ignored by Claude Code. That’s why the wrapper treats a missing last-failure.json as exit 1 instead of assuming a retryable error: if the hook crashes or isn’t installed, you’d rather hear about it than loop.

Don’t bother with any of this if the job takes a few minutes and a human watches it, or if CI already reruns failed jobs for you. The value is in runs long enough that a mid-run failure costs real work and nobody is watching.

First 15 minutes

Create the hook file and the settings entry, run claude -p "say hi" once with a deliberately wrong ANTHROPIC_API_KEY, and look in .claude/state/failures.jsonl. Any line there means the hook is wired up. The documented type for a rejected key is authentication_failed, though that exact mapping wasn’t observed against the live API. Then add the wrapper, point your scheduler at it, and have the alert key off exit code 3.