tokenise by Rosvetic
DevOps How-to

Claude Code's CI retry watchdog waits forever by default. Put a ceiling on it

CLAUDE_CODE_RETRY_WATCHDOG stops a nightly job dying in a three-minute outage. Without a second setting, it can also hold your runner for six hours.

8 min read

A ribbon cable looped in a circle with its plug returning to the same socket on a dark circuit board

A nightly Claude Code job hits an API overload at 2am. By default the CLI retries up to 10 times with exponential backoff, then gives up and the job goes red. Against a server that answers every request with a 529, that took three minutes. If the overload lasts four, you get a failed run and a morning of “just re-run it”.

CLAUDE_CODE_RETRY_WATCHDOG=1 is the documented fix for unattended sessions. It also has no ceiling unless you add one. This piece covers what the watchdog does, how to watch it work with a stub server you can run in a minute, and the two limits that turn it from a liability into a sensible default for CI.

The three behaviours

Without the watchdog, Claude Code retries transient failures up to CLAUDE_CODE_MAX_RETRIES times, which defaults to 10. The errors page lists what counts: server errors, overloaded responses, request timeouts, temporary 429 throttles and dropped connections before any of the response has arrived.

With CLAUDE_CODE_RETRY_WATCHDOG=1 (Claude Code 2.1.186 or later), the env-vars reference says three things change:

  • 429 and 529 capacity errors are retried indefinitely instead of stopping after CLAUDE_CODE_MAX_RETRIES.
  • Backoff grows to as much as 5 minutes between attempts, or until the limit resets when the response carries a reset time. A session that hits a usage limit waits out the window.
  • On 2.1.199 or later, other transient errors (server errors, timeouts, dropped connections) get 300 retries instead of 10, which the docs put at roughly three hours of backoff.

One exception matters for budgeting. A standard-speed request that gets a 429 reporting a spend limit or exhausted credits fails at once, including a gateway spend cap that resets on a schedule. The watchdog rides out capacity problems, not an empty wallet.

The third piece arrived in 2.1.295: CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS. It’s the maximum time, in milliseconds, that each API request spends waiting out 429 and 529 errors. Once that time is spent, the next such error ends the request. Unset, the wait has no limit.

Note the word each. It’s a per-request budget, not a per-job one. Keep that in mind for the workflow below.

Watch it happen against a stub

A Python server that rejects everything with an overloaded error is enough to see all of this without waiting for a real outage. Save it as stub529.py:

import http.server, time

t0 = time.time()

class Handler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("content-length", 0)))
        print(f"{time.time() - t0:6.1f}s POST {self.path} -> 529", flush=True)
        body = b'{"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}'
        self.send_response(529)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

http.server.HTTPServer(("127.0.0.1", 8529), Handler).serve_forever()

Start it with python3 stub529.py &, then point Claude Code at it with ANTHROPIC_BASE_URL. The API key can be any string, because the stub never checks it. This is the default behaviour:

ANTHROPIC_BASE_URL=http://127.0.0.1:8529 ANTHROPIC_API_KEY=sk-ant-test \
  claude -p "say hi" --max-turns 1 </dev/null
echo "exit=$?"

API Error: 529 Overloaded. This is a server-side issue, usually temporary ... (trimmed)
exit=1

The stub logged 11 requests (the original plus 10 retries) over 181 seconds. That’s the whole patience budget of a default CI job: about three minutes.

Now the watchdog with nothing else set, wrapped in timeout 90 so the experiment ends:

CLAUDE_CODE_RETRY_WATCHDOG=1 timeout 90 claude -p "say hi" --max-turns 1 </dev/null
echo "exit=$?"

exit=124

Exit 124 is timeout killing it. By then the stub had logged 8 requests, with the gaps between them still widening, and nothing in Claude Code was going to stop. Against a real multi-hour incident, nothing would have stopped it either, except the CI platform.

Add the ceiling, 20 seconds here so the run is short:

CLAUDE_CODE_RETRY_WATCHDOG=1 CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS=20000 \
  claude -p "say hi" --max-turns 1 --output-format stream-json --verbose </dev/null

The system/api_retry events in the stream show each wait. Trimmed to the interesting fields, the run printed:

attempt 1  max_retries 300  retry_delay_ms 517   error_status 529  error overloaded
attempt 2  max_retries 300  retry_delay_ms 1135  error_status 529  error overloaded
attempt 3  max_retries 300  retry_delay_ms 2406  error_status 529  error overloaded
attempt 4  max_retries 300  retry_delay_ms 4782  error_status 529  error overloaded
attempt 5  max_retries 300  retry_delay_ms 8798  error_status 529  error overloaded
attempt 6  max_retries 300  retry_delay_ms 2362  error_status 529  error overloaded

Delays roughly double up to attempt 5, then the sixth drops to 2,362 ms. That’s not the next step in the doubling pattern. It fits a wait that’s being clipped to whatever is left of the 20 seconds, though the docs don’t spell out that mechanism, so treat it as an observation. After that sixth wait the process exited with code 1. (A plain-text run with the same two settings took 27 seconds end to end.)

The max_retries of 300 is the watchdog’s raised count for other error types, visible in the event. The 529s themselves aren’t bound by it.

A gotcha in the JSON result

With --output-format json, the final result for that failed run looked like this (trimmed):

type: result   subtype: success   is_error: True
result: API Error: 529 Overloaded ... (trimmed)
total_cost_usd: 0

The subtype says success while is_error is true. A script that gates on subtype will wave a failed run through. Gate on the exit code, which was 1 here and which the headless docs say is non-zero when a run fails, and check is_error if you parse the JSON.

The workflow

Here’s a nightly job with both limits and a hard stop from the platform. The prompt is a placeholder, so swap in your own chore:

name: nightly-claude
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch:

jobs:
  digest:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: npm install -g @anthropic-ai/claude-code
      - name: Draft changelog
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          CLAUDE_CODE_RETRY_WATCHDOG: "1"
          CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS: "600000"
        run: |
          claude --bare -p "Summarise the commits from the last 24 hours into CHANGELOG-draft.md" \
            --allowedTools "Read,Write,Bash(git log *)" \
            --max-turns 20 --max-budget-usd 3 \
            --output-format json > result.json
          jq -e '.is_error == false' result.json

Three layers, each doing a different job. CLAUDE_CODE_RETRY_WATCHDOG rides out a short outage. CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS caps each request at 10 minutes of waiting. timeout-minutes: 45 is the real cap on the whole job, because the wait budget applies per request and a run can make dozens of them.

GitHub’s default job timeout is 360 minutes, per the workflow syntax reference, so a job with no timeout-minutes and an unbounded watchdog can hold a runner for six hours. Steps accept timeout-minutes too, with the same 360 maximum, if you’d rather bound the one step than the job.

--bare is there because the headless docs recommend it for scripted calls: it skips hooks, plugins, auto memory and CLAUDE.md from the host, so the run behaves the same everywhere. It also means the key must come from ANTHROPIC_API_KEY. The jq -e line is belt and braces: -e makes jq exit non-zero when the expression is false, and the step already fails on Claude Code’s own exit code.

Picking the numbers

This part is opinion. Work backwards from when the output stops being useful.

If a morning digest is worthless after 7am, a 2am job doesn’t need to wait through a five-hour incident. Set the job timeout to what you’d tolerate, say 60 minutes, and set the per-request wait to a third of it. A job that can fail early gets to fail early, and its alert fires when someone’s awake to see it.

If the job is a long migration where a restart costs hours of work, lean the other way: a bigger per-request wait, a generous job timeout, and an alert on duration rather than failure. The watchdog exists for exactly that shape of work.

The per-request budget resets for every request. A 30-minute MAX_WAIT_MS across a 40-request run is a theoretical 20 hours of waiting, which is why the job timeout is the number to take seriously.

Variation: a job someone is waiting on

A review step on a pull request is the opposite case. A developer is watching the check, and a ten-minute stall is worse than a red cross they can re-run. Leave the watchdog off, keep the default 10 retries (about three minutes against the stub), and set timeout-minutes: 10 on the job anyway. The retry count bounds the overload case, and the timeout catches everything else, such as a hung tool call.

Dropping the retry count is the cheaper lever if three minutes is still too long. The errors page says CLAUDE_CODE_MAX_RETRIES overrides the default of 10, so CLAUDE_CODE_MAX_RETRIES=3 fails fast.

What this doesn’t cover

The stub proves the retry mechanics, not the shape of a real incident. A real 429 can carry a reset time that the watchdog will wait for, which can be longer than your MAX_WAIT_MS. In that case the cap ends the request sooner than the reset would have. That’s the point of the cap, but expect the job to fail rather than finish.

The watchdog doesn’t help with a spend limit. Per the docs, those 429s fail at once. A budget alert on your account does more for you there than any retry setting.

To get retries into your own logs, run with --output-format stream-json --verbose and filter for system/api_retry events, which carry the attempt number, delay and error category.

Don’t bother if the job is cheap, idempotent and runs hourly. Let it fail, and let the next run pick it up.

First 15 minutes

Run the stub, run the three commands above, and watch the attempt log with your own eyes. Then add CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS and a timeout-minutes to whichever scheduled Claude Code job you already have.