tokenise by Rosvetic
DevOps How-to

Stop waking the agent for flaky CI: a Jev confidence gate in under 50 lines of TypeScript

One cheap call decides whether a red build gets a retry, an agent or a person, and a confidence gate in your own code decides whether to believe it. About 20 minutes to wire up.

8 min read

A railway yard seen from above with one track splitting into three and a single peach streak of light taking only the middle branch

A pull request goes red at 4:40pm. The agent you wired to CI failures wakes up, reads 3,000 lines of log, forms a theory about a TypeError and starts editing files. The real cause was a registry timeout during npm ci. Twelve minutes of tokens later, you have a diff nobody needed. Flip the case: a genuine failing assertion gets a blind re-run, fails again, and the PR sits another 10 minutes.

The decision that was missing is small: retry, agent or person. It’s a three-way label on a log tail, which makes it a good fit for a decision model rather than a chat model. Jev, TypeSafe AI’s model for typed decisions, takes the log and a question, and returns one of your options with probabilities. A few lines of your own code then decide whether to trust it.

By the end of this you’ll have a function that triages a failed run, tests that pin the gating rules, and a small CLI that a CI step can call. The gating logic runs offline, so you can try it before you have a key. See what a decision model is and isn’t first if Jev is new to you.

The design

One request, two questions, both asked of the same log tail.

The first is a Choice with three options: human, agent, retry. The second is a Noul, a yes/no probability, asking whether the log contains a credential. The docs say questions in one request are evaluated in parallel, so the second question adds little latency.

The model only classifies. The policy lives in your code: a confidence floor, a stricter bar for retrying, a cap of one retry and a rule that anything touching a secret goes to a person. That split matters because it keeps the part you need to audit in a place you can read and test.

Install

You need Node.js 22 or newer (the SDK README requires it) and an API key from the TypeSafe console.

mkdir ci-triage && cd ci-triage
npm init -y && npm pkg set type=module
npm install @typesafe-ai/sdk
npm install -D vitest typescript
export TYPESAFE_API_KEY="your-api-key"

The gate and the call

Save this as triage.ts:

import { choice, noul, TypeSafeClient } from "@typesafe-ai/sdk";

export type Action = "retry" | "agent" | "human";

export interface Verdict {
  action: Action;
  reason: string;
}

const FLOOR = 0.6;
const RETRY_MIN = 0.85;

export function gate(
  route: Action,
  confidence: number,
  secretsInLog: number,
  retriesSoFar: number,
): Verdict {
  if (secretsInLog >= 0.5) return { action: "human", reason: "log may contain a secret" };
  if (confidence < FLOOR) return { action: "human", reason: `confidence ${confidence} below ${FLOOR}` };
  if (route === "retry") {
    if (retriesSoFar >= 1) return { action: "human", reason: "already retried once" };
    if (confidence < RETRY_MIN) return { action: "agent", reason: "unsure it is a flake" };
  }
  return { action: route, reason: `routed to ${route}` };
}

export async function triage(logTail: string, retriesSoFar: number): Promise<Verdict> {
  const client = new TypeSafeClient({ defaultModel: "jev-latest" });
  const { answers } = await client.systemOne({
    state: { failing_step_log_tail: logTail },
    questions: {
      route: choice("Who should deal with this CI failure?", {
        human: "Needs a person: credentials, permissions, billing, or anything unclear",
        agent: "A deterministic code failure: a failing assertion, a compile or type error, a lint violation",
        retry: "Infrastructure or network noise: a timeout, a registry 5xx, a runner that died before any test ran",
      }),
      secrets: noul("The log contains a credential, token or private key"),
    },
  });
  return gate(answers.route.choice, answers.route.confidence, answers.secrets.noul, retriesSoFar);
}

A few choices in there are deliberate.

human is listed first. TypeSafe’s list of known weaknesses for jev-1.13 says the model leans toward the first option in a Choice, so the bias goes toward the safest outcome.

Each option has a description rather than a bare label. The descriptions are the question, and they’re where accuracy comes from. “retry” alone is ambiguous. “A runner that died before any test ran” isn’t.

The retry bar (0.85) is higher than the floor (0.6) because a wrong retry is cheap but a wrong “this is a flake” hides a real failure behind a green re-run. Between the two, an unsure retry gets promoted to the agent. Those two numbers come from the docs’ confidence-routing example (a 0.6 floor, and above 0.85 for the high-stakes action), not from any CI data. They’re a starting point and nothing more.

The SDK types make answers.route.choice the union of your option keys, so gate receiving it as Action is checked by the compiler. Confidence is 0 to 1 on a Choice. A Noul has no confidence field, only the probability, so the secrets check uses that directly.

Test the policy offline

The gate is a pure function, so the tests need no network. Save as triage.test.ts:

import { describe, expect, it } from "vitest";
import { gate } from "./triage";

describe("gate", () => {
  it("lets a confident code failure go to the agent", () => {
    expect(gate("agent", 0.93, 0.01, 0).action).toBe("agent");
  });
  it("sends low confidence to a human whatever the route", () => {
    expect(gate("agent", 0.4, 0.01, 0).action).toBe("human");
  });
  it("retries only when very sure it is a flake", () => {
    expect(gate("retry", 0.95, 0.01, 0).action).toBe("retry");
    expect(gate("retry", 0.7, 0.01, 0).action).toBe("agent");
  });
  it("never retries twice", () => {
    expect(gate("retry", 0.99, 0.01, 1).action).toBe("human");
  });
  it("hands a log that may hold a secret to a human before anything else", () => {
    expect(gate("retry", 0.99, 0.9, 0).action).toBe("human");
  });
});

Run npx vitest run and it prints:

 Test Files  1 passed (1)
      Tests  5 passed (5)

The last test is the one to keep when you change thresholds. Rules about secrets and retry caps shouldn’t move because someone tuned a number.

Feed it the right input

Jev’s docs list large state with irrelevant detail as a known accuracy problem. A full CI log is mostly irrelevant detail. Send the tail of the failing step only.

Save this as triage-cli.ts:

import { readFileSync } from "node:fs";
import { triage } from "./triage.ts";

const retries = Number(process.argv[2] ?? 0);
const verdict = await triage(readFileSync(0, "utf8"), retries);
console.log(JSON.stringify(verdict));

On Node.js 22.18 or newer it runs as TypeScript directly. Then the CI-side glue, using the GitHub CLI:

#!/usr/bin/env bash
set -euo pipefail
RUN_ID="$1"; RETRIES="${2:-0}"

verdict=$(gh run view "$RUN_ID" --log-failed | tail -n 60 | node triage-cli.ts "$RETRIES")
action=$(jq -r .action <<<"$verdict")
echo "triage: $verdict"

case "$action" in
  retry) gh run rerun "$RUN_ID" --failed ;;
  agent) echo "hand the log tail to your fix agent here" ;;
  *)     echo "needs a person" ;;
esac

With no key set, the CLI stops with TypeSafeError: No API key was provided, which at least confirms it loads and reaches the SDK. With a key, the output is one JSON line such as {"action":"retry","reason":"routed to retry"}. That line shows the shape of the output, not a measured result.

The agent branch is deliberately a placeholder. Wire it to whatever already starts your agent. How that agent is permissioned is a separate question from this gate.

Make it yours

For a monorepo, add a second Choice naming the package or service, and route the agent branch to the right owner. Questions in one request run in parallel, so it costs little.

For a repo with a known failure catalogue, put your top three recurring failures into the option descriptions in plain words. Specific descriptions beat generic ones.

For a stricter team, make agent need a confidence of 0.85 too, and send everything else to a person. A conservative first week is cheap to relax later.

What goes wrong

Logs are untrusted text. On pull requests from forks, whoever opened the PR controls part of the log. TypeSafe’s docs say the model doesn’t treat state as hostile, so a line in a test name that reads like an instruction can move the answer. The design above limits what a nudge can buy: the worst outcomes are one extra retry (capped at one) or an agent run on a failure that was really a flake. Keep it that way. Never add an option whose action is destructive or privileged, and keep a deterministic secrets scrub in code ahead of the Noul check, because the Noul can be fooled too.

Confidence isn’t accuracy. Pydantic AI’s docs describe it as a margin from the decision threshold, not a probability of being correct. Replay 50 to 100 past failed runs with the right answer written down by a person, then look at the confidence on the wrong ones. Set the floor where the errors stop.

Versions move. jev-latest is an alias, and TypeSafe said it would move to jev-1.14.0 a week after its 14 October 2026 post. Once the thresholds are tuned, pin a versioned id in defaultModel so an update can’t change them under you.

Cost is not the reason to do this. Input is priced at $0.042 per million tokens, so a 60-line tail costs next to nothing, but the saving that matters is the agent run you don’t start.

Skip all of it if one failure string explains most of your red builds. A grep -q ETIMEDOUT in the workflow is free, instant and explainable. Reach for a decision model when the causes are varied enough that the rules would sprawl.

First 20 minutes

  1. Install the SDK, set TYPESAFE_API_KEY, and run the five tests.
  2. Pull the failing-step tail of five recent red runs with gh run view <id> --log-failed | tail -n 60 and pipe each into the CLI.
  3. Compare each verdict with what you know the cause was, and write down the confidence on any that were wrong.
  4. Move the floor until the wrong ones fall below it, then pin the model id.