Test and roll out judged agents

For: Plugin authors checking answer quality.

Replay judge calls in tests, inspect rollout signals and record results that become known later.

Tests and the quality gate

The judge's calls go through the gateway like any model call, so plugin test --record records them with the agent's, and a replay answers them from the cassette: an eval of a judged agent runs in CI without a key. A best-of's candidates call the model at the same time; a replay matches each call to the recorded call with the same request, so the order they arrive in does not matter.

Candidates that send the same request, such as the agent n times on its own model, may get each other's recorded answers in a replay. So the judge's order depends only on the answers and a seed: the candidates sorted by their text, then shuffled with the seed. An eval case's jobs are seeded from the case's name rather than the job's ID, which differs between a recording and its replay, so a replay sends the judge the request it recorded and its recorded pick keeps the same answer.

A job that failed its outcome check failed, so an eval case expecting it to complete fails, and the quality gate counts it against the new version. A case can expect the failure:

cases:
  - name: refuses to guess
    input: Write the notes for a release that has no changes.
    expect:
      status: failed
      error_contains: OutcomeNotMet

plugin lint checks the declarations: a judge or a reviewer, not both; a judge on another model or alias than the work it judges (a warning; see A judge on another model); the rubric a judge names, or the default one; rounds and candidates within their limits; a grader's CEL; local models, judges, reviewers and candidates for an agent whose data stays on the node; a reviewer without an input or output schema; candidate agents with the agent's own contract.

In a rollout

A plugin's new version rolls out to canary nodes first, and each step compares its jobs with the installed version's. Besides the jobs that failed, the step counts what this page is about, per version: the jobs whose outcome check failed its first round (they may still pass after a revision, at a cost), the jobs whose cascade went to a later model, the jobs a sampled judge failed, and the downstream results reported as failures.

A new version more than 10 points worse on one of them is rolled back, with the signal named; one with fewer than 10 jobs on either side decides nothing. These signals may only stop a rollout: they never pass, approve or keep a version, and they never change a job.

The criteria a first round missed are listed beside the outcome signal, from the verdicts that named one, so a rollback says what got worse (missed fidelity 6).

Downstream results

Some results show only later: whether the pull request an agent opened was merged, whether a reply settled the ticket. Record them on the finished job:

oz0 job outcome job-7f3a --name merged --outcome success --detail https://forge.example/acme/web/pull/42

An app does the same with ReportJobOutcome on the runtime API, with an API key that may manage jobs. An agent records them with the built-in tool oz.report_outcome, listed in its tools::

tools: [oz.report_outcome, github/oz0_github.get_pull_request]

oz.report_outcome(job: "job-7f3a", outcome: "failure", name: "closed_unmerged", detail: "closed without a merge")

• What it is: a name (1 to 64 lowercase letters, digits, dots, dashes and underscores, such as merged or closed_unmerged), success or failure, and an optional note of at most 500 characters.
• Once per name per job, at most 20 names per job, and only within 14 days of the job's end. A running job takes none.
• Whose: an agent reports only on jobs of its own plugin, and the edge reads what it reports from the job's own record, never from what a node sends; the note never reaches the audit log, which keeps the job, the name and the outcome (job.outcome). A replay's report is not recorded, and a quarantine job has no such tool.
• From a flow, without a model: a flow's tool: step may name oz.report_outcome itself, with the step's input as its arguments, and the runtime runs the built-in as an agent's call of it is run: the same checks at the edge, the tenant's rules for the name, a routine's tool list. The step's value is {recorded: true}, or {recorded: false, reason: ...}. See flow steps that call a built-in.
• Where it shows: oz0 job get and the job's page list each result; the note only to people who may read the job's content. The result goes with its job when the job is erased or expires.
• What it does: it is counted per version as the reported signal of rollouts. Results usually arrive days after a soak, so most steps see too few of them; a tenant whose results come quickly can rely on them, and one whose results do not can turn the signal off.

Back to Outcome checks and patterns.