Improve

Outcome checks and patterns

Have a judge check each answer against a rubric, run several candidates and keep the best, or pair a writer with a reviewer.

For: Plugin authors checking answer quality.

An outcome check judges an answer against a rubric. A failed answer returns to the agent with the judge's reasons until it passes, reaches the round limit or exhausts its budget. The guide separates that loop from sampling, reviewer agents and candidate selection.

In this guide

Choose a judge and rubric: Choose an independent judge, a rubric and the evidence the judge may read.
Measure a judge over time: Use fixed examples to detect judge drift and sample completed jobs for quality measurement.
Use a reviewer agent: Use an agent with tools to review the writer's answer.
Choose among candidate answers: Run candidate answers and select one, within the agent's hooks and data restrictions.
Test and roll out judged agents: Replay judge calls in tests, inspect rollout signals and record results that become known later.

Outcome checks

agents/acme-writer.yaml

name: acme-writer
runtime: pydantic-ai
model:
  use: smart
instructions: prompts/acme-writer.md
outcome:
  judge: judge            # a tenant's alias, or <provider>/<model>
  rubric_file: rubrics/release-notes.md
  max_rounds: 3           # verdicts per job, the first answer's included

After the agent answers:

The judge reads the rubric, the job's task and the answer, each in its own block, and answers with a verdict: pass, reasons and, when the answer fails, the criterion it missed. It never sees the agent's conversation, its tools or its instructions. With files: read it also reads the job's files.
A pass ends the check, and the job ends as usual, with its job_end hooks.
A fail goes back to the agent as the next turn of its conversation: the round, the reasons and the rubric, and a request to answer again in full. The agent can use its tools again.
When the last round fails too, the job fails with OutcomeNotMet and the last reason. When the job's budget runs out between rounds, it fails with OutcomeNotMet and says so.

A judge's call that a quota refuses waits for a budget decision on the Approvals page, as the agent's own calls do, and is made again once the budget is raised.

A harness agent gets a new session for each round, with the task, its previous answer and the reasons, since a session starts without the conversation before it. An openai-agents agent's answer that does not pass goes back to it as the next turn of the SDK's conversation, as a Pydantic AI agent's does, and an openai-agents reviewer answers with the verdict.

Outcome checks run for pydantic-ai, openai-agents and harness agents; plugin lint refuses outcome: on a remote agent, and its jobs fail with OutcomeUnsupported rather than answer unchecked.

A harness agent may also name a verify script (outcome.verify_file, a shell script in the plugin): after each session it runs in the session's work directory, and a non-zero exit fails the round with its output before the judge is asked.

The rubric and the verify script are the agent's definition's as it was when the job started, pinned in the job's history (the trace shows outcome check pinned), so a session that rewrites either in its work directory, or a plugin upgrade during the job, does not change how the job is judged; a changeset that touches them is refused (Reviewed changes).

The judge's model is resolved when the check first runs, like an agent's alias, and every judge call is a model call of its own, metered on the job. A judge on a small, cheap model checking an agent on a large one is the usual setup.

What you see

run and job get say how the check went:

Outcome: passed in round 2 of 3 (judge judge: fail, pass)

The job's trace has a judge step for each verdict, with its model, tokens and cost; opening one shows the reasons, for those who may see content. The findings say outcome_passed ("passed its outcome check in round 2, after 1 revision") or, as a warning, outcome_not_met, and they are in the job's summary, so the jobs list can filter on them.

The job's live stream gets a verdict event for each round, and the job tree's board a note of kind verdict ("writer's answer, round 1 of 3: did not pass (reviewer): ..."), which the tree's agents can read with oz.board_read and the job page shows on its Board tab. An agent with data: local gets only how the round went on the board, not the reasons.

Track quality and regressions

Schedule evals, investigate regressions and keep a failed job as a reproducible case.

Choose a judge and rubric

Choose an independent judge, a rubric and the evidence the judge may read.

On this page

Outcome checks and patterns

In this guide

Outcome checks

What you see

Previous section links