Choose a judge and rubric

For: Plugin authors checking answer quality.

Choose an independent judge, a rubric and the evidence the judge may read.

A judge on another model than the agent

A judge must not run on the model whose answer it judges: a model asked whether its own answer is good tends to say yes, so its verdict would vouch for itself. The platform checks it where it can see both models:

• When a job runs. The judge's model and the agent's are compared once both are resolved: the agent's as its job resolved it, through the tenant's mapping rules, and the judge's alias or model. They are the same model when they name the same provider record and model, or the same model behind records that share an endpoint (a record's base URL, or any of its pool's members).
  A judge or a reviewer on the agent's model, or on a model of its cascade, fails the job with JudgeSameModel before the judge is asked: the check gates the job's answer, so it cannot also be the thing it checks. The error names both models and why they are one; give the judge another model or alias, or map its alias to another model.
  A best-of's judge on any candidate's model fails the same way, before any candidate runs.
• At install. plugin install resolves each judge and the agent it judges as the tenant's aliases and mapping rules are now, and warns where they are one model. It does not refuse: the aliases and rules can change after the install, and the job's check is the one that binds.
• At lint. plugin lint warns where a judge names the same model or alias as its agent, its reviewer the same as its writer, or a best-of's judge the same as a candidate. It cannot see the tenant's aliases or rules, so it says the install and the job check again.

The comparison is by the tenant's records: the same vendor model behind two records that share nothing (a vendor's own API and a reseller's) is not recognised as one. The trace marks the step whose answer said so, the job's findings have a judge_same_model warning, and the audit log records run.judge_same_model with the judge's kind and both models.

This keeps a model from vouching for its own work. It does not make a judge a security control: a judge on another model can still be wrong, or be steered by what it reads.

The default rubric

An outcome check with a judge or a reviewer, and a best-of with a judge, that names neither rubric nor rubric_file is judged against the default deliverable rubric:

outcome:
  judge: judge            # no rubric: the default deliverable rubric

Its six criteria are in rank order, each outranking every one after it, so an answer that is polished but misses what was asked for never beats a plainer one that delivers it. This is its text, about 300 tokens, which the judge reads on every call:

Judge the deliverable, not the effort behind it. Check the answer against these six criteria in this order. Each criterion outranks every criterion after it: an answer that misses an earlier one is worse than an answer that misses only later ones, however well it does on the rest.

1. delivered: the thing the task asked for exists in the answer, or in a file the answer names. A plan, a promise or a description of the thing is not the thing.
2. shape: it has the form the task asked for: the format, the length limits, the sections and the fields.
3. fidelity: what it says agrees with the sources the job used, as far as you can see them: the task, the files you are shown and what they hold. Nothing is made up, and nothing a source says is turned around.
4. consistency: it does not contradict itself. Names, numbers, dates and claims agree from one part to the next.
5. usability: the person who asked can use it as it is, without asking again or fixing it first.
6. finish: nothing is left undone: no placeholders, notes to self, broken formatting or cut-off parts.

Length and polish earn nothing on their own: a longer or smoother answer is not a better one.

Fail an answer on the first criterion it misses, and say which criterion that is. When you compare answers, prefer the one whose first miss comes latest.

• Pinned with the version. plugin lint and the install fill the text into the plugin's bundle and mark it, so a version is judged by the rubric it was loaded with, an older node judges with the same text, and a later release's change to the rubric reaches a plugin only with its next version. plugin inspect says which agents use it.
• Which criterion. A failed verdict names the first criterion the answer missed (delivered, shape, fidelity, consistency, usability or finish). A check with a rubric of the agent's own always counts its misses as own, so its criteria are never counted as the default's. It is in the job's outcome, on the judge's step in the trace and in the job's findings as outcome_missed_<criterion>, so oz0 job search "finding:outcome_missed_fidelity agent:acme-writer" finds the jobs per agent and version.
• A rubric of your own replaces the default; write one only for what the default cannot know, such as a house style or a required citation. A grader alone takes no rubric, as before.

A judge that reads the job's files

When the agent's work is a file, or its answer must agree with the files it read, the judge cannot check fidelity from the task and the answer alone. files: read lets it read them:

outcome:
  judge: judge
  files: read             # the judge reads the job's files, read-only

• Which files. The files of the job's own tree that the task, the tool results or the answer name by their oz0-file:sha256: address, each shown to the judge with its role: input (named by the task), source (by a tool's result) or answer (by the answer). Never the conversation around them: the judge still cannot see what the agent said or how it got there.
• How. The judge reads them inside its own step, through the edge, with that step's own credentials, so a file only reaches a judge that its job's tree may read, as for any file between tools. The edge refuses a judge's write, and the audit log records each read as run.file_read under the judge's step, with its size and digest, never its content.
• Bounds. At most 8 files a round, 64 KiB a file and 256 KiB in all; text only. A file of another kind is listed by its address, type and size; a file past a bound is named as skipped.
• Untrusted data. The files are in blocks the judge is told never to follow. Their labels join the job's own on the judge's model call: when the tenant's data rules do not allow the judge's model for that label, the round fails closed with a reason that names the label, and the file is never sent.
• Labels follow what the judge read. A judge's reasons can quote a file, so the labels of the files it was shown join the job's labels before the reasons reach the agent's next round, the board or the job's answer: the agent's next model call carries them, and the gateway's data rules apply to it as to any call.
  A file above the job's clearance or the agent's own, or whose compartments a wall separates from the job's, is not shown; the judge reads only that it was not cleared for it, never its label.
• Data that stays on the node. An agent with data: local keeps its local/ judge, files or not.
• Off by default. Without files, the judge sees what it always has: the rubric, the task and the answer. A node whose runtime predates the key judges without files. A reviewer is a job of the tree and reads files with its own tools, so files goes with a judge only, and a best-of's judge, which compares answers, reads none. Judge jobs of sampled judging run in a tree of their own and read no files of the job they judge.

A grader instead of a judge

An outcome check can also use a grader: the same checks a best-of's grader makes (contains, not_contains, equals, and one cel expression over output, calls and seconds), with no model call. Alone, it needs no rubric:

outcome:
  grader:
    contains: ["## Breaking changes"]
    cel: 'calls < 12'          # the job's model requests so far; seconds counts from the job's start

A grader works the same on pydantic-ai, openai-agents and harness agents. For a harness agent, calls counts the turns of every session the job has run so far.

With a judge or a reviewer, the grader runs first in each round, and only an answer it passes goes to the judge: a failed grader is that round's verdict, with its reason ("the answer does not contain '## Breaking changes'"), and costs nothing. run and job get show such a check as (grader: fail, pass).

Cascades

With a model cascade (model.cascade), a round that does not pass restarts on the next model as a new conversation, with the task, the answer that failed, the reasons and the earlier rounds' tool calls and their results, instead of going back to the same model. A grader makes this cheap: the larger model runs only for answers the grader turns down, and no judge is paid for the ones it passes.

With a cascade, max_rounds defaults to one round per model, and a smaller value is refused by plugin lint, since the last model could never run.

The job's trace has a cascade step for each restart ("round 2 on anthropic/claude-sonnet-5-5", after the grader or the judge), and the finding model_cascade says which model answered. Neither is a model_fallback: each round ran on the model its definition named for it.

Back to Outcome checks and patterns.