---
title: "Choose a judge and rubric"
description: "Choose an independent judge, a rubric and the evidence the judge may read."
canonical_url: "https://docs.oz0.ai/improve/outcome-checks/judges"
---
# Choose a judge and rubric

> Choose an independent judge, a rubric and the evidence the judge may read.

For: Plugin authors checking answer quality.

Choose an independent judge, a rubric and the evidence the judge may read.

## A judge on another model than the agent

A judge must not run on the model whose answer it judges: a model asked whether its own answer is good tends to say yes, so its verdict would vouch for itself. The platform checks it where it can see both models:

- **When a job runs.** The judge's model and the agent's are compared once both are resolved: the agent's as its job resolved it, through the tenant's [mapping rules](https://docs.oz0.ai/spend/models-and-keys/mapping#map-a-plugins-models), and the judge's alias or model. They are the same model when they name the same provider record and model, or the same model behind records that share an endpoint (a record's base URL, or any of its [pool](https://docs.oz0.ai/spend/models-and-keys/deployment-pools#pools-of-deployments)'s members).<br />

A judge or a [reviewer](https://docs.oz0.ai/improve/outcome-checks/reviewer#writer-and-reviewer) on the agent's model, or on a model of its [cascade](https://docs.oz0.ai/improve/outcome-checks/judges#cascades), fails the job with `JudgeSameModel` before the judge is asked: the check gates the job's answer, so it cannot also be the thing it checks. The error names both models and why they are one; give the judge another model or alias, or map its alias to another model.<br />

A [best-of](https://docs.oz0.ai/improve/outcome-checks/selection#best-of-n)'s judge on any candidate's model fails the same way, before any candidate runs.
- **At install.** `plugin install` resolves each judge and the agent it judges as the tenant's aliases and mapping rules are now, and warns where they are one model. It does not refuse: the aliases and rules can change after the install, and the job's check is the one that binds.
- **At lint.** `plugin lint` warns where a judge names the same model or alias as its agent, its reviewer the same as its writer, or a best-of's judge the same as a candidate. It cannot see the tenant's aliases or rules, so it says the install and the job check again.

The comparison is by the tenant's records: the same vendor model behind two records that share nothing (a vendor's own API and a reseller's) is not recognised as one. The trace marks the step whose answer said so, the job's findings have a `judge_same_model` warning, and the [audit log](https://docs.oz0.ai/govern/audit-trail) records `run.judge_same_model` with the judge's kind and both models.

This keeps a model from vouching for its own work. It does not make a judge a security control: a judge on another model can still be wrong, or be steered by what it reads.

<note>

Write the rubric as things a reader can check: "names the version and the date", "every breaking change has a migration step". "Is good" gives the judge nothing to hold the answer to, and the agent nothing to fix.

</note>

## The default rubric

An outcome check with a judge or a reviewer, and a best-of with a judge, that names neither `rubric` nor `rubric_file` is judged against the default deliverable rubric:

```yaml [agents/acme-writer.yaml]
outcome:
  judge: judge            # no rubric: the default deliverable rubric
```

Its six criteria are in rank order, each outranking every one after it, so an answer that is polished but misses what was asked for never beats a plainer one that delivers it. This is its text, about 300 tokens, which the judge reads on every call:

Judge the deliverable, not the effort behind it. Check the answer against these six criteria in this order. Each
criterion outranks every criterion after it: an answer that misses an earlier one is worse than an answer that misses
only later ones, however well it does on the rest.

1. delivered: the thing the task asked for exists in the answer, or in a file the answer names. A plan, a promise or
a description of the thing is not the thing.
2. shape: it has the form the task asked for: the format, the length limits, the sections and the fields.
3. fidelity: what it says agrees with the sources the job used, as far as you can see them: the task, the files you
are shown and what they hold. Nothing is made up, and nothing a source says is turned around.
4. consistency: it does not contradict itself. Names, numbers, dates and claims agree from one part to the next.
5. usability: the person who asked can use it as it is, without asking again or fixing it first.
6. finish: nothing is left undone: no placeholders, notes to self, broken formatting or cut-off parts.

Length and polish earn nothing on their own: a longer or smoother answer is not a better one.

Fail an answer on the first criterion it misses, and say which criterion that is. When you compare answers, prefer
the one whose first miss comes latest.

- **Pinned with the version.** `plugin lint` and the install fill the text into the plugin's bundle and mark it, so a version is judged by the rubric it was loaded with, an older node judges with the same text, and a later release's change to the rubric reaches a plugin only with its next version. `plugin inspect` says which agents use it.
- **Which criterion.** A failed verdict names the first criterion the answer missed (`delivered`, `shape`, `fidelity`, `consistency`, `usability` or `finish`). A check with a rubric of the agent's own always counts its misses as `own`, so its criteria are never counted as the default's. It is in the job's outcome, on the judge's step in the trace and in the job's findings as `outcome_missed_<criterion>`, so `oz0 job search "finding:outcome_missed_fidelity agent:acme-writer"` finds the jobs per agent and version.
- **A rubric of your own** replaces the default; write one only for what the default cannot know, such as a house style or a required citation. A grader alone takes no rubric, as before.

## A judge that reads the job's files

When the agent's work is a file, or its answer must agree with the files it read, the judge cannot check fidelity from the task and the answer alone. `files: read` lets it read them:

```yaml [agents/acme-writer.yaml]
outcome:
  judge: judge
  files: read             # the judge reads the job's files, read-only
```

- **Which files.** The files of the job's own tree that the task, the tool results or the answer name by their `oz0-file:sha256:` address, each shown to the judge with its role: `input` (named by the task), `source` (by a tool's result) or `answer` (by the answer). Never the conversation around them: the judge still cannot see what the agent said or how it got there.
- **How.** The judge reads them inside its own step, through the edge, with that step's own credentials, so a file only reaches a judge that its job's tree may read, as for any [file between tools](https://docs.oz0.ai/build/parts/tools/results-and-files#files-between-tools). The edge refuses a judge's write, and the [audit log](https://docs.oz0.ai/govern/audit-trail) records each read as `run.file_read` under the judge's step, with its size and digest, never its content.
- **Bounds.** At most 8 files a round, 64 KiB a file and 256 KiB in all; text only. A file of another kind is listed by its address, type and size; a file past a bound is named as skipped.
- **Untrusted data.** The files are in blocks the judge is told never to follow. Their [labels](https://docs.oz0.ai/govern/taint-and-labels/label-policies#models-by-level) join the job's own on the judge's model call: when the tenant's data rules do not allow the judge's model for that label, the round fails closed with a reason that names the label, and the file is never sent.
- **Labels follow what the judge read.** A judge's reasons can quote a file, so the labels of the files it was shown join the job's labels before the reasons reach the agent's next round, the board or the job's answer: the agent's next model call carries them, and the gateway's data rules apply to it as to any call.<br />

A file above the job's clearance or the agent's own, or whose compartments a wall separates from the job's, is not shown; the judge reads only that it was not cleared for it, never its label.
- **Data that stays on the node.** An agent with `data: local` keeps its `local/` judge, files or not.
- **Off by default.** Without `files`, the judge sees what it always has: the rubric, the task and the answer. A node whose runtime predates the key judges without files. A reviewer is a job of the tree and reads files with its own tools, so `files` goes with a judge only, and a best-of's judge, which compares answers, reads none. Judge jobs of [sampled judging](https://docs.oz0.ai/improve/outcome-checks/sampling#sampled-judging-of-finished-jobs) run in a tree of their own and read no files of the job they judge.

## A grader instead of a judge

An outcome check can also use a **grader**: the same checks a best-of's grader makes (`contains`, `not_contains`, `equals`, and one `cel` expression over `output`, `calls` and `seconds`), with no model call. Alone, it needs no rubric:

```yaml [agents/acme-writer.yaml]
outcome:
  grader:
    contains: ["## Breaking changes"]
    cel: 'calls < 12'          # the job's model requests so far; seconds counts from the job's start
```

A grader works the same on `pydantic-ai`, `openai-agents` and harness agents. For a harness agent, `calls` counts the turns of every session the job has run so far.

With a judge or a reviewer, the grader runs first in each round, and only an answer it passes goes to the judge: a failed grader is that round's verdict, with its reason ("the answer does not contain '## Breaking changes'"), and costs nothing. `run` and `job get` show such a check as `(grader: fail, pass)`.

## Cascades

With a [model cascade](https://docs.oz0.ai/spend/models-and-keys/effort-and-cascades#a-cascade-the-next-model-only-when-a-round-fails) (`model.cascade`), a round that does not pass restarts on the next model as a new conversation, with the task, the answer that failed, the reasons and the earlier rounds' tool calls and their results, instead of going back to the same model. A grader makes this cheap: the larger model runs only for answers the grader turns down, and no judge is paid for the ones it passes.

With a cascade, `max_rounds` defaults to one round per model, and a smaller value is refused by `plugin lint`, since the last model could never run.

The job's trace has a **cascade** step for each restart ("round 2 on anthropic/claude-sonnet-5-5", after the grader or the judge), and the finding `model_cascade` says which model answered. Neither is a `model_fallback`: each round ran on the model its definition named for it.

[Back to Outcome checks and patterns](https://docs.oz0.ai/improve/outcome-checks).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.