--- title: "Judges and candidate selection" description: "Agent definition fields for judges and candidate selection." canonical_url: "https://docs.oz0.ai/reference/agent-definition/quality" --- # Judges and candidate selection > Agent definition fields for judges and candidate selection. For: Plugin authors looking up agent fields. Look up the fields below when editing an agent definition. The model that checks each answer against the rubric: a tenant's alias, such as `smart`, or `/`. Its call is its own, metered on the job, and never part of the agent's conversation. It must resolve to another model than the agent's: a judge on the agent's own model fails the job with `JudgeSameModel` before it is asked, and so does a reviewer on the writer's model and a best-of's judge on a candidate's. See [A judge on another model](https://docs.oz0.ai/improve/outcome-checks/judges#a-judge-on-another-model-than-the-agent). An agent that checks each answer instead of a model (writer and reviewer): it runs as a child job and answers with a verdict. Set `judge` or `reviewer`, not both; or neither, with a `grader`. A reviewer has no `input_schema` or `output_schema`. What a good answer must do. `outcome.rubric_file` names a file of it instead, relative to the plugin's root. With neither, a judge or a reviewer checks the answer against the [default deliverable rubric](https://docs.oz0.ai/improve/outcome-checks/judges#the-default-rubric), whose text the plugin's version carries. `read` lets the judge read, read-only, the files of the job's own tree that the task, the tool results or the answer name, never the conversation; each read is audited (`run.file_read`). The one value is `read`; unset, the judge sees the rubric, the task and the answer. Only with `outcome.judge`: a reviewer reads files with its own tools, and `best_of` takes no `files`. See [A judge that reads the job's files](https://docs.oz0.ai/improve/outcome-checks/judges#a-judge-that-reads-the-jobs-files). A harness agent's shell script, relative to the plugin's root, that the session's work directory must pass before the judge looks: it runs in the work directory after each session, and a non-zero exit fails the round with its output. Its text is read when the plugin loads and pinned when the job starts, so a job that rewrites it in its work directory is still judged by the original. See [Reviewed changes](https://docs.oz0.ai/govern/reviewed-changes/protected-paths#the-job-never-changes-its-own-check). A harness agent ends with a changeset of its work directory instead of writing to its target. Its settings are: - `review`: `required`, or `policy` with `policy: {paths, max_files, max_bytes, executables}`. - `apply`: the `.` that applies an approved changeset, such as `oz0_github.push_changes`. - `arguments`: more arguments for that call. - `base`, `commit` and `repository`: the job's input fields containing the repository archive, commit and name. Without an archive, a checkout the session makes is diffed against that commit. - `target` and `evidence`: the job's input fields containing the target plugin and the findings, eval runs and comparisons supporting the change. [The change's record](https://docs.oz0.ai/govern/reviewed-changes/changesets#where-a-change-came-from) keeps them when they belong to the tenant. See [Reviewed changes](https://docs.oz0.ai/govern/reviewed-changes). What a failed re-check does when an approval resolves after the tenant's threshold (an hour by default): `refuse` (the default) refuses the call, which the model reads with why; `fail` fails the job (`RecheckFailed`). See [Checked again after a long wait](https://docs.oz0.ai/agents/approvals/security#checked-again-after-a-long-wait). How many verdicts a job may ask for, the first answer's included. Unset means 3, or one per model with `model.cascade`; at most 10. An answer that has not passed by then fails the job with `OutcomeNotMet`. A YAML file of the judge's anchors, relative to the plugin's root: 3 to 30 `{name, task, answer, pass}`, the verdict a person gave. The judge judges them again at most daily when an eval routine runs the agent's evals, so a judge that drifted is told apart from an agent that did. Only with `outcome.judge`. See [Anchors](https://docs.oz0.ai/improve/outcome-checks/sampling#anchors-is-it-the-judge-or-the-agent). Sampled judging of the agent's finished jobs, after the fact: `rate` (0 to 1), `judge` (an alias or `/`), `rubric` or `rubric_file`, an optional `anchors_file`, `budget_usd` per judge job (0.05 by default, at most 1) and `daily_budget_usd` per UTC day (1 by default, at most 100). A signal and a finding, never a block; jobs about a data subject only where the tenant allows it. Not beside `outcome:`; a `data: local` agent's judge must be local. A judge on the agent's own model still judges, with a `judge_same_model` finding, since its verdict is a weaker signal. See [Sampled judging](https://docs.oz0.ai/improve/outcome-checks/sampling#sampled-judging-of-finished-jobs). Checks each round's answer without a model call: `contains`, `not_contains`, `equals` and one `cel` expression over `output`, `calls` (the job's model requests so far; for a harness agent, every session's turns) and `seconds` (since the job started), as a best-of's grader does, on every runtime. Alone it needs no rubric; with a `judge` or a `reviewer` it runs first, and a failed grader is the round's verdict. See [A grader instead of a judge](https://docs.oz0.ai/improve/outcome-checks/judges#a-grader-instead-of-a-judge). How many candidates of each job run in parallel, 2 to 10. Unset means one per entry in `models` or `agents`, or 3. What the candidates run on, in turn: aliases or `/`. Unset runs the agent's own model N times. With `data: local`, only `local/` models. For `pydantic-ai`, `openai-agents` and harness agents; a remote agent chooses its own model. Other agents, harness agents included, that take the job as candidates instead. They must have the agent's input and output schemas. The agent's own `job_start` and `job_end` hooks run once around them. The model that picks the best candidate against `best_of.rubric` (or `best_of.rubric_file`), or, with neither, the [default deliverable rubric](https://docs.oz0.ai/improve/outcome-checks/judges#the-default-rubric). It reads the candidates without their models' or agents' names, in an order shuffled per job. Picks without a model instead: `contains`, `not_contains`, `equals` and `cel` (over `output`, `calls` and `seconds`), as an eval case's `expect`. The cheapest candidate that passes is kept. [Back to the agent definition](https://docs.oz0.ai/reference/agent-definition). ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.