Judges and candidate selection

The model that checks each answer against the rubric: a tenant's alias, such as smart, or <provider>/<model>. Its call is its own, metered on the job, and never part of the agent's conversation. It must resolve to another model than the agent's: a judge on the agent's own model fails the job with JudgeSameModel before it is asked, and so does a reviewer on the writer's model and a best-of's judge on a candidate's. See A judge on another model.

An agent that checks each answer instead of a model (writer and reviewer): it runs as a child job and answers with a verdict. Set judge or reviewer, not both; or neither, with a grader. A reviewer has no input_schema or output_schema.

What a good answer must do. outcome.rubric_file names a file of it instead, relative to the plugin's root. With neither, a judge or a reviewer checks the answer against the default deliverable rubric, whose text the plugin's version carries.

read lets the judge read, read-only, the files of the job's own tree that the task, the tool results or the answer name, never the conversation; each read is audited (run.file_read). The one value is read; unset, the judge sees the rubric, the task and the answer. Only with outcome.judge: a reviewer reads files with its own tools, and best_of takes no files. See A judge that reads the job's files.

A harness agent's shell script, relative to the plugin's root, that the session's work directory must pass before the judge looks: it runs in the work directory after each session, and a non-zero exit fails the round with its output. Its text is read when the plugin loads and pinned when the job starts, so a job that rewrites it in its work directory is still judged by the original. See Reviewed changes.

A harness agent ends with a changeset of its work directory instead of writing to its target. Its settings are:

• review: required, or policy with policy: {paths, max_files, max_bytes, executables}.
• apply: the <tool>.<function> that applies an approved changeset, such as oz0_github.push_changes.
• arguments: more arguments for that call.
• base, commit and repository: the job's input fields containing the repository archive, commit and name. Without an archive, a checkout the session makes is diffed against that commit.
• target and evidence: the job's input fields containing the target plugin and the findings, eval runs and comparisons supporting the change. The change's record keeps them when they belong to the tenant.

See Reviewed changes.

What a failed re-check does when an approval resolves after the tenant's threshold (an hour by default): refuse (the default) refuses the call, which the model reads with why; fail fails the job (RecheckFailed). See Checked again after a long wait.

How many verdicts a job may ask for, the first answer's included. Unset means 3, or one per model with model.cascade; at most 10. An answer that has not passed by then fails the job with OutcomeNotMet.

A YAML file of the judge's anchors, relative to the plugin's root: 3 to 30 {name, task, answer, pass}, the verdict a person gave. The judge judges them again at most daily when an eval routine runs the agent's evals, so a judge that drifted is told apart from an agent that did. Only with outcome.judge. See Anchors.

Sampled judging of the agent's finished jobs, after the fact: rate (0 to 1), judge (an alias or <provider>/<model>), rubric or rubric_file, an optional anchors_file, budget_usd per judge job (0.05 by default, at most 1) and daily_budget_usd per UTC day (1 by default, at most 100). A signal and a finding, never a block; jobs about a data subject only where the tenant allows it. Not beside outcome:; a data: local agent's judge must be local. A judge on the agent's own model still judges, with a judge_same_model finding, since its verdict is a weaker signal. See Sampled judging.

Checks each round's answer without a model call: contains, not_contains, equals and one cel expression over output, calls (the job's model requests so far; for a harness agent, every session's turns) and seconds (since the job started), as a best-of's grader does, on every runtime. Alone it needs no rubric; with a judge or a reviewer it runs first, and a failed grader is the round's verdict. See A grader instead of a judge.

How many candidates of each job run in parallel, 2 to 10. Unset means one per entry in models or agents, or 3.

What the candidates run on, in turn: aliases or <provider>/<model>. Unset runs the agent's own model N times. With data: local, only local/ models. For pydantic-ai, openai-agents and harness agents; a remote agent chooses its own model.

Other agents, harness agents included, that take the job as candidates instead. They must have the agent's input and output schemas. The agent's own job_start and job_end hooks run once around them.

The model that picks the best candidate against best_of.rubric (or best_of.rubric_file), or, with neither, the default deliverable rubric. It reads the candidates without their models' or agents' names, in an order shuffled per job.

Picks without a model instead: contains, not_contains, equals and cel (over output, calls and seconds), as an eval case's expect. The cheapest candidate that passes is kept.