---
title: "Choose among candidate answers"
description: "Run candidate answers and select one, within the agent's hooks and data restrictions."
canonical_url: "https://docs.oz0.ai/improve/outcome-checks/selection"
---
# Choose among candidate answers

> Run candidate answers and select one, within the agent's hooks and data restrictions.

For: Plugin authors checking answer quality.

Run candidate answers and select one, within the agent's hooks and data restrictions.

## Best-of-N

```yaml [agents/acme-titler.yaml]
name: acme-titler
runtime: pydantic-ai
model:
  primary: anthropic/claude-sonnet-5-5
best_of:
  models: [smart, fast, openai/gpt-5.5]   # one candidate on each
  judge: judge
  rubric: The shortest title that still says what changed.
```

A job of `acme-titler` runs three candidates in parallel, each a child job of the agent on one of the models, and asks the judge to pick one against the rubric. The answer of the candidate it picks is the job's answer; the others stay in the trace as children with the role *candidate*, and the one kept says *picked*.

Without `models`, the candidates run the agent's own model `n` times (3 by default); with `agents: [a, b]`, other agents take the job, harness agents included. `models` works for `pydantic-ai`, `openai-agents` and harness agents: an ACP session runs on the candidate's model, the Claude harness on a candidate's Anthropic model (an alias resolves to one), and another JSON-lines harness gets the model's `<provider>/<model>`.

A remote agent's model is its provider's, so `plugin lint` refuses `models` for it (and its jobs fail with `BestOfUnsupported`); give it `n` or `agents`. Candidate agents must take and give what the agent does (its input and output schemas), since the answer kept is the agent's answer: a candidate whose answer breaks the agent's output schema is never kept.

The judge sees every candidate's answer without its model's or its agent's name, and no candidate is marked as the agent's own run: every candidate is a child job like the others. It reads them in an order shuffled for each job, seeded from the job's ID, so a candidate's place says nothing about where it came from, and a replay of the job sees the same order. A judge without a rubric of its own picks by the [default rubric](https://docs.oz0.ai/improve/outcome-checks/judges#the-default-rubric).

Instead of a judge, a **grader** picks without a model call, with the checks of an [eval case](https://docs.oz0.ai/improve/evals):

```yaml
best_of:
  n: 4
  grader:
    contains: ["## Summary"]
    cel: "calls <= 6"           # over output, calls and seconds
```

The cheapest candidate that passes, by the tokens of its own model calls, is kept. When none passes, the job fails with `BestOfNoPick` and why each one did not. A harness agent's session reports no tokens, so a harness candidate counts as the cheapest; tokens also do not weigh one model's price against another's.

The cost of a best-of is the sum of its candidates and the judge, all in the job's tree, so a budget on the job covers them all. `job get` says which candidate was kept:

```text
Best of: candidate 2 of 3 kept, picked by judge judge (job job-42c7.../acme-titler-c73a2742)
```

A best-of can also be declared where a job starts instead of on the agent:

- **On a delegation**: `delegation.best_of` runs every delegation of the agent as candidates of the agent it delegates to.
- **On a flow step**: an agent step with `best_of:` runs that agent as candidates.

```yaml [flows/release.yaml]
steps:
  - id: notes
    agent: acme-writer
    input: ${{ input }}
    best_of:
      n: 3
      judge: judge
      rubric_file: rubrics/release-notes.md
```

A routine's fire runs its agent once, without a best-of, since its safe outputs belong to its own run.

## Hooks

A best-of runs the [hooks](https://docs.oz0.ai/build/parts/hooks) of the agent it was declared for once, around all its candidates: the `job_start` hooks before the candidates start, which then take the input those hooks return, and the `job_end` hooks on the answer kept. A candidate skips the `job_start` and `job_end` hooks that match that agent, since they ran already, and runs the ones that match only its own agent.

So a hook that redacts the input of `support-bot`, or asks a person before it runs, still sees every `support-bot` job when its best-of names other agents as candidates, and asks once, not once per candidate.

## Data that stays on the node

For an agent with `data: local`, a best-of and an outcome check keep the data on the node too: every candidate's model and every judge must be a `local/` model, and a reviewer or candidate agent must be a `data: local` agent on the same node. A job that would break this fails with `DataLocal` before any model call, whether the best-of is declared on the agent, on a flow step or on a delegation, and `plugin lint` checks it against the agent the best-of runs on.

[Back to Outcome checks and patterns](https://docs.oz0.ai/improve/outcome-checks).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.