Choose among candidate answers

For: Plugin authors checking answer quality.

Run candidate answers and select one, within the agent's hooks and data restrictions.

Best-of-N

name: acme-titler
runtime: pydantic-ai
model:
  primary: anthropic/claude-sonnet-5-5
best_of:
  models: [smart, fast, openai/gpt-5.5]   # one candidate on each
  judge: judge
  rubric: The shortest title that still says what changed.

A job of acme-titler runs three candidates in parallel, each a child job of the agent on one of the models, and asks the judge to pick one against the rubric. The answer of the candidate it picks is the job's answer; the others stay in the trace as children with the role candidate, and the one kept says picked.

Without models, the candidates run the agent's own model n times (3 by default); with agents: [a, b], other agents take the job, harness agents included. models works for pydantic-ai, openai-agents and harness agents: an ACP session runs on the candidate's model, the Claude harness on a candidate's Anthropic model (an alias resolves to one), and another JSON-lines harness gets the model's <provider>/<model>.

A remote agent's model is its provider's, so plugin lint refuses models for it (and its jobs fail with BestOfUnsupported); give it n or agents. Candidate agents must take and give what the agent does (its input and output schemas), since the answer kept is the agent's answer: a candidate whose answer breaks the agent's output schema is never kept.

The judge sees every candidate's answer without its model's or its agent's name, and no candidate is marked as the agent's own run: every candidate is a child job like the others. It reads them in an order shuffled for each job, seeded from the job's ID, so a candidate's place says nothing about where it came from, and a replay of the job sees the same order. A judge without a rubric of its own picks by the default rubric.

Instead of a judge, a grader picks without a model call, with the checks of an eval case:

best_of:
  n: 4
  grader:
    contains: ["## Summary"]
    cel: "calls <= 6"           # over output, calls and seconds

The cheapest candidate that passes, by the tokens of its own model calls, is kept. When none passes, the job fails with BestOfNoPick and why each one did not. A harness agent's session reports no tokens, so a harness candidate counts as the cheapest; tokens also do not weigh one model's price against another's.

The cost of a best-of is the sum of its candidates and the judge, all in the job's tree, so a budget on the job covers them all. job get says which candidate was kept:

Best of: candidate 2 of 3 kept, picked by judge judge (job job-42c7.../acme-titler-c73a2742)

A best-of can also be declared where a job starts instead of on the agent:

• On a delegation: delegation.best_of runs every delegation of the agent as candidates of the agent it delegates to.
• On a flow step: an agent step with best_of: runs that agent as candidates.

steps:
  - id: notes
    agent: acme-writer
    input: ${{ input }}
    best_of:
      n: 3
      judge: judge
      rubric_file: rubrics/release-notes.md

A routine's fire runs its agent once, without a best-of, since its safe outputs belong to its own run.

Hooks

A best-of runs the hooks of the agent it was declared for once, around all its candidates: the job_start hooks before the candidates start, which then take the input those hooks return, and the job_end hooks on the answer kept. A candidate skips the job_start and job_end hooks that match that agent, since they ran already, and runs the ones that match only its own agent.

So a hook that redacts the input of support-bot, or asks a person before it runs, still sees every support-bot job when its best-of names other agents as candidates, and asks once, not once per candidate.

Data that stays on the node

For an agent with data: local, a best-of and an outcome check keep the data on the node too: every candidate's model and every judge must be a local/ model, and a reviewer or candidate agent must be a data: local agent on the same node. A job that would break this fails with DataLocal before any model call, whether the best-of is declared on the agent, on a flow step or on a delegation, and plugin lint checks it against the agent the best-of runs on.

Back to Outcome checks and patterns.