Use a reviewer agent

For: Plugin authors checking answer quality.

Use an agent with tools to review the writer's answer.

Writer and reviewer

Name an agent as the reviewer instead of a model as the judge:

name: acme-author
runtime: pydantic-ai
model:
  use: smart
outcome:
  reviewer: acme-critic
  rubric: The change has a test, and the test fails without the change.
  max_rounds: 4

name: acme-critic
description: Reviews changes against a rubric.
runtime: pydantic-ai
model:
  use: smart
tools: [acme_repo.read_file, acme_repo.run_tests]

Each round, the reviewer runs as a child job of the writer's, with the rubric, the task and the answer as its input, and its run answers with the same verdict a judge gives. Unlike a judge, a reviewer has tools: it can read the files, run the tests, or ask a person. Every review is its own job in the tree, with its own cost, and the trace shows it as a child with the role reviewer and its verdict.

A reviewer must not have an output_schema, since its answer is the verdict, nor an input_schema, since its input is the rubric, the task and the answer as text. A harness agent can review too: its answer must start with PASS or FAIL, or be only a JSON verdict, such as {"pass": false, "reasons": ["..."]}. JSON inside a longer answer does not count, so a reviewer that quotes the writer's answer is read by its first word.

Writer and reviewer is a declaration on the writer, not a step type: a flow gets it by running the writer as an agent step.

Back to Outcome checks and patterns.