Evals and quality gates

For: Plugin authors testing agents and flows.

Evals are test cases in your plugin: an input for an agent or a flow, and what a good answer looks like. oz0 plugin test runs them on a dev server and a node of its own, and model answers can be recorded once and replayed after, so tests in CI are fast, free and repeatable.

agent: greeter            # or flow: <name>
threshold: 0.8            # the share of cases that must pass; all of them by default
cases:
  - name: greets by name
    input: Please greet Ada
    expect:
      contains: ["Hello, Ada"]
      cel: "calls <= 3 && cost_usd < 0.02"
  - name: refuses an empty name
    input: { name: "" }
    expect:
      status: failed
      error_contains: InvalidInput

List eval files or folders in the manifest with evals: [evals/]. plugin lint checks them, CEL included.

In this guide

• Case expectations: Specify the answer, tool choices, constraints and approval decisions a case must satisfy.
• Record and replay answers: Record model answers and flow outputs, then reuse them in repeatable tests.
• Check policy paths: Check which policy branches your cases exercise.
• Gate a plugin update: A candidate plugin version must pass its evals before the quality gate admits it.
• Track quality and regressions: Schedule evals, investigate regressions and keep a failed job as a reproducible case.

Run them

oz0 plugin test ./greeter            # every eval of the plugin
oz0 plugin test ./greeter --eval greetings

  PASS  greetings / greets by name (replayed, 2 calls, $0.0011, 0.5 s)
  FAIL  greetings / says goodbye (replayed, 2 calls, $0.0011, 0.3 s): the answer does not contain "Goodbye"
greetings (greeter): 2 of 3 passed (67%), threshold 60%: ok; $0.0032, 1.2 s

The command starts a dev server and a node, installs the plugin from its working tree, and runs the cases one at a time as jobs. It exits non-zero when an eval is below its threshold, so it can gate a pull request. --json prints the report for machines. It needs what oz0 dev needs: the node binary next to the CLI and a runtime bundle.

The dev server's tenant runs with the taint-default rule turned off: a case whose agent reads an open-world result and then calls another outbound tool would otherwise need an answer in every such case. An eval that ships a tool policy runs under that policy as written, with the rule on unless it says "taint_default": "off", and its cases answer what the rule asks with expect.approvals.

Previous section links

• a-model-changed-behind-its-name: open this section.
• approvals-in-a-case: open this section.
• constraints: open this section.
• drift-which-of-the-model-the-judge-or-the-agent-changed: open this section.
• each-case-runs-three-times: open this section.
• how-a-regression-is-decided: open this section.
• keep-a-job-as-an-eval-case: open this section.
• one-node-tests-it: open this section.
• policy-path-coverage: open this section.
• quality-gates: open this section.
• quality-over-time: open this section.
• record-once-replay-after: open this section.
• recorded-step-outputs-of-flows: open this section.
• scheduled-evals: open this section.
• skills-no-job-reads: open this section.
• the-evals-run-there: open this section.
• the-verdict: open this section.
• the-version-waits-as-the-plugins-candidate: open this section.
• what-a-case-can-expect: open this section.
• which-tools-and-skills-it-chose: open this section.