---
title: "Evals and quality gates"
description: "Test cases for your agents and flows, run with recorded model answers, that stop a worse version."
canonical_url: "https://docs.oz0.ai/improve/evals"
---
# Evals and quality gates

> Test cases for your agents and flows, run with recorded model answers, that stop a worse version.

For: Plugin authors testing agents and flows.

Evals are test cases in your plugin: an input for an agent or a flow, and what a good answer looks like. `oz0 plugin test` runs them on a dev server and a node of its own, and model answers can be recorded once and replayed after, so tests in CI are fast, free and repeatable.

```yaml [evals/greetings.yaml]
agent: greeter            # or flow: <name>
threshold: 0.8            # the share of cases that must pass; all of them by default
cases:
  - name: greets by name
    input: Please greet Ada
    expect:
      contains: ["Hello, Ada"]
      cel: "calls <= 3 && cost_usd < 0.02"
  - name: refuses an empty name
    input: { name: "" }
    expect:
      status: failed
      error_contains: InvalidInput
```

List eval files or folders in the manifest with `evals: [evals/]`. `plugin lint` checks them, CEL included.

## In this guide

- [Case expectations](https://docs.oz0.ai/improve/evals/cases): Specify the answer, tool choices, constraints and approval decisions a case must satisfy.
- [Record and replay answers](https://docs.oz0.ai/improve/evals/recordings): Record model answers and flow outputs, then reuse them in repeatable tests.
- [Check policy paths](https://docs.oz0.ai/improve/evals/coverage): Check which policy branches your cases exercise.
- [Gate a plugin update](https://docs.oz0.ai/improve/evals/quality-gates): A candidate plugin version must pass its evals before the quality gate admits it.
- [Track quality and regressions](https://docs.oz0.ai/improve/evals/regressions): Schedule evals, investigate regressions and keep a failed job as a reproducible case.

## Run them

```bash [Terminal]
oz0 plugin test ./greeter            # every eval of the plugin
oz0 plugin test ./greeter --eval greetings
```

```text
PASS  greetings / greets by name (replayed, 2 calls, $0.0011, 0.5 s)
  FAIL  greetings / says goodbye (replayed, 2 calls, $0.0011, 0.3 s): the answer does not contain "Goodbye"
greetings (greeter): 2 of 3 passed (67%), threshold 60%: ok; $0.0032, 1.2 s
```

The command starts a dev server and a node, installs the plugin from its working tree, and runs the cases one at a time as jobs. It exits non-zero when an eval is below its threshold, so it can gate a pull request. `--json` prints the report for machines. It needs what `oz0 dev` needs: the node binary next to the CLI and a runtime bundle.

The dev server's tenant runs with the [taint-default rule](https://docs.oz0.ai/govern/taint-and-labels/rules#the-taint-default-rule) turned off: a case whose agent reads an open-world result and then calls another outbound tool would otherwise need an answer in every such case. An eval that [ships a tool policy](https://docs.oz0.ai/improve/evals/coverage#policy-path-coverage) runs under that policy as written, with the rule on unless it says `"taint_default": "off"`, and its cases answer what the rule asks with `expect.approvals`.

## Previous section links

- <span id="a-model-changed-behind-its-name">

a-model-changed-behind-its-name

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#a-model-changed-behind-its-name).
- <span id="approvals-in-a-case">

approvals-in-a-case

</span>

: [open this section](https://docs.oz0.ai/improve/evals/cases#approvals-in-a-case).
- <span id="constraints">

constraints

</span>

: [open this section](https://docs.oz0.ai/improve/evals/cases#constraints).
- <span id="drift-which-of-the-model-the-judge-or-the-agent-changed">

drift-which-of-the-model-the-judge-or-the-agent-changed

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#drift-which-of-the-model-the-judge-or-the-agent-changed).
- <span id="each-case-runs-three-times">

each-case-runs-three-times

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#each-case-runs-three-times).
- <span id="how-a-regression-is-decided">

how-a-regression-is-decided

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#how-a-regression-is-decided).
- <span id="keep-a-job-as-an-eval-case">

keep-a-job-as-an-eval-case

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#keep-a-job-as-an-eval-case).
- <span id="one-node-tests-it">

one-node-tests-it

</span>

: [open this section](https://docs.oz0.ai/improve/evals/quality-gates#one-node-tests-it).
- <span id="policy-path-coverage">

policy-path-coverage

</span>

: [open this section](https://docs.oz0.ai/improve/evals/coverage#policy-path-coverage).
- <span id="quality-gates">

quality-gates

</span>

: [open this section](https://docs.oz0.ai/improve/evals/quality-gates#quality-gates).
- <span id="quality-over-time">

quality-over-time

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#quality-over-time).
- <span id="record-once-replay-after">

record-once-replay-after

</span>

: [open this section](https://docs.oz0.ai/improve/evals/recordings#record-once-replay-after).
- <span id="recorded-step-outputs-of-flows">

recorded-step-outputs-of-flows

</span>

: [open this section](https://docs.oz0.ai/improve/evals/recordings#recorded-step-outputs-of-flows).
- <span id="scheduled-evals">

scheduled-evals

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#scheduled-evals).
- <span id="skills-no-job-reads">

skills-no-job-reads

</span>

: [open this section](https://docs.oz0.ai/improve/evals/regressions#skills-no-job-reads).
- <span id="the-evals-run-there">

the-evals-run-there

</span>

: [open this section](https://docs.oz0.ai/improve/evals/quality-gates#the-evals-run-there).
- <span id="the-verdict">

the-verdict

</span>

: [open this section](https://docs.oz0.ai/improve/evals/quality-gates#the-verdict).
- <span id="the-version-waits-as-the-plugins-candidate">

the-version-waits-as-the-plugins-candidate

</span>

: [open this section](https://docs.oz0.ai/improve/evals/quality-gates#the-version-waits-as-the-plugins-candidate).
- <span id="what-a-case-can-expect">

what-a-case-can-expect

</span>

: [open this section](https://docs.oz0.ai/improve/evals/cases#what-a-case-can-expect).
- <span id="which-tools-and-skills-it-chose">

which-tools-and-skills-it-chose

</span>

: [open this section](https://docs.oz0.ai/improve/evals/cases#which-tools-and-skills-it-chose).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.