Track quality and regressions

For: Plugin authors testing agents and flows.

Schedule evals, investigate regressions and keep a failed job as a reproducible case.

Quality over time

The gate checks a version once, when it is installed. After that the model behind a pinned name can change, a prompt can drift, and a failure a customer saw can come back. Three things watch for that.

Scheduled evals

An eval routine is a routine whose target is a plugin's evals instead of an agent: on its schedule, the edge runs the evals against the live models the agents use, through the gateway, on the fleet's nodes, billed to the tenant.

name: support-nightly
owner_team: support
eval:
  plugin: support          # the installed plugin whose evals run
  agent: helper            # optional: only this agent's or flow's evals
  evals: [refunds, smoke]  # optional: only these evals
  version: 1.4.0           # optional: wait while the plugin is at another version
  regression:              # optional; these are the defaults
    max_drop_points: 5     # the pass rate may not drop by more than 5 points
    confidence: 0.95
    max_cost_rise: 0.25    # cost per successful case may not rise by more than 25%
    max_latency_rise: 0.5  # seconds per successful case may not rise by more than 50%
    runs: 3                # each case runs three times (1 to 5)
    max_consistency_drop: 0.1  # how far the runs' agreement may fall before it counts
triggers:
  - name: nightly
    schedule: { cron: "0 3 * * *", timezone: Europe/Stockholm }
limits:
  budget_usd: 2            # what one run may spend: required
  daily_budget_usd: 5      # what its runs may spend per UTC day: required

oz0 routine save -f evals-routine.yaml
oz0 routine run support-nightly        # run it now
oz0 eval runs --routine support-nightly

• Schedules only. An eval routine takes schedule triggers and "run now", never a webhook or an event: what it sends to a model is your plugin's own cases. An interval (every: 6h) runs once at once, which is the first run to compare with; a cron runs at its next time.
• Budgets are required. A run stops starting cases once it spent budget_usd and ends as budget. A run may spend no more than what is left of daily_budget_usd today, and a confirmation (below) is a second run under the same cap; once the day's budget is spent, the routine waits until midnight UTC. Its spend shows with the routine's, and in the cost log under its case jobs (evs.<routine>.…).
• Runs the fleet answers. Each case is a job on the installed version's queue, marked as an eval's (oz0.replay: eval), so KPIs, the scorecard and rollout gates leave it out. A flow's case with a recording replays it, as at the gate. A run waits, and says so, while no node runs the plugin, while a kill switch stops its agent, or while the plugin is not at version; it is tried again every 5 minutes until it runs, and a "run now" or a canary's run asked for meanwhile waits with it.
  Three runs in a row that could not run at all pause the routine.
• The series keeps, per run and eval, the version, the cases run and passed, what each cost and took, and the model names the provider reported for the run's calls. The agent's page shows it under Evals, and oz0 eval runs lists it. It never keeps what a case answered.

To measure what a change to an agent saves on these same cases (another model, lazy tools, fewer tools), run a comparison: oz0 compare-savings start <agent> --cases evals. It runs each case on both arms as shadow runs, side-effect free, and states a saving only at an equal pass rate.

How a regression is decided

A run is compared with its baseline case by case. The baseline is the eval's first run, and moves only when a run improved on it, when the eval's cases changed (a run that shares fewer than three cases with it starts the series again), or when an admin accepts a run (below).

A run that was only noise or borderline never moves it, so a drop each run shows too weakly to decide is never made the bar. One flipped case proves little (one rerun of a flaky case looks just like it), so the rule resamples the pairs of cases 2,000 times:

• The target: a regression when, at the rule's confidence, the pass rate dropped by more than max_drop_points; borderline when it dropped that much but the rule is not confident (more runs would tell); improved when it rose as much.
• The guardrails: cost and seconds per successful case (failed cases are paid for too), against max_cost_rise and max_latency_rise.
• The diagnostics: the cases that flipped each way, and the models reported then and now.

A run that regresses or breaks a guardrail is run again at once, against the same baseline. Only when the second run agrees is a finding raised: eval support-nightly/refunds of helper regressed, confirmed by a second run: 3 case(s) that passed now fail (refund twice, partial refund, wrong currency); the provider reported anthropic/claude-sonnet-5-5-20261009 (was anthropic/claude-sonnet-5-5).

It shows on the agent's page, in oz0 eval findings and as a row of oz0 doctor, and the audit log records it; a notification rule for eval.regression sends it by email, webhook or chat, with its facts only (model.changed does the same for a canary's change). It resolves itself when a later run against the same baseline is noise or improved (a borderline run says nothing either way), or an admin resolves it:

oz0 eval findings --state open
oz0 eval resolve 0199d1… --reason "fixed the refund prompt"
oz0 eval resolve 0199d1… --reason "the new model words it differently, which is fine" --accept   # the latest run is the new baseline

With fewer than three cases in common with the baseline, nothing is decided (too_few), and a run with at least three cases of its own becomes the new baseline. With few cases the rule is cautious: on ten cases, two that start failing (20 points) stay borderline, run after run, against the same baseline; three (30 points) are a regression.

Each case runs three times

One run of a case proves little: an agent that answers differently each time looks, in one run, just like one that got worse. So each case of an eval routine runs runs times (three by default), one run of the eval after another, each run a job of its own (the jobs after the first end in .r2, .r3).

The case passes when most of its runs pass, and its cost and seconds are the runs' means, so a routine with runs: 1 decides exactly as before; a run of three costs three times as much, under the same budget_usd.

How often a case's runs agree with each other is its consistency: 1 when all agree, 1/3 when one of three differs. The run's consistency is the mean over its cases, and oz0 eval runs shows it beside the pass rate. Against the baseline it is decided as the pass rate is, pair by pair with 2,000 resamples: it fell when, at the rule's confidence, it dropped by more than max_consistency_drop. A fall is a drift finding (below) even when the pass rate held.

Drift: which of the model, the judge or the agent changed

Three things change an agent's results and look alike in a score: the model behind a pinned name, the judge that scores it, and the agent itself. When a regression is confirmed, or consistency fell, the finding names which one, with its numbers, as a drift finding of the agent:

helper: the model behind smart changed on 2026-10-03; consistency 0.81 -> 0.64; the judge is unchanged; a sampled review rate of 12% -> 19% is suggested

The cause is decided in this order:

1. model: a model canary found a change, after the baseline ran, in a model the agent uses (as provider/model, a fallback, a cascade step, a judge, or through an alias the tenant maps to it), or the provider reported other model names than in the baseline;
2. judge: the agent's judge agrees with fewer of its anchors than it did;
3. consistency: the agent's runs agree less with each other;
4. version: the plugin's version changed since the baseline;
5. unknown: none of these explains it.

The finding names the numbers of all three detectors whatever the cause, and links the evidence (the runs, the canary's finding, the anchor runs) in its detail. For an agent whose jobs are sampled for judging, it suggests a sampled review rate that keeps the share of unjudged failed jobs where it was: 1 - (1 - rate) x (1 - passed before) / (1 - passed now).

It only suggests; nothing changes the rate but you. The regression's own finding carries the same cause.

One drift finding stays open per agent and cause; a later cause opens another. A drift only the eval found resolves itself with the regression, or, when it came from a fall in consistency alone, once a later run agrees with itself again; one a canary or the anchors found, alone or beside the eval, stays until you resolve it (oz0 eval resolve ID --reason ...), and keeps the evidence of each (the canary's finding, the eval runs, the anchor runs).

oz0 eval findings --kind drift lists them, oz0 doctor counts them, and a notification rule for quality.drift sends them.

A model changed behind its name

A model canary watches a pinned model on a provider and, when the answers it watches change, runs the eval routines of the plugins whose agents use that model at once, and opens a drift finding with the cause model for each agent that uses it. The evals then say whether quality moved.

Skills no job reads

Beside the evals' own findings, oz0 eval findings lists skill_unused: a skill of an installed plugin that no job of the agents listing it read over 30 days, leading with any skill it pushes past an agent's start-up budget. A finding is open, resolved or kept: keep one with oz0 eval resolve <id> --keep --reason ... when the skill stays on purpose, and no finding is raised about it until it changes. See Skills no job reads.

Keep a job as an eval case

A job that went wrong becomes a case of its agent's or flow's evals: its input, its status and its error become a case, with a cassette of its model calls so it replays offline in plugin test. The preview shows its tool calls and answer to you; the case does not hold them.

oz0 job keep-as-eval job-0199d1… --eval kept --preview            # what the case would hold; nothing runs
oz0 job keep-as-eval job-0199d1… --eval kept --case "refund twice" \
  --redact /customer/email --expect-file expect.yaml --confirm-shared

Run 1 (kp.4f2c….1): failed: the job failed: ModelRejected: …, $0.0021, 1.4 s
Run 2 (kp.4f2c….2): failed: …
Run 3 (kp.4f2c….3): failed: …
Kept job-0199d1… as case "refund twice" of eval kept: regression (failed 3 of 3 runs)

• It must reproduce first. One failure proves little, so an agent's case runs three times against the live model (--runs 1 to 5), each run spending at most --budget (twice what the job cost, $0.05 to $5, by default) and graded by the case's expect (yours, or "the job completes").
  Failing in most runs, and at least twice, makes it a regression case; failing in some makes it flaky, with its rate; not failing makes it not reproduced, which you may still keep. The cassette is the first failing run's model calls, each line tagged with the kept job's ID.
• Flows are not run again, since a flow's live run calls its tools: their case takes the job's recorded steps, so run the flow in the test bench with its steps recorded first.
• Where it goes: into your draft of the plugin in the workspace (evals/<eval>.yaml, the cassette under evals/cassettes/; tenant admins write the workspace), or into a plugin folder on disk with --dir. The case's kept: block records the job, the label and rate, and the job's status and error. On the job's page, Keep as eval case under More does the same.
• A case is plugin source. Every reader of the tenant's workspace reads it, without the content permission the job's trace needs, and every installed version and Git export of the plugin carries it. So nothing runs before you say you know that (--confirm-shared, a checkbox on the page), and the case holds only what you looked at: the input, after your redactions, and the job's status and error.
  Two files beside it hold content too. A flow's recorded steps hold every step's input, output and error: they pass the same redactions and the preview shows them as they are written. An agent's cassette holds the reproductions' model answers, which a replay answers with as they are, so they are not rewritten: when one holds a value you redacted or one of the tenant's secrets, the cassette is left out and the result says so (run that case with --live, or record its cassette yourself).
  Both land in your own draft, which only you read until you save a revision. A chat's turn is never kept: what was said belongs to the chat and those it is shared with.
• Label policies. A case is made only from a job that holds no compartment and the lowest level: the workspace has no walls, so a case of a job that holds a compartment would show what it said to people the tenant's label policies wall it off from. Such a job is refused with what it holds; write the case by hand from what may be shared. A job whose labels cannot be read is refused too.
• Personal data. A job about a data subject is kept only by a tenant admin, with --confirm-subject, after looking at what the case holds (--preview), and only into the workspace. There it stays: installing or exporting a folder that holds such a case is refused, so it never reaches an installed version, the fleet's nodes, the quality gate, an eval routine or a Git remote.
  Erasing the subject removes the case from its eval file, in your drafts and every revision, and replaces its cassette and recorded steps with a note; the folder installs again without it. The case is found by the job it names, and in its own folder also by its eval and name when someone deleted its kept: block; files that name no kept job, such as another folder's recording at the same path, are left alone.
  --redact and --redact-value-file replace what the case need not hold before anything runs or is kept: --redact replaces the value at a JSON pointer of the input, and that value (a string or number of at least four characters) wherever else the case holds it; --redact-value-file replaces each value its file holds, one a line (- reads standard input), wherever it appears.
  Such values come from a file rather than the command line, where every user of the machine and shell history would read them. The tenant's secrets are replaced too. Whether a case may be kept from such a job at all is your organisation's call, as is sending it to a live model again.
• Retention. A kept case follows its job's data: when the tenant's retention removes the job's data, it removes the case from the workspace the same way, and installing a version that still holds it is refused. A legal hold keeps both. Erasure and retention do not reach copies a person made outside the workspace (a download, a paste into the test bench, a folder on disk), nor versions of a case without a subject that were installed or exported before its job's data expired.
• A regression case fails until the agent is fixed, in plugin test and at the next version's gate, which is the point. Put flaky cases in an eval of their own with a lower threshold.

Back to Evals and quality gates.