Measured savings

For: Plugin authors measuring cost changes.

Measure cost changes on your own eval cases. A comparison runs the installed baseline and one variant several times on the same cases. It reports a saving only when the variant maintains the pass rate within the allowed margin and lowers billed cost per successful run, with an interval to show the uncertainty.

Fewer tokens alone do not establish a saving: a cheaper model can fail more often, and lazy tool loading can add calls. To compare a changed definition on quality, use comparisons of a changed agent.

What a comparison runs

• The agent as installed when you start: a comparison whose plugin moves to another version while it runs stops and says so. Agents on pydantic-ai only, and not a best_of: agent, whose candidates are jobs of their own.
• The variant, one change to the agent's settings:
  • --variant-model runs it on another model alone: an alias or <provider>/<model>, resolved as the agent's own model is, so the tenant's data rules and data: local still hold. "A cascade against the large model alone" is the agent as defined against --variant-model set to its large model.
  • --variant-tool-loading lazy or eager switches lazy tool loading on or off for the run.
  • --variant-tools offers only some of the agent's plugin functions (its built-ins stay).
  • --variant-cache off sends the variant's model calls as the agent wrote them, without the gateway's prompt-cache management, so the comparison measures what the cache does for this agent on these cases. Its arms run side by side, so a case's two runs share no cache unless their requests are alike. A tenant whose prompt cache is off cannot compare it: both arms would run without it, so such a comparison is refused.
  • --variant-offload off, on or a threshold in tokens sets how the variant's large tool results reach the model: off inline and whole, on as a stub with oz.read_result above the job's threshold (or the platform's default where the job has none), a number above that many tokens.
    Either arm stays within the tenant's ceiling, as an agent's own setting does, so a comparison the ceiling would change is refused before it runs: under a ceiling, an arm with off or a number above the ceiling (oz0 tenant offload <tenant> shows it), and in any tenant two arms that would run at the same threshold.
    The agent needs a plugin function, whose results are the only ones offloaded. --baseline-offload sets the baseline's the same way. It is measured on read-only results only: a shadow run answers every other function from a stub sentence, so a write's result is never large on either arm. The arms' requests differ in their tools too: an arm with offloading on offers oz.read_result in every request.
    An offloaded result is a file of the run's own job, kept and erased as the run's job data.
• The cases: --cases evals runs the agent's eval cases (the first 50), each graded by its own expect; --cases recent replays the inputs of the agent's top-level jobs that completed in the last seven days (--max-cases, 10 by default), checked against its input schema, and a run passes when it completes, which means the agent's outcome check and output contract held. A job started for a data subject is never a case, nor a chat's turn or a routine's fire.
• Runs per arm and case (--runs, 3 by default, at most 10). The two arms of a case run side by side, and which one starts first alternates.

Nothing acts on the world

Every run is a shadow run, as a rollout's quality gate makes them: a function the tenant's policy does not list as read-only answers from a stub sentence instead of running, every approval is denied at once without asking anyone, a hook that cannot be asked fails closed, and agents the run delegates to are shadow runs too.

A run that cannot replay its input this way counts for nothing, and the result lists it with the reason. So an agent whose answer depends on what a write returns is measured on the stub's sentence, on both arms alike.

Run jobs are named cmp.<comparison>.<b|v>.<case>.<hash>.<n>, and oz0 job get on one says which comparison and arm it belongs to and what it ran with. Nobody can start a job under cmp. or shadow- by hand.

Plan it first

A comparison spends the tenant's money: cases × runs × two arms. --dry-run shows the plan without starting anything:

oz0 compare-savings start clerk --variant-tool-loading lazy --runs 3 --budget 2 --dry-run

Plan:                4 cases x 3 runs x 2 arms = 24 runs
Estimate:            $0.3000 ($0.0100 a run, the median billed cost of 12 recent jobs of clerk; x1.25 for the variant), budget $2.00
Power:               2% to show an equal pass rate (baseline 92%, 12 recent jobs of clerk); 475 paired runs give 80%

• The estimate is the agent's median billed cost per top-level job over the last 30 days (or --cost-per-run when it has none), times 1.25 for the variant, times every run. A start whose estimate does not fit --budget, or whose budget is more than is left of the tenant's monthly budget, is refused before anything runs.
  Each run gets its share of what is left as its own budget, in proportion to what a run of its arm is expected to cost. A pair starts only when what is left covers it; when it does not, or a run's budget cuts it short, the comparison stops and is judged on what ran (budget).
  A run cut short by money counts for neither arm, never as a failure.
• The power is the chance of showing an equal pass rate when the two arms really are equal, assuming they fail independently at the baseline's recent pass rate, and how many runs per arm 80% needs. Even an agent that passes every case needs enough runs to show it: about 73 per arm for a 5-point margin at 95%, 16 for a 20-point margin.
  One that passes 90% needs several hundred for 5 points. A comparison with too few runs ends with "no measured saving: too few runs to show an equal pass rate", and says how many would, which is the honest answer.
• At most three comparisons per tenant wait or run at once. The tenant's operators and admins may start one, on recent inputs only with the content permission (it sends what people's jobs were given to a model again), and not while the tenant's walls apply to them, since the recent jobs may be any compartment's; everyone in the tenant may read and export them.
  Each run of a recent job's input starts holding that job's compartments and level, so what a wall closes on the job it closes on the run; a recent job whose labels cannot be read is skipped.

The result

oz0 compare-savings start clerk --variant-tool-loading lazy --runs 3 --budget 2 --wait
oz0 compare-savings show <ID>

For each arm: its runs and passes, the pass rate with its Wilson interval, the billed cost and the tokens (input, output, cache reads and cache writes apart) per successful run with their bootstrap intervals, and the median and interquartile range of a successful run's cost. Per successful run means the arm's total over its passes: failed runs were paid for too.

Then the arms are compared in pairs, by case: the cases are resampled with replacement 2,000 times, with a generator seeded from the comparison's ID, so the same runs always give the same result. That gives intervals, at the comparison's confidence (95% by default), for the change in pass rate and in cost and tokens per successful run.

• An equal pass rate means non-inferiority: the lower end of the pass rate change's interval is above minus the margin (--margin, 5 points by default). That interval is the wider of the bootstrap's and Newcombe's score interval, which counts runs: when every run agrees the bootstrap alone says 0, 0, however few runs there were.
• A measured saving is stated only when the pass rate is equal and the upper end of the cost per successful run change's interval is below zero. It is given with its interval, never as a number alone.
• Otherwise the result says no measured saving and why: the pass rate is not shown equal, too few runs to show it, the cost interval includes zero, the variant costs more, too few cases ran on both arms (three at least), or the budget ran out first. Fewer tokens without a lower bill are reported as such, and are not a saving.

The console shows comparisons under Cost → Measured savings, with the per-arm table, the paired changes and the verdict in words, and a form that plans before it starts.

Export it

oz0 compare-savings export <ID> --format csv -o saving.csv
oz0 compare-savings export <ID> --format json -o saving.json

Every file carries the method, the margin, the confidence, the resamples and the seed with the verdict and every run. In CSV they are lines starting with # before the table of runs (most CSV readers skip them with a comment option). The evidence bundle carries the tenant's finished comparisons of measured savings as comparisons.jsonl (comparisons of a changed agent are in a file of their own).

Who started a comparison is in the audit log (comparison.start, comparison.finish) and in no export. Every figure is per agent and variant, never per person.

If you quote a saving anywhere, quote an export like this one with its method.

What a comparison does not prove

• Other cases. It measures the cases it ran. Eval cases are what the plugin's author chose; recent inputs are last week's.
• Other times. Providers change models, prices and caching. A comparison is a measurement on the day it ran.
• Another tenant's work, or the same agent with other tools, policies or hooks.
• Writes. A stubbed write does not show what the real one would have returned, so a change that alters what the agent does after a write is not measured.
• A changed trajectory beyond the cases. A variant can take more steps, retry more or ask for help differently on inputs the comparison did not see.

Where the money shows

A comparison spends on the tenant's keys and counts toward its budget. Its runs appear in usage under their cmp. job IDs, summed apart with shadow runs and eval cases, and are left out of the Overview's cost, the scorecard, the KPIs and the spend forecast.