oz0 compare-savings start
Plan a comparison of AGENT as defined against a variant and start it.
For: Operators and developers looking up a command.
Plan a comparison of AGENT as defined against a variant and start it. The plan says how many runs it makes, what they are expected to cost (the agent's median billed cost per job over 30 days, or --cost-per-run when it has none), and the power to show an equal pass rate. A start whose estimate does not fit --budget, or what is left of the tenant's monthly budget, is refused before anything runs.
Cases are the agent's eval cases (--cases evals, graded by each case's expect) or the inputs of its recent top-level jobs (--cases recent, passed when the job completes; a data subject's job is never a case).
Usage
oz0 compare-savings start AGENT --budget USD [--variant-model M | --variant-tool-loading lazy|eager | --variant-tools F,... | --variant-cache off | --variant-offload on|off|N] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--baseline-cache | string | off: the baseline's model calls go without the gateway's prompt-cache management | |
--baseline-model | string | as defined | the baseline runs on this model alone |
--baseline-offload | string | the baseline's large tool results: off (all inline), on, or a threshold in tokens | |
--baseline-tool-loading | string | as defined | the baseline's tool loading |
--baseline-tools | strings | all | the baseline offers only these plugin functions |
--budget | float | what the comparison may spend, in dollars (required) | |
--cases | string | evals | evals (the agent's eval cases) or recent (inputs of its jobs that completed in the last 7 days) |
--confidence | float | 0.95 | the intervals' confidence |
--cost-per-run | float | the estimate's cost of one run, when the agent has no recent jobs to estimate from | |
--dry-run | bool | only show the plan: the estimate and the power | |
--json | bool | print JSON instead of a table | |
--margin | float | 5 | the non-inferiority margin, in points of pass rate |
--max-cases | int32 | 10, at most 50 | with --cases recent, at most this many inputs |
--output (-o) | string | with --wait, write the result's JSON export to this file | |
--poll | duration | 2s | with --wait, how often to look |
--runs | int32 | 3, at most 10 | runs per arm and case |
--tenant | string | default | tenant |
--variant-cache | string | off: the variant's model calls go without the gateway's prompt-cache management | |
--variant-model | string | the variant runs on this model alone: an alias or <provider>/<model> | |
--variant-offload | string | the variant's large tool results: off (all inline), on, or a threshold in tokens; measured on read-only results | |
--variant-tool-loading | string | the variant loads its tools eagerly or lazily: eager or lazy | |
--variant-tools | strings | the variant offers only these of the agent's plugin functions | |
--wait | bool | wait for the result and print it |
Inherited flags
| Flag | Type | Default | Description |
|---|---|---|---|
--context | string | $OZ0_CONTEXT or <config dir>/orchestrator-zero/context.json | CLI context file written by server init |
--log-level | string | info | log level: debug, info, warn or error |
See all oz0 commands and the global flags.