oz0 compare start
For: Operators and developers looking up a command.
Compare AGENT as installed with one change, on its eval cases (--cases evals, graded by each case's expect) or its recent inputs (--cases recent, passed when the job completes; a data subject's job is never a case). The variant is:
revision=candidate:<id> a private candidate of its plugin (its owner or an admin) revision=gate the candidate its plugin's quality gate tests now revision=rollout its plugin's rollout's candidate revision=workspace:<revision> a saved revision of its plugin's workspace folder revision=version:<version> a version of its plugin the tenant had draft=<folder> your studio draft of its plugin withhold-skill=<skill> the installed agent without one of its skills ([<plugin>/]<skill>) inline-delegate=<agent> the installed agent with one of its plugin's delegates folded in: the delegate's instructions as a skill named after it, its tools beside the agent's, and no oz.delegate to it (refused when the two hold the sides of a duty pair)
A draft, a workspace revision and a past version run as a private candidate of the plugin on the nodes --nodes names, for the comparison's life, and those nodes take no other work of the tenant meanwhile. Agents on any runtime may be compared; a best-of may not, since its candidates are jobs of their own.
Usage
oz0 compare start AGENT --variant revision=<ref>|draft=<folder>|withhold-skill=<skill>|inline-delegate=<agent> --budget USD [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--budget | float | what the comparison may spend, in dollars (required) | |
--cases | string | evals | evals (the agent's eval cases) or recent (inputs of its jobs that completed in the last 7 days) |
--confidence | float | 0.95 | the intervals' confidence |
--cost-per-run | float | the estimate's cost of one run, when the agent has no recent jobs to estimate from | |
--dry-run | bool | only check the change and show the plan | |
--json | bool | print JSON instead of a table | |
--margin | float | 5 | same means not worse than the baseline by more than this many points of pass rate |
--max-cases | int32 | 10, at most 50 | with --cases recent, at most this many inputs |
--nodes | stringToString | [] | for a draft, a workspace revision or a past version: the labels of the nodes it runs on (k=v,...) |
--poll | duration | 2s | with --wait, how often to look |
--runs | int32 | 3, at most 10 | runs per arm and case |
--tenant | string | default | tenant |
--variant | string | the change: revision=<ref>, draft=<folder>, withhold-skill=<skill> or inline-delegate=<agent> (required) | |
--wait | bool | wait for the result and print it |
Inherited flags
| Flag | Type | Default | Description |
|---|---|---|---|
--context | string | $OZ0_CONTEXT or <config dir>/orchestrator-zero/context.json | CLI context file written by server init |
--log-level | string | info | log level: debug, info, warn or error |
See all oz0 commands and the global flags.