oz0 job keep-as-eval
For: Operators and developers looking up a command.
Make a job an eval case: its input, its status and its error become a case of evals/<eval>.yaml, with a cassette of its model calls so it replays offline in plugin test.
Before it is kept, an agent's case runs --runs times (3) against the live model, each spending at most --budget (twice the job's cost by default), and is kept as a regression case when it fails in most runs (at least two), flaky with its rate when in some, and not reproduced otherwise. A flow's case takes the job's recorded steps (run it in the test bench with its steps recorded).
The case goes into your workspace draft of the plugin (tenant admins write the workspace), or with --dir into a plugin folder on disk. A case is plugin source: every reader of the workspace reads it, and installed versions and Git exports carry it, so nothing runs before --confirm-shared says you know that.
A job about a data subject is kept only by a tenant admin with --confirm-subject, after you looked at what the case holds (--preview), and only into the workspace, where erasing the subject removes it: such a case is never installed or exported. A chat's turn is never kept.
--redact replaces the value at a JSON pointer of the input (and that value wherever else the case holds it), --redact-value-file the values in a file (one a line, - for standard input) wherever they appear, before anything runs or is kept; the values never go on the command line, where every user of the machine and shell history read them. The tenant's secrets are redacted too.
A flow's recorded steps are redacted the same way and shown in the preview. A cassette whose recorded answers hold a redacted value or a secret is left out, and the output says so. Operators with the content permission may keep other jobs.
Usage
oz0 job keep-as-eval JOB --eval NAME [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--budget | float | twice the job's cost, $0.05 to $5 | what each run may spend, in US dollars |
--case | string | kept-<the job's first 12 characters> | the case's name |
--confirm-shared | bool | you know the case becomes plugin source that every reader of the workspace reads (required to keep one) | |
--confirm-subject | bool | keep a data subject's job: you looked at what the case holds, and it is erased with the subject | |
--dir | string | write the case into this plugin folder on disk instead of your workspace draft | |
--eval | string | the eval to keep the case in: evals/<eval>.yaml (required) | |
--expect-file | string | the job must complete | a YAML or JSON file with the case's expect, as eval files write it |
--preview | bool | only show what the case would hold; nothing runs and nothing is kept | |
--redact | strings | a JSON pointer into the input whose value is replaced by [redacted], such as /customer/email; repeatable | |
--redact-value-file | string | a file of values replaced by [redacted] wherever the case holds them, one a line (- for standard input), so they never go on the command line | |
--runs | int32 | 3 | how many live runs reproduce it, 1 to 5 |
--tenant | string | default | tenant |
Inherited flags
| Flag | Type | Default | Description |
|---|---|---|---|
--context | string | $OZ0_CONTEXT or <config dir>/orchestrator-zero/context.json | CLI context file written by server init |
--log-level | string | info | log level: debug, info, warn or error |
See all oz0 commands and the global flags.