oz0 plugin test
For: Operators and developers looking up a command.
Start a dev server and a node with the plugin from its working tree, run every case of its evals as a job, one at a time, and grade the answers. A case with a recording (evals/cassettes/<eval>/<case>.jsonl) is answered from it, without calling a model; others call the real API with ANTHROPIC_API_KEY or OPENAI_API_KEY from the environment. --record calls the real API for every case and keeps the answers.
A flow's case with a recording of its step outputs (evals/steps/<eval>/<case>.json) replays it instead of calling the flow's tools and agents; --record-steps runs the flows' cases and keeps those recordings. Exits non-zero when an eval passes fewer cases than its threshold. The plugins it uses are installed first, as dev installs them: from --with folders, or from their uses entries' sources.
Once the test node has started the plugin's servers, it writes what they list into tools.listing.json: commit it, so plugin inspect and the hub show the tools and what their definitions cost, and it fails when an agent or the plugin is over its tool-size budget (limits.tool_tokens, tool_tokens), as plugin lint does. --report writes a report of every eval of exactly these files, which plugin inspect --test-report puts in the listing.
After the evals, a cost-amplification probe calls each function of the plugin's tools that declares it only reads and stays inside the tenant (readOnlyHint true, openWorldHint false) with a small input, a large one and the small one again, and writes each function's cost multiplier to tools.amplification.json when it changed, which plugin inspect and installs show (--probe all calls every function, outside services included; --probe off none).
A case answers the approvals its jobs wait for with its expect.approvals (approve, deny or edit); an approval it does not answer is denied at once and fails the case. An eval's policy: file (the JSON tenant tool-policy show --json prints) is the tenant's tool policy while its cases run.
--coverage measures policy-path coverage: which of the plugin's decision points (the rules of the policies its evals ship, approvals, flow steps and when: branches, loop rules, budgets, outcome checks, and the taint, origin and sequence rules) the cases reached, branch by branch.
It prints a table, writes the report to evals/coverage.json (which the studio shows) and to --coverage-out, and appends a Markdown summary to the file $GITHUB_STEP_SUMMARY names, when it is set; --min-coverage fails the run below a percentage, naming what no case reached.
--drill then runs a failure drill on the same server and node, against the plugin's own fake (tests/drill.yaml starts it; a plugin with none is skipped): each write it names is made as a tool job, the test node's runtime is crashed while the job waits for its approval and approved once it is back, and crashed again once the fake made the write and before its answer arrived, so the runtime makes the call again under the same idempotency key; the fake's own count must show one write.
Each eval case that makes a write runs again with the runtime crashed after its step --drill-step, and must give the same answer. A write seen twice fails the run; a function that is not idempotent by its annotations is a warning that names it. --drill-only runs the drill without grading the evals.
Usage
oz0 plugin test [plugin-dir] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--cache-dir | string | where the test node keeps uv, Python and Node.js between runs | |
--cassettes | string | <plugin>/evals/cassettes | where the recordings live |
--coverage | bool | measure policy-path coverage: which of the plugin's decision points the cases reached | |
--coverage-out | string | also write the coverage report, as JSON, to this file (implies --coverage) | |
--drill | bool | after the evals, run the failure drill against the plugin's own fake (tests/drill.yaml): crash the test node's runtime around each write and check the fake saw it once | |
--drill-only | bool | run the drill without grading the evals (implies --drill) | |
--drill-step | int | 1 | the drill crashes the runtime after this many finished model or tool steps of an eval case's job |
--eval | string | run only this eval | |
--json | bool | print JSON instead of a table | |
--live | bool | call the real API for every case, ignoring recordings | |
--llm-anthropic-url | string | https://api.anthropic.com | Anthropic API address for live calls |
--llm-openai-url | string | https://api.openai.com | OpenAI API address for live calls |
--min-coverage | float | fail below this policy-path coverage, a percentage from 0 to 100 (implies --coverage) | |
--node-binary | string | next to this binary, else on PATH | orchestrator-zero-node to run |
--plugin-sandbox | string | the node's, as an installed plugin meets it | the test node's --plugin-sandbox: auto for a plugin that reaches something on this machine, such as a stand-in for its API on localhost, which the node's default refuses |
--probe | string | read-only | the cost-amplification probe after the evals: read-only (the functions that say readOnlyHint and not openWorldHint), all, or off; its results go to tools.amplification.json |
--record | bool | call the real API for every case and keep the answers as the cases' recordings | |
--record-steps | bool | run the flows' cases for real and keep each step's output as the cases' recordings (evals/steps/) | |
--report | string | also write a report of every eval, of exactly these files, for plugin inspect --test-report (an index's CI records it) | |
--runtime-bundle | string | $OZ0_RUNTIME_BUNDLE, else the newest in ./dist/runtime | runtime bundle directory |
--with | stringArray | a plugin the one under test uses, from this folder, installed first; repeatable |
Inherited flags
| Flag | Type | Default | Description |
|---|---|---|---|
--context | string | $OZ0_CONTEXT or <config dir>/orchestrator-zero/context.json | CLI context file written by server init |
--log-level | string | info | log level: debug, info, warn or error |
See all oz0 commands and the global flags.