Gate a plugin update
For: Plugin authors testing agents and flows.
A candidate plugin version must pass its evals before the quality gate admits it.
Quality gates
When you install a version of a plugin that has evals, the edge runs them before the fleet gets the version, against your tenant's real models and keys:
The version waits as the plugin's candidate
The fleet keeps the version it has. A plugin's first version waits too: its agents run once it passes.
One node tests it
The edge picks an online node of the tenant that matches the plugin's selector and the labels its agents need, and prefers nodes labelled oz0.gate=true. That node runs the new version instead of the installed one, on queues of its own: no fleet job reaches it, and no eval case reaches the fleet.
While no such node is online, the gate waits for one. On shared pools the test node also leaves the pools the plugin shares with the tenant's other plugins, so the edge prefers a node whose leaving keeps another online node on each of them. When the only node it can test on is the last one of a pool, as on a one-node dev server, the pool's other plugins wait until the gate ends, and the gate's reason in oz0 plugin show names them; another node with the pool's labels keeps them running.
The evals run there
Every case runs as a job through the gateway, as in production: real models, real tools, real cost. Recorded model answers are not used; a flow's case with recorded step outputs replays them, and calls nothing.
The verdict
The version passes when every eval reaches its threshold and passes at least the share of cases that the installed version passed at its own gate. It then becomes the installed version and rolls out, and its results become the bar for the next version. Otherwise it is stopped: the fleet keeps the installed version and the test node goes back to it.
oz0 plugin install https://github.com/acme/team --ref v0.2.0
Testing team 0.2.0 (commit ff4d7eae2c6d…) in tenant default before the fleet gets it: its evals run on a test node.
Until it passes, the fleet keeps team 0.1.0.
Follow with: oz0 plugin list (--no-gate installs without the evals)
plugin list and the Plugins page of the web UI show the gate: waiting, testing on a node, or how it ended and why, with each eval's result. A version that does worse reads like this:
Gate of team 0.2.0: stopped: eval greetings: 0 of 1 cases passed (0%), below its threshold of 100%
The audit log records every verdict. plugin install --no-gate installs a version at once, without running its evals.
Good to know:
- A dedicated test node keeps gates away from the nodes that serve jobs: give it the label
oz0.gate=truewith its join token (token create --label oz0.gate=true) or when you accept it. While a gate runs, its node serves no fleet jobs of that plugin, so a tenant with a single node waits for the gate before those agents run again. - A version that cannot start on the test node, or does not start within 10 minutes, is stopped. A case may run for 10 minutes.
- Children stay with the candidate: a flow's agent steps and an agent's delegations to agents of the same plugin run the new version on the test node; agents of other plugins answer from the fleet, in their installed versions.
- Installing again during a gate: the version under test again (a
plugin updatewhile it runs) lets its gate go on; a newer version replaces it and its gate starts over; the installed version again, or--no-gate, cancels it. A version that was stopped is tested again when you install it again. - Approvals are denied. A case's approvals are put to no person on your tenant: each one is denied at once, and a case that approves or edits one is skipped (approvals in a case). An eval routine does the same.
oz0 devandplugin testinstall your working tree without a gate.- Servers share the work: any server can drive a gate. If it stops, another one runs the gate again from the start within 30 seconds.
After the gate, the new version rolls out to canary nodes first, next to the old version on real jobs, through health and quality gates of its own; a quiet tenant's rollout replays recent real inputs as shadow runs.