Roll out a plugin

For: tenant administrators and plugin authors.

Release a new version through canaries and handle failed or interrupted rollouts.

New versions roll out to canaries first

When a plugin runs on two or more online nodes, its new version does not reach them all at once. It goes to a canary node first, then to the rest in steps, and each step must pass the same health and quality gates as a runtime rollout. A version with evals passes its quality gate first, then rolls out.

oz0 plugin update github-tools
oz0 plugin install https://github.com/acme/oz-github-tools --ref v1.5.0 --canary 2 --batch 5 --soak 10m

Rolling team 0.2.0 (4f1c2d9) out in tenant default, canaries first: rollout rollout-6b1e0c2a.
Until it is done, the other nodes keep team 0.1.0.
Follow with: oz0 rollout show rollout-6b1e0c2a (--all-at-once gives a version to every node at once)

Each step:

1. gives the new version to its nodes, which run it instead of the installed one, and waits until they report it ready, for at most --start-timeout (5 minutes);
2. sends those nodes their share of the plugin's new jobs: with one node of three on the new version, a third of them (the nodes of a step that is starting take no new jobs until they run the new version);
3. watches them for --soak (1 minute), and then judges the step.

The new version answers on task queues of its own, roll.<commit>.<pool> for each pool its agents and flows fall in, so each job runs on one version from its start to its end, and the quality gate can tell them apart. With shared pools (the default) that is one queue per node selector and label set the plugin's agents use, such as roll.4f1c2d9e8a7b.pool.~342aaaea, and only this plugin's jobs are routed there: the pool's other plugins go on as before.

The quality gate compares the jobs that failed on those queues with the plugin's jobs that failed on its usual pools over the same time; on a shared pool it tells the plugin's jobs from the others' by their agent or flow. Until the last step the rollout's nodes take none of the shared pools the plugin is a member of, so the old version's jobs never meet the new version; the pool's other plugins run on the other nodes meanwhile. A rollout keeps the queue names it started with until it ends, even if the servers switch --queue-pools meanwhile.

A child job from the same plugin stays on its parent's version. A node takes the new version's jobs once it runs the new version, and in the last step, when no node is left on the old version, the rollout's nodes also finish the jobs left on the usual queues.

When the last step passes, the new version becomes the installed version on every node. When a step fails, or you abort with rollout abort, the candidate is dropped and the canaries go back to the installed version. Either way, the nodes on the installed version finish the jobs that are still on the rollout's queues; rollout show says draining until none is left. Here a version that changed its agent's input contract failed its canary's jobs:

Rollout rollout-838939ab: tenant default, plugin team from 0.2.0 (25e707a) to 0.3.0 (dc6f7d2), rolled-back, draining
  canary  failed   node-030651b413c5: step 1: 54 of 54 jobs failed on 0.3.0 (100%), against 0% on 0.2.0
  step 2  pending  node-d81206129a00
  step 3  pending  node-e4c46863afd8
Reason: step 1: 54 of 54 jobs failed on 0.3.0 (100%), against 0% on 0.2.0

In make e2e-rollout, which runs jobs in four loops meanwhile, the rollout ended 28 seconds after plugin install, and only the jobs that reached the canary failed: 60 of 233.

• A tenant runs one rollout at a time, of a runtime or of a plugin. A plugin rollout asked for while another one runs waits for its turn, and its steps are planned when it starts.
• While a plugin has a rollout waiting or running, a new version of it is refused; wait for the rollout to end, or abort it.
• A first install, a plugin on one node, and --all-at-once go to every node at once.
• An agent the new version adds can be started once the rollout is done. Jobs check their input against the installed version's schema.
• A job whose plugin changed under it in a way its history cannot follow, such as a flow that lost a step, fails with the reason instead of waiting forever.

Shadow runs for a quiet tenant

The quality gate needs five jobs that ended on the new version. When a step's soak ends with fewer, the gate replays recent real inputs of the plugin's agents on the new version instead: it takes the inputs of up to --shadow-runs (5) jobs that ended in the last week (on any version but this rollout's new one), starts each again as a shadow run on the canary, and compares how they end with how they ended the first time.

In a shadow run, tools that are not read-only answer from a stub instead of running, nobody is asked to approve anything, and the agents it delegates to run as shadow runs too. So a shadow run never calls a real write tool. A tool on a remote MCP server (one with a url:) is never read-only to a shadow run, whatever its annotations say, so it always answers from a stub.

A hook is a call too, so a shadow run asks only the hooks whose functions are read-only. A hook it cannot ask fails closed: a call that a pre_tool_call or post_tool_call hook guards answers from a stub, even when the tool is read-only, so it neither runs with arguments the hook might refuse nor hands the model a result the hook might redact; and an agent that a job_start or job_end hook watches is not replayed at all, since its input or answer would go unchecked.

A hook the tenant's tool policy refuses, or sends to a person, behaves as in a real run: it is not called, and counts as its on_timeout.

The step waits up to five minutes for its shadow runs. It counts them together with the real jobs that did end on the new version, and fails when the share of them that failed is more than 10 points above what was expected: the share of the replayed inputs that failed the first time, and the old version's share for the real jobs. Four real jobs that all failed are not outweighed by five replays that pass:

  canary  passed   node-9a1c: health gate passed; 0 jobs ended on 0.1.2, too few for the quality gate; shadow runs passed:
                   4 recent inputs ran on 0.1.2 with side effects stubbed, 0 failed (0 had failed the first time)

The evidence of the first step's shadow runs serves the later steps. Shadow runs that do not end in time, or could not be replayed, count for nothing either way, and the step says why. Shadow runs call models, so they cost what those calls cost; their job IDs start with shadow-, and the audit log records them (rollout.shadow). Harness agents and flows are replayed too (see below). Not the fires of routines: a fire runs bound to its routine, and only a fire's outputs reach the edge.

Only inputs that fit the agent's input schema on the new version are replayed: after an input contract changed on purpose (a new major version, or --allow-breaking), older inputs do not fit by design, and the edge would refuse them now. The gate reads up to four recent jobs per shadow run it wants, leaves out those that do not fit, and the step says how many it left out:

  canary  passed   node-9a1c: health gate passed; 0 jobs ended on 0.2.1, too few for the quality gate; 4 recent inputs of
                   helper did not fit the input schema on 0.2.1, and were not replayed; shadow runs passed: 2 recent inputs
                   ran on 0.2.1 with side effects stubbed, 0 failed (0 had failed the first time)

A rollout makes at most 20 shadow runs per step (--shadow-runs up to 20), and a shadow run that nothing waits for any more, such as one of a rollout rolled back by its health gate, stops six minutes after it started at the latest. --shadow-runs 0 turns them off, and the quiet tenant passes on health alone, as before.

Flows and harness agents

A harness agent (such as one on the Claude Agent SDK) runs its own tools, so its shadow run sends every tool call of its session through the permission bridge that hooks and approvals use.

A call to one of the harness's read-only tools (for the Claude Agent SDK: Read, Glob, Grep, LS, NotebookRead, TodoWrite) runs; every other call, such as Bash or Write, is refused with the shadow stub as its reason, which the model reads as it reads a stubbed write.

A harness the platform does not know has no read-only tools, and a harness that cannot ask before its calls is not replayed. The agents a harness agent delegates to run as shadow runs too.

A flow is replayed with its steps' side effects taken from its first run. Tool steps that only read run; agent steps run as shadow runs; a tool step that does not only read, and an approval step, answer with what they did when the job first ran, which the gate reads from that job's history. So the flow's later steps and its output see what they saw the first time.

A recorded result goes through the job's post_tool_call hooks again before a later step reads it, as in the first run: a hook that redacts it redacts it again, and one that withholds it fails the step.

A step that must not run and has no recorded result, such as a write step the new version added, and a tool step guarded by a hook the shadow run cannot ask, end the shadow run as not replayed, and the step says why:

  canary  passed   node-3250d982: health gate passed; 0 jobs ended on 0.2.3, too few for the quality gate; shadow runs passed:
                   4 recent inputs ran on 0.2.3 with side effects stubbed, 0 failed (0 had failed the first time)

The gate counts these shadow runs as it counts agents'. Flows and harness agents get shadow runs too, with their writes stubbed.

The Plugins page shows each rollout's steps next to the plugin, with Roll back, or Drop for one that waits.

Quality signals

A version can complete every job and still answer worse. Once a step's health and failed-job gates pass, five quality signals compare the new version's jobs of the step with the installed version's, each as a share:

A signal fails the step when the new version's share is more than its threshold above the installed version's: 0.1, or 10 points, by default. Both sides count the plugin's own agents' and flows' real jobs (no replays, evals or shadow runs) that ended between the step's start and the end of its soak. In the last step no node runs the installed version any more, so when its side has too few jobs there, it is counted over the rollout so far.

A signal with fewer than 10 jobs on either side says too few and decides nothing, and so do signals whose approval figures come from fewer people than works-council mode's threshold: those say hidden, with no numbers.

A quality signal may only stop and roll back a version, exactly as a failed health gate does. It never passes a step the other gates fail, never shortens a soak, never promotes or keeps a version, and never changes the jobs it counts. After a rollback you can start the same version again: with a new rollout, or with --all-at-once.

Rollout rollout-ae164a54: tenant default, plugin team from 0.4.0 (f5ef830) to 0.5.0 (8fb4d81), rolled-back, draining
Quality signals: outcome 10 points, cascade 10 points, denials 10 points, sampled 10 points, reported 10 points, at least 10 jobs
  canary  failed   node-6ff7f12dac9c: step 1: outcome: 18 of 18 jobs with a check failed their outcome check's first round (100%) on 0.5.0, against 0 of 55 (0%) on 0.4.0: more than 10 points worse
            outcome: 18 of 18 jobs with a check failed their outcome check's first round (100%) on 0.5.0, against 0 of 55 (0%) on 0.4.0: more than 10 points worse
            cascade: 0 of 18 jobs went to a later model (0%) on 0.5.0, against 0 of 55 (0%) on 0.4.0
            denials: too few approvals people decided (0 on 0.5.0, 0 on 0.4.0; each needs 10)
            sampled: too few judged jobs (0 on 0.5.0, 0 on 0.4.0; each needs 10)
            reported: too few reported results (0 on 0.5.0, 0 on 0.4.0; each needs 10)
  step 2  pending  node-75a2bf16e364
  step 3  pending  node-f1c31d8d709a

Two things can make a step wait a little past its soak. The job summaries the signals count are written every 30 seconds, so a step waits for the summaries of the jobs that ended in it, at most 2 minutes. When the plugin's agents are sampled, a step also waits for the judgments of the new version's jobs that the sampling rule picks, at most 15 minutes; judgments still pending then count for nothing, and the step says how many. A step never waits for a person: an approval still waiting when the soak ends counts for nothing.

Shadow runs ask nobody, so they give no denials. When the signals cannot be counted at all (the database does not answer, say), the step tries again for 5 minutes and then goes on without them, as the failed-job gate does when it cannot count, and its record says so: a signal stops only a step it counted.

The thresholds are the plugin's: set them in its configuration, on the Configure tab or in the rollout part of oz0 plugin config. A threshold is a number of points from 0 to 1, or off, and min_jobs the floor, 1 to 1000:

rollout:
  quality:
    outcome: 0.05     # at most 5 points more first-round failures than the installed version
    cascade: 0.1
    denials: 0.1
    sampled: 0.1
    reported: off     # results come in days later than a soak
    min_jobs: 10

They are a governance setting: widening a threshold, turning a signal off or raising the floor loosens the gate, so the change needs a reason, and another admin approves it. For one rollout, plugin install, plugin update and rollout start take --quality-outcome, --quality-cascade, --quality-denials, --quality-sampled, --quality-reported and --quality-min-jobs, which may only make the gates stricter than the plugin's configuration; a looser value is refused, and a signal the flags leave out keeps the configuration's threshold, as the floor does. A runtime rollout counts every job of the tenant by its runtime version, with the defaults unless its flags tighten them.

rollout show lists each step's signals with their numbers, the audit log's rollout.step and rollout.rolled-back entries carry them (rollout.rolled-back names the signal), and oz0_rollouts_total counts a rollback a signal caused with the verdict rolled-back:<signal>, such as rolled-back:outcome.

In make e2e-rollout, a version whose outcome check passed only a second draft ran while jobs ran in four loops: none of its jobs failed, all 18 that ended in the step failed their first round against none of the installed version's 30, and it was rolled back 94 seconds after plugin install, with a soak of 45 seconds; its record names the criterion the first drafts missed (missed own 18: the agent's own rubric).