--- title: "Roll a runtime out in steps" description: "Start a canary rollout and inspect its health and quality verdicts before the remaining nodes move." canonical_url: "https://docs.oz0.ai/fleet/upgrades/runtime-rollouts" --- # Roll a runtime out in steps > Start a canary rollout and inspect its health and quality verdicts before the remaining nodes move. For: cluster operators. Start a canary rollout and inspect its health and quality verdicts before the remaining nodes move. ## Roll a runtime out in steps `node upgrade` moves the nodes you name at once. A rollout moves all of a tenant's nodes in steps, and stops by itself when the new version does worse: ```bash [Terminal] oz0 rollout start --runtime 0.5.0+1a2b3c4d5e6f # tenant default: a canary, then the rest in two steps oz0 rollout start --runtime 0.5.0+1a2b3c4d5e6f --tenant acme --canary 2 --batch 5 --soak 10m oz0 rollout show rollout-3f9c2a1b oz0 rollout list oz0 rollout abort rollout-3f9c2a1b # roll back by hand ``` In the console, Upgrade on the *Nodes* page starts one: choose a runtime the edge serves, the canary, the nodes per later step, the soak and the start timeout. The panel above the nodes follows each step to its verdict, and Roll back aborts it. A rollout plans every online node of the tenant that does not already run only the new version: a canary of one node (`--canary`), then the others in steps of half of them each (`--batch`). Each step: 1. tells its nodes to run the new version, and waits until they run it, for at most `--start-timeout` (5 minutes); 2. sends the new version its share of the tenant's new jobs: with one node of four upgraded, a quarter of them; 3. watches the upgraded nodes for `--soak` (1 minute), and then judges the step. A step passes two gates: - Health: every upgraded node stays online, keeps running the new version and stays [healthy](https://docs.oz0.ai/fleet/nodes#health-scores), all through the soak. A node that goes bad fails the step at once. - Quality: of the jobs on the new version that ended during the soak, the share that failed is at most 10 points worse than on the old version over the same time. The gate needs five such jobs.
When there are fewer, it replays the inputs of up to `--shadow-runs` (5) recent jobs of the tenant's agents on the new version, pinned to it, with tools that are not read-only answering from a stub, and compares how they and the few real jobs end with how they first ended (see [shadow runs](https://docs.oz0.ai/build/plugins/install/rollouts#shadow-runs-for-a-quiet-tenant)). With `--shadow-runs 0`, or no recent jobs to replay, a quiet tenant passes on health alone, and the step says so. When the last step passes, the new version becomes current and the old one drains, as above. When a step fails, or you abort, the rollout rolls back by itself: the upgraded nodes are told to run the old version again, the old version becomes current again, and jobs that started on the new version finish there. A runtime that does not start never gets past its canary. Here the canary's new runtime exited as soon as it started, with `--start-timeout 20s`: ```text Rollout rollout-0ca52a09: tenant default, runtime from 0.5.0+1a2b3c4d5e6f to 0.5.1+7c6d5e4f3a2b, rolled-back canary failed node-07ecbf417308: step 1: node node-07ecbf417308 did not run 0.5.1+7c6d5e4f3a2b (RUNTIME_STATE_CRASHED after 4 restarts: exited with status 0) within 20s step 2 pending node-5cd237748ea5 step 3 pending node-e8db051fed0d Reason: step 1: node node-07ecbf417308 did not run 0.5.1+7c6d5e4f3a2b (RUNTIME_STATE_CRASHED after 4 restarts: exited with status 0) within 20s ``` In `make e2e-rollout`, which does this while jobs run back to back, none of the 39 jobs that ran meanwhile failed: the other nodes, and the canary's old runtime, took them. A tenant has one rollout at a time: `rollout start` refuses while another runs. A server with the edge role drives it and holds a lease on it in the database; if that server stops, another one carries on from the same step within half a minute. The audit log has every rollout (`oz0 audit --action rollout.`: `create`, `start`, `step`, `done`, `rolled-back` and `abort`), and the metric `oz0_rollouts_total` counts how they ended. Plugin versions roll out the same way, canary first, when a plugin that runs on two or more nodes gets a new version: see [New versions roll out to canaries first](https://docs.oz0.ai/build/plugins/install/rollouts#new-versions-roll-out-to-canaries-first). Such a rollout waits while a runtime rollout runs, and `rollout list` shows both kinds. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.