--- title: "Verify releases and switch channels" description: "Review signed distribution and rollback allowances before moving a fleet between releases." canonical_url: "https://docs.oz0.ai/fleet/upgrades/release-verification" --- # Verify releases and switch channels > Review signed distribution and rollback allowances before moving a fleet between releases. For: cluster operators. Review signed distribution and rollback allowances before moving a fleet between releases. ## Releases and their signatures `make dist` signs `dist/manifest.json` with a release key, and every binary it builds trusts that key. The first time, it makes a key of its own in `~/.config/orchestrator-zero/release.key`. Keep that file safe: whoever has it can sign node binaries your nodes will run. Set `RELEASE_KEY` to keep it somewhere else. Mirror a release that is served elsewhere, such as a release's download folder, into the edge's dist directory: ```bash [Terminal] oz0 server dist pull https://releases.example.com/orchestrator-zero/0.7.0 --dist-dir /srv/oz0/dist ``` The server checks the release's signature against the keys its own binary trusts, downloads every file it names and checks each digest in a staging folder inside the dist directory. New files go in place beside the ones served now, then the manifest, and only then a file that takes the place of one on disk, so nodes see the new release whole or not at all, and a pull that fails leaves what the edge served as it was. Node binaries of releases older than the one before are removed from the dist directory; runtimes stay until you delete them. The edge itself serves a dist directory's release only while its `manifest.json` verifies against `manifest.json.sig` with the keys its binary trusts, is of its flavour and names a valid channel, read again whenever the files change: a release that does not verify is never served, and a change that does not verify keeps the release verified before. It serves the manifest and its signatures as the bytes it checked, and a node binary or runtime only when its content has the digest the signed manifest gives it. A server binary built without a release key serves no release. A release that verifies is served only when the cluster takes it: it is on the channel the cluster follows, and its node version is no older than any release this edge served on a channel the cluster takes, nor than the newest release `node upgrade --binary` told any of the cluster's nodes to run, unless it is the version a cluster admin allowed going back to. What a node says it runs never counts. The edge keeps its record of what it served in its data directory (`dist-served.json`, apart from the dist folder), so a restart does not forget it; a record it cannot read stops it serving any release until you remove the file, and it says so. Until then the edge keeps the release it served, logs why and looks again every 15 seconds, so an allowance or a channel switch takes effect without a restart. To go back on purpose: pull the older release with `server dist pull --allow-older --reason ...`, then allow it with `node upgrade --binary --allow-older --reason ...`. An allowance is for one step back: an edge that serves a newer release after it no longer goes back to that version, and an upgrade to another version without `--allow-older`, or a move to another channel, ends it for the whole cluster. To go back again later, allow it again. `server dist status` and the version an upgrade offers show the folder's verified release whether or not the edge serves it yet. The pull keeps to the cluster's [release channel](https://docs.oz0.ai/fleet/releases#release-channels): it refuses a release of another channel, and an older node version than the folder's verified release on the same channel, or one that names no node version at all. On a server's machine with the database (`--db`, `OZ0_DB` or `--dev`) it reads the cluster's channel; without it, the folder's own, and a folder whose release does not verify sets no order, so the pull then needs the database or `--allow-older`. A signature proves that a release is yours, not that it is the newest. Nodes that run a release refuse an older one, and one of another channel, whatever their edge serves. A node keeps the channel it was installed on in its data directory, read from the signed release it runs, and the edge's word can only narrow it: a node refuses a binary of a channel its installed channel does not take, and takes a runtime only from a signed release of that channel, no older than the release it last took one from. That is the release its runtime came from, not its binary's, because nodes may be upgraded before the servers: a new machine ahead of its edge takes the runtime the edge serves. So a machine that has taken no runtime yet has nothing to compare with, and takes what its edge serves of that channel. A binary older than the release last taken lowers nothing, since a node whose runtime moved ahead of its binary looks the same: only a step back management signed does, to the release it names, until the node has taken a runtime from it; a newer release an edge that has not moved yet serves meanwhile does not end the step. A node told to go back to the binary it already runs uses the allowance the same way, and takes that release's runtime the next time it starts one. If you put an older binary back by hand, give an allowance for it, or remove `runtime-release` from the node's data directory. Going back needs management's word, not the edge's: `node upgrade --binary --allow-older` signs an allowance with the cluster CA's key, and a node checks that signature against the CA certificate it joined with before it uses it. With no valid one it stays where it is and logs why. An allowance names one version, lasts seven days from when it was given, and takes each node back once, so a node that was offline for longer needs it given again. Within those seven days someone who controls an edge can still use an allowance you gave, on a node that has not used it, after you moved the cluster on. The order a node keeps is of releases, not of the runtimes in one: a signed release lists older runtimes too, so that a rollout can roll back and jobs on an older version can finish, and which of them a node runs is its edge's word. So someone who controls an edge can move a node to an older runtime that a release it takes still lists, known flaws included, with no allowance. A runtime leaves that reach once the node has taken a runtime from a release that no longer lists it. To retire a release for good, sign the next ones with a new key and upgrade the nodes to binaries that trust only it. ## Switch release channels Moving a cluster to another channel, or nodes back to an older release, is an explicit act with a reason, recorded in the audit log: ```bash [Terminal] oz0 server dist channel # the channel the cluster follows (OZ0_DB set) oz0 server dist channel lts-1.0 --reason "stay on the 1.0 line" # move it (on a server's machine) oz0 server dist pull https://releases.example.com/1.0.8 --dist-dir /srv/oz0/dist orchestrator-zero-node release-channel lts-1.0 --data-dir /var/lib/orchestrator-zero-node # on each node's machine: the edge cannot do this oz0 node upgrade --all-in default --binary --allow-older --reason "onto the 1.0 line" ``` A node stays on the channel it was installed on until someone who runs its machine moves it with `orchestrator-zero-node release-channel`, so that whoever controls an edge cannot move a fleet to another line of signed releases. Until then it refuses the new channel's release and keeps running, and its log says why; moved after that, it takes the release when it next connects to an edge, so restart it. A preview node takes stable releases and stays on preview. Give the `--data-dir` the node runs with: the command refuses a folder no node has joined from and says which file it wrote. A node that has recorded no channel yet (one that came from a bundle, or was behind the edge's release when it first looked) and runs a release refuses release candidates until it is moved the same way. `server dist channel` needs the database, not the master key, like `server operator disable`. `server dist pull --channel lts-1.0 --switch-channel --reason ...` does both steps at once on a machine with the database; without it, the switch moves only that edge's dist folder, and the cluster's channel stays until `server dist channel` moves it: until then management upgrades no node to the folder's release, and `oz0 doctor` says why. Going from stable to an LTS line usually means going back in version (`v1.2.0` to `v1.0.8`): `node upgrade --binary` refuses it until `--allow-older` and a reason say so. Only a cluster admin may pass `--allow-older`, whatever role they hold in the nodes' tenant, since an older release can bring back a vulnerability a newer one fixed, and management, which signs the allowance, is the only one that can give it. `--allow-older` on `server dist pull` lets an older release replace a newer one in the dist folder in the same way. Leaving an LTS line for stable goes forward, so it needs only the switch. The edges serve the LTS release to new machines once `node upgrade --binary --allow-older` has allowed going back to it; until then they keep the release they served. An edge looks again within 15 seconds, and a node told sooner than that fails its update and asks again when it next connects, so restart a node that stays where it is (`oz0 node show` says what it runs). Allow it after the switch: moving the cluster to another channel ends an allowance given before. The one-line install trusts the edge: it pins the cluster's CA and checks the binary's digest from the edge's manifest. From then on, the node checks signatures itself. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.