Releases

For: cluster operators.

Choose the channel and platform artifact you need, then verify it before installation. Use the upgrade guide when replacing an existing deployment.

What a release holds

A release is a flat list of files, as a GitHub release holds them:

The binaries trust the release key: an edge serves a dist folder's release only once its signature verifies, and a node checks the key's signature again before it installs a new binary or runtime from its edge (upgrades).

Release channels

Every release is on one channel, which its signed manifest.json (in the dist archive) and release.json both name, so an edge cannot change it without breaking the signature:

A cluster follows one channel. oz0 server init (and a dev server's first start) picks the channel of the release in the server's own dist folder, so a server installed from an LTS release starts on its line, and stable when there is none; --release-channel sets another. From then on:

• oz0 server dist pull mirrors only a release of the cluster's channel, and never an older node version over a newer one on the same channel;
• oz0 node upgrade --binary tells nodes only about a release of that channel, and refuses to send a node back to an older version;
• each node checks the same itself, against the signed manifest, before it replaces its binary.

Versions are ordered as Semantic Versioning says: v1.0.0-rc.2 comes before v1.0.0, which comes before v1.0.1. A build of a commit that is not a release (make dist from a working tree: a commit hash, or git describe output such as v1.0.0-5-gabcdef1) has no place in that order: a node that runs a release does not move to one unless you allow it, and a node that runs such a build may move to anything. Going back, or to another channel, is always an explicit act with a reason: see Switch channels.

oz0 server dist status shows the release a dist folder serves, its channel and whether its signature verifies; with the database (OZ0_DB, --db-file or --dev) also the channel the cluster follows. oz0 doctor warns when the two disagree.

A build's flavour, such as a FIPS build, is a property beside the channel: no channel makes a cluster take a build of another flavour.

LTS lines

An LTS line starts from a release tag: its branch lts/<major>.<minor> is cut from v<major>.<minor>.0, and its releases are v<major>.<minor>.Z built with make release-candidate CHANNEL=lts-<major>.<minor>.

Fixes go back to the line as cherry-picks that name the commit they came from, never with a database migration (a fix that needs one reaches the line's clusters with the next minor release); security fixes reach the main branch and every supported release branch no later than any LTS line. The backported commits are part of the core source. How long each line is supported is published with its first release.

With repository access, you can build a line from its branch and sign it with your own key; the binaries then trust that key, as with any build (upgrades). Public source downloads are coming soon.

Install from a release

On Linux or macOS, the server:

curl -fsSL <release-url>/install.sh | sh -s - server
oz0 server start --dev

As root it goes to /opt/orchestrator-zero/<version> with a link in /usr/local/bin; otherwise to ~/.local/share/orchestrator-zero and ~/.local/bin. --prefix and --bin-dir change both. The dist folder comes with it, so the edge can serve nodes at once.

For nodes that should reach nothing but the edge, unpack the toolchain archive of each of your nodes' platforms into the same folder: it adds dist/toolchain/ with that platform's files and manifest (dist/toolchain/platforms/<os>-<arch>.json), the edge serves every platform it finds there, and nodes run with oz0-node run --offline (or OZ0_NODE_OFFLINE=1) never fall back to the internet:

tar -xzf orchestrator-zero-toolchain_<v>_linux_amd64.tar.gz -C /opt/orchestrator-zero/<version>

A node, joined to a cluster and run as a service:

curl -fsSL <release-url>/install.sh | OZ0_JOIN_TOKEN='K10…' sh -s - node --server https://oz0-edge:7443

--foreground runs it in the terminal instead (and so does a machine without systemd, such as a container), --no-run joins without starting, --service-name and --no-linger pass to service install, --no-sandbox-setup skips the sandbox setup it runs as root on Linux (bubblewrap, and Ubuntu's own AppArmor profile for it), --data-dir and --name work as for oz0-node join, and --from installs from another URL or from a folder, for machines without internet access. The one-line install from the console is still the quickest way to add a node to a running cluster: it installs from your edge.

--channel stable|preview|lts-<major>.<minor> makes the script refuse a release that a cluster on that channel would not take, as the release's signed release.json says, before it installs anything: --channel preview takes a stable release too.

The script checks every file against SHA256SUMS, and SHA256SUMS against the release key it carries, with openssl (OpenSSL 3 checks ed25519 signatures; macOS's LibreSSL and OpenSSL before 3 do not). It installs only once openssl says the signature verifies. Where there is no openssl that can check it, it stops and says so: install OpenSSL 3, or check SHA256SUMS with cosign yourself and pass --digests-only, which checks each file against SHA256SUMS alone and says so. Any other answer of openssl stops the install, with or without the flag.

It is tested on Ubuntu 24.04, Debian 13, Fedora 42 and Alpine 3.22. On Alpine (musl) the server runs and the node binary installs and joins, but the node's Python runtime does not install yet, so oz0-node run stops: one of its dependencies, cel-expr-python, publishes no musl wheel. Use a glibc distribution for nodes.

Check a release

Get the release's public key from somewhere other than the release itself, then:

sha256sum -c SHA256SUMS
cosign verify-blob --key release.pub --bundle SHA256SUMS.sigstore.json --insecure-ignore-tlog=true SHA256SUMS
cosign verify-blob-attestation --key release.pub --bundle provenance.sigstore.json \
  --type slsaprovenance1 --insecure-ignore-tlog=true orchestrator-zero-node_<v>_linux_amd64

The project's releases are signed by release key A, or by its standby B, on a machine that never goes online; every official binary trusts both, so a release signed by B installs and updates like one signed by A. Their public halves, to check release.pub against:

release.json's key names the one that signed; the commands above are the same whichever it is. The signatures are made with the release key and recorded in no transparency log, so --insecure-ignore-tlog=true is expected. cosign reads at most 128 MiB of a file it checks against the provenance, and the toolchain archives are larger: set COSIGN_MAX_ATTACHMENT_SIZE=512MiB for them. From a checkout of the source, go run ./scripts/releasetool verify <folder> release.pub checks all of it, and also that the binaries trust the key, that each SBOM belongs to its file and that the dist archive serves the release's own node binaries.

Every file but the images is reproducible: building the same commit with make release-candidate gives the same bytes. It refuses a tree with uncommitted changes or untracked files, since those would be built in too. With ALLOW_DIRTY=1 it builds one as a trial: the version ends in -dirty, release.json and the provenance say "dirty": true, and verify prints a warning.

Sign your own releases offline

A cluster that builds its own releases can keep its key off the build machine too. make release-candidate SIGN=offline RELEASE_PUB=<your key's public half> builds with no private key, reads none and makes none, and prints a folder to carry to the machine that holds the key.

There, release-sign sign-bundle --key <seed file, or - for stdin> <folder> checks the folder against the digests it lists, refuses a key other than the one it asks for, and writes the signatures into the folder; release-sign show <folder> first says what it is asked to sign.

It signs only files it can name (the dist manifest, and in Enterprise a sector pack's payload) and refuses any text it would print with a control character in it, so what the terminal shows is what gets signed.

Bring the folder back and run the same make command with SIGNED=<folder>.

A release takes two such trips: the first signs the dist manifest, which the dist archive and the server image hold, and the second the provenance and SHA256SUMS.

The build machine refuses anything that is not what it wrote, signed by that key, and the result checks with the same commands as above.

release-sign runs without Go: the release holds it, built static for Linux, and make release-signer builds the same bytes on another machine to compare, without reading or making a key there.

What the provenance says

Each release carries an SLSA v1 provenance in provenance.sigstore.json, signed with the release key. Its buildType and its builder's id are links to the two sections below, which say what they mean.

Release candidate v1

https://docs.oz0.ai/fleet/releases#release-candidate-v1 is the buildType of a release built with make release-candidate: it builds the server and node binaries, the runtime bundle, the toolchain archives and the images from one commit, writes their SBOMs and release.json, and signs SHA256SUMS and the provenance with the release key. Its fields are:

go run ./scripts/releasetool verify refuses a provenance of any other buildType. A candidate built before the buildType was this link (it pointed at a file in the source repository) does not verify: build it again. No release was published with the earlier value.

Local builder

https://docs.oz0.ai/fleet/releases#local-builder is the builder id of a release candidate built on a person's machine. builder.version names the version of each tool the build used. SLSA counts such a build as Build L1: the provenance exists and is signed, but nothing stronger than the release key vouches for the machine that built it. Rebuilding the same commit and comparing the bytes is how anyone checks it further.

Two kinds of SBOM

Releases and plugins each have their own SBOMs, both CycloneDX:

See SBOMs for the plugins'.

For a build of your own from source, make sbom writes the same kind of SBOMs into dist/sbom/: one each for the server and node binaries in bin/, one for the runtime bundle (with the bundle as one archive beside it), and one for each plugin in the repository's plugins/ folder, as plugin pack writes it.

It uses the release's own pinned syft, downloaded once into ~/.cache/oz0-tools and checked against its published SHA-256, and the release's own normalising, so each SBOM is bound to the SHA-256 of the file it describes and the same build gives the same SBOM.

RUNTIME=0 and PLUGINS=0 leave those out, and SBOM_OUT names another folder (a new or empty one, or one an earlier run wrote: it never clears a folder that holds anything else); images and toolchain archives are only built, with their SBOMs, by make release-candidate.

The FIPS flavour

A release is built in one flavour. The FIPS flavour's release candidate is make release-candidate FLAVOUR=fips: the same files, every Go binary built against Go's FIPS 140-3 module snapshot (FIPS140_MODULE, default v1.26.0), in dist/release/<v>-fips. Its release.json and the dist folder's manifest.json say "flavour": "fips", and release.json's parameters name the snapshot (gofips140).

It has no container images yet. Its install.sh downloads from the tag <v>-fips unless RELEASE_URL names another place, and, like the default release's, installs only binaries of its own flavour: the two releases name their files alike and are signed with the same key. An edge serves, oz0 server dist pull mirrors and a node installs only a release of its own flavour, so a FIPS cluster never upgrades to a default build.

Upgrade from an earlier version

Each release is tested by upgrading the data of earlier versions to it, on SQLite and PostgreSQL, with a node that replaces its own binary across the same versions.

Install the new release next to the old one, stop the server, and start the new one on the same data: it migrates the database. install.sh server puts the new dist folder next to the new binary, so the edge serves the new node binaries and runtime.

An edge on another machine takes the release from a folder: unpack orchestrator-zero-dist_<version>.tar.gz there and run oz0 server dist pull --from <the unpacked dist folder> (or serve the folder over HTTP and give oz0 server dist pull its URL).

Either way the signature is checked first, and the toolchain comes along when the folder has one. A cluster with no way out takes it from an offline folder.

Then upgrade the nodes from the console or with oz0 node upgrade --binary. See upgrades.