Upgrade node binaries and runtimes

For: cluster operators.

Install verified node and runtime versions while existing jobs finish on their original runtime.

Upgrade nodes

Every job stays on the runtime version it started on. Each tenant's Temporal namespace has a Worker Deployment, oz0-node, and a runtime's version is its Build ID: the runtime's workers for agents, flows and gates are versioned, and jobs are pinned to the version that started them, so an upgrade never replays a job's history with other workflow code. The node's own queue, node.<id>, which carries its tool calls, is not versioned.

The edge points each tenant's deployment at the version its nodes are meant to run: the version most of the tenant's online nodes are told to run, or run when told nothing, becomes current as soon as one of them runs it, and new jobs go there. While a rollout runs, it moves the deployment instead.

Upgrade the runtime

The edge serves every runtime version in its dist directory: make dist adds the one you build to dist/runtime/, and the newest is what new nodes install. Tell nodes to run another one:

oz0 node runtimes               # the versions the edge serves, and how many nodes run each
oz0 node upgrade node-7f3a01b2c4d5 --runtime 0.5.0+1a2b3c4d5e6f
oz0 node upgrade --all-in default --runtime 0.5.0+1a2b3c4d5e6f
oz0 node list                   # RUNTIME: what each node runs, and what it moves to

A node told to run a version downloads it from the edge, checks its digest, installs it and starts it next to the one it runs. New jobs go to the new version once most of the tenant's nodes run it, and jobs that started on the old version finish there. When no job is pinned to the old version any more, which Temporal reports within half a minute of the last one ending, the nodes stop it:

NODE               NAME    TENANT   STATE     STATUS         HEALTH       RUNTIME
node-40f20944d825  node-a  default  accepted  online on dev  healthy 100  0.5.0+1a2b3c4d5e6f, 0.4.0+9f8e7d6c5b4a

job get shows the runtime a job runs on. A version that cannot be downloaded or installed leaves the node on what it runs, and node show says why. node upgrade --runtime "" lets nodes keep what they run.

Upgrade the node binary

Nodes replace their own binary with the one the edge serves:

oz0 node upgrade node-7f3a01b2c4d5 --binary
oz0 node upgrade --all-in default --binary
oz0 node show node-7f3a01b2c4d5        # Supervisor: the version it runs

A node told to run another binary downloads the edge's release, and checks it before it changes anything:

1. The release's manifest.json must be signed with a key the node's own binary trusts.
2. The manifest's release channel must be one the cluster follows, and the version must not be older than the one the node runs, unless a cluster admin allowed it (below).
3. The binary for its platform must match the digest in that manifest.
4. The new binary must start and say it is the version asked for.

Once the release's signature and channel check out, the node drains, as for any stop: it takes no new work and waits for what runs there to end, within its stop grace (ten minutes by default).

Then it checks the release again, installs it, keeps its old binary next to the new one as orchestrator-zero-node.previous, stops its runtimes and starts again as the new binary, in the same process.

A release the node does not trust leaves it on what it runs, taking work again, and its log says why. A runtime-only upgrade (--runtime) needs no drain: the old version keeps the jobs that started on it, as above.

Because each node drains before it updates, oz0 node upgrade --all-in TENANT --binary drains every node of the tenant at the same time: the tenant takes no new job and starts no new session until each node's running work has ended or its grace has run out, up to about ten minutes with the default grace. When work must keep flowing, upgrade the binaries a few nodes at a time, naming them (oz0 node upgrade NODE NODE --binary), and the next batch once those are back.

The new binary has two minutes to reach an edge (OZ0_UPDATE_GRACE changes it). If it does not, or it stops before then, the node goes back:

• the previous binary is put back in place, and the new one is kept as orchestrator-zero-node.failed;
• the node does not install that version again. Upgrade it to another one, or delete node-update.json from the node's data directory and restart the node to try again.

The node records the update in node-update.json before the new binary starts. A new binary that crashes, and is started again by a service manager, still finds it and goes back.

Runtimes are checked the same way: a node takes a runtime's digest only from the signed manifest. A node that starts while no edge offers a runtime it trusts, because the edge is down or serves a release the node does not trust, starts the runtime it already has.