Upgrade a cluster
For: cluster operators.
Back up the cluster and check the migration notes before changing servers. Some changes require plugin or node preparation first.
Choose a task
| Task | Guide |
|---|---|
| Prepare for sandbox changes | Open guide |
| Review policy changes | Open guide |
| Review data and privacy migrations | Open guide |
| Prepare storage and identity migrations | Open guide |
| Prepare plugins for an upgrade | Open guide |
| Keep calls compatible during upgrades | Open guide |
| Review model and budget migrations | Open guide |
| Migrate to shared worker pools | Open guide |
| Upgrade node binaries and runtimes | Open guide |
| Understand runtime shutdown | Open guide |
| Review runtime behaviour changes | Open guide |
| Verify releases and switch channels | Open guide |
| Roll a runtime out in steps | Open guide |
| Back up a cluster | Open guide |
| Restore a cluster | Open guide |
Prepare the upgrade
- Read the migration guides above for every release being crossed. Follow any plugin, runtime, database or backend preparation they require.
- Back up the cluster, including its keys and erasure ledgers.
- Upgrade the server roles in the order required by the applicable migration. The usual server procedure follows.
- Upgrade node binaries and runtimes, then inspect node health and running jobs. A runtime rollout changes a tenant in steps.
Upgrade servers
Upgrade one server at a time. server start applies pending schema migrations, ours and Temporal's, under a PostgreSQL advisory lock, so the first upgraded server migrates and the others wait. A binary refuses to start on a database whose schema is newer than it knows.
Temporal must move one minor version at a time. Read the release notes before you skip a release.
Previous section links
Installed plugins are held to the egress allowlist: open guide.
Error groups start again, and older failures lose their words: open guide.
Upgrading to v1.0: open guide.
Upgrading to encryption by default, retention in Temporal and the key checks: open guide.
Works-council mode is on after the upgrade: open guide.
Lockfiles and hash pins: open guide.
Eval recordings and the status block: open guide.
Taint and the taint-default rule (v1.1): open guide.
Budgets reserve each call's worst case: open guide.
Call bindings and nodes on an older runtime: open guide.
A node's reports and permission asks carry their call's binding: open guide.
Deferred erasures name no article: open guide.
Approvals that wait across the upgrade are asked again: open guide.
Rewording a rule asks no waiting approval again: open guide.
Files between tools become their job tree's: open guide.
A key decides an approval only when it is named: open guide.
Memory records get a kind, a purpose and an end: open guide.
An edge's payload role needs the job files' scopes: open guide.
The edges' database role makes nothing: open guide.
Sign-in by email needs a verified address: open guide.
Private networks only through the tenant's egress list: open guide.
An index on notification deliveries: open guide.
The tool policy's words and oz0 policy: open guide.
Manifests: one part on its own, addresses and requires.oz0: open guide.
Addresses in commands, the APIs and tool policy rules: open guide.
What an install prints, and switches the server places: open guide.
Updates are checked against the plugins that use them: open guide.
Verified plugins' writes ask by default, through the plugin's suggested policy: open guide.
Job IDs are idempotency keys, and the admin API's Idempotency-Key: open guide.
Reasoning effort beside the model: open guide.
: open guide.
: open guide.
Shell and write sessions run on sandbox tier 1, whoever started the job: open guide.
The management name and signed calls: open guide.
Event sources: what a node answers is held to the delivery: open guide.
Event sources: bound values compare by type and presence: open guide.
Audit exports: --key and --complete take a signed head: open guide.
Join tokens are bounded, and waiting nodes too: open guide.
Key commands get a minimal environment: open guide.
Key files other users can read are refused: open guide.
Erasures are recorded in the ledger or refused: open guide.
The erasure ledger on a volume of its own: open guide.
Subjects are matched by their normal form: open guide.
Nodes look only for secrets a guess would not find: open guide.
Secrets come from files or the environment, never arguments: open guide.
What a plugin's MCP servers say is held to more: open guide.
Each call's quota charge carries the call's ID: open guide.
Upgrade nodes: open guide.
Upgrade the runtime: open guide.
The runtime's shutdown order: open guide.
Lost harness sessions start again, and session calls' idempotency keys: open guide.
Tenant entries need a runtime from the same release: open guide.
Strict budgets count more of a call: open guide.
Timings on model calls, and per-second prices from the send: open guide.
Failure kinds: open guide.
A session's streams are sockets: open guide.
Upgrade the node binary: open guide.
Releases and their signatures: open guide.
Switch release channels: open guide.
Roll a runtime out in steps: open guide.
Back up: open guide.
Restore: open guide.