---
title: "Upgrade a cluster"
description: "Back up the cluster and check the migration notes before changing servers. Some changes require plugin or node preparation first."
canonical_url: "https://docs.oz0.ai/fleet/upgrades"
---
# Upgrade a cluster

> Back up the cluster and check the migration notes before changing servers. Some changes require plugin or node preparation first.

For: cluster operators.

Back up the cluster and check the migration notes before changing servers. Some changes require plugin or node preparation first.

## Choose a task

<table>
<thead>
  <tr>
    <th>
      Task
    </th>
    
    <th>
      Guide
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Prepare for sandbox changes
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/sandbox-changes">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Review policy changes
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/policy-changes">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Review data and privacy migrations
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/data-migrations">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Prepare storage and identity migrations
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/storage-and-identity">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Prepare plugins for an upgrade
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/plugin-changes">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Keep calls compatible during upgrades
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/call-compatibility">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Review model and budget migrations
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/model-and-budget-changes">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Migrate to shared worker pools
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/shared-pools">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Upgrade node binaries and runtimes
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/nodes">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Understand runtime shutdown
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/runtime-shutdown">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Review runtime behaviour changes
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/runtime-changes">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Verify releases and switch channels
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/release-verification">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Roll a runtime out in steps
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/upgrades/runtime-rollouts">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Back up a cluster
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/backups">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Restore a cluster
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/backups/restore">
        Open guide
      </a>
    </td>
  </tr>
</tbody>
</table>

## Prepare the upgrade

1. Read the migration guides above for every release being crossed. Follow any plugin, runtime, database or backend preparation they require.
2. [Back up the cluster](https://docs.oz0.ai/fleet/backups), including its keys and erasure ledgers.
3. Upgrade the server roles in the order required by the applicable migration. The usual server procedure follows.
4. [Upgrade node binaries and runtimes](https://docs.oz0.ai/fleet/upgrades/nodes), then inspect node health and running jobs. A [runtime rollout](https://docs.oz0.ai/fleet/upgrades/runtime-rollouts) changes a tenant in steps.

## Upgrade servers

Upgrade one server at a time. `server start` applies pending schema migrations, ours and Temporal's, under a PostgreSQL advisory lock, so the first upgraded server migrates and the others wait. A binary refuses to start on a database whose schema is newer than it knows.

Temporal must move one minor version at a time. Read the release notes before you skip a release.

## Previous section links

<span id="installed-plugins-are-held-to-the-egress-allowlist">

Installed plugins are held to the egress allowlist

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/sandbox-changes#installed-plugins-are-held-to-the-egress-allowlist).

<span id="error-groups-start-again-and-older-failures-lose-their-words">

Error groups start again, and older failures lose their words

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#error-groups-start-again-and-older-failures-lose-their-words).

<span id="upgrading-to-v10">

Upgrading to v1.0

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/policy-changes#upgrading-to-v10).

<span id="upgrading-to-encryption-by-default-retention-in-temporal-and-the-key-checks">

Upgrading to encryption by default, retention in Temporal and the key checks

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#upgrading-to-encryption-by-default-retention-in-temporal-and-the-key-checks).

<span id="works-council-mode-is-on-after-the-upgrade">

Works-council mode is on after the upgrade

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#works-council-mode-is-on-after-the-upgrade).

<span id="lockfiles-and-hash-pins">

Lockfiles and hash pins

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#lockfiles-and-hash-pins).

<span id="eval-recordings-and-the-status-block">

Eval recordings and the status block

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#eval-recordings-and-the-status-block).

<span id="taint-and-the-taint-default-rule-v11">

Taint and the taint-default rule (v1.1)

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/policy-changes#taint-and-the-taint-default-rule-v11).

<span id="budgets-reserve-each-calls-worst-case">

Budgets reserve each call's worst case

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/model-and-budget-changes#budgets-reserve-each-calls-worst-case).

<span id="call-bindings-and-nodes-on-an-older-runtime">

Call bindings and nodes on an older runtime

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/call-compatibility#call-bindings-and-nodes-on-an-older-runtime).

<span id="a-nodes-reports-and-permission-asks-carry-their-calls-binding">

A node's reports and permission asks carry their call's binding

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/call-compatibility#a-nodes-reports-and-permission-asks-carry-their-calls-binding).

<span id="deferred-erasures-name-no-article">

Deferred erasures name no article

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#deferred-erasures-name-no-article).

<span id="approvals-that-wait-across-the-upgrade-are-asked-again">

Approvals that wait across the upgrade are asked again

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/policy-changes#approvals-that-wait-across-the-upgrade-are-asked-again).

<span id="rewording-a-rule-asks-no-waiting-approval-again">

Rewording a rule asks no waiting approval again

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/policy-changes#rewording-a-rule-asks-no-waiting-approval-again).

<span id="files-between-tools-become-their-job-trees">

Files between tools become their job tree's

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#files-between-tools-become-their-job-trees).

<span id="a-key-decides-an-approval-only-when-it-is-named">

A key decides an approval only when it is named

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/policy-changes#a-key-decides-an-approval-only-when-it-is-named).

<span id="memory-records-get-a-kind-a-purpose-and-an-end">

Memory records get a kind, a purpose and an end

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#memory-records-get-a-kind-a-purpose-and-an-end).

<span id="an-edges-payload-role-needs-the-job-files-scopes">

An edge's payload role needs the job files' scopes

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#an-edges-payload-role-needs-the-job-files-scopes).

<span id="the-edges-database-role-makes-nothing">

The edges' database role makes nothing

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#the-edges-database-role-makes-nothing).

<span id="sign-in-by-email-needs-a-verified-address">

Sign-in by email needs a verified address

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#sign-in-by-email-needs-a-verified-address).

<span id="private-networks-only-through-the-tenants-egress-list">

Private networks only through the tenant's egress list

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/sandbox-changes#private-networks-only-through-the-tenants-egress-list).

<span id="an-index-on-notification-deliveries">

An index on notification deliveries

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#an-index-on-notification-deliveries).

<span id="the-tool-policys-words-and-oz0-policy">

The tool policy's words and `oz0 policy`

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/policy-changes#the-tool-policys-words-and-oz0-policy).

<span id="manifests-one-part-on-its-own-addresses-and-requiresoz0">

Manifests: one part on its own, addresses and requires.oz0

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#manifests-one-part-on-its-own-addresses-and-requiresoz0).

<span id="addresses-in-commands-the-apis-and-tool-policy-rules">

Addresses in commands, the APIs and tool policy rules

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#addresses-in-commands-the-apis-and-tool-policy-rules).

<span id="what-an-install-prints-and-switches-the-server-places">

What an install prints, and switches the server places

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#what-an-install-prints-and-switches-the-server-places).

<span id="updates-are-checked-against-the-plugins-that-use-them">

Updates are checked against the plugins that use them

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#updates-are-checked-against-the-plugins-that-use-them).

<span id="verified-plugins-writes-ask-by-default-through-the-plugins-suggested-policy">

Verified plugins' writes ask by default, through the plugin's suggested policy

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/policy-changes#verified-plugins-writes-ask-by-default-through-the-plugins-suggested-policy).

<span id="job-ids-are-idempotency-keys-and-the-admin-apis-idempotency-key">

Job IDs are idempotency keys, and the admin API's Idempotency-Key

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/call-compatibility#job-ids-are-idempotency-keys-and-the-admin-apis-idempotency-key).

<span id="reasoning-effort-beside-the-model">

Reasoning effort beside the model

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/model-and-budget-changes#reasoning-effort-beside-the-model).

<span id="shared-pools">

Shared pools

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/shared-pools#shared-pools).

<span id="shared-pools-by-default">

Shared pools by default

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/shared-pools#shared-pools-by-default).

<span id="shell-and-write-sessions-run-on-sandbox-tier-1-whoever-started-the-job">

Shell and write sessions run on sandbox tier 1, whoever started the job

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/sandbox-changes#shell-and-write-sessions-run-on-sandbox-tier-1-whoever-started-the-job).

<span id="the-management-name-and-signed-calls">

The management name and signed calls

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#the-management-name-and-signed-calls).

<span id="event-sources-what-a-node-answers-is-held-to-the-delivery">

Event sources: what a node answers is held to the delivery

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#event-sources-what-a-node-answers-is-held-to-the-delivery).

<span id="event-sources-bound-values-compare-by-type-and-presence">

Event sources: bound values compare by type and presence

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/plugin-changes#event-sources-bound-values-compare-by-type-and-presence).

<span id="audit-exports-key-and-complete-take-a-signed-head">

Audit exports: `--key` and `--complete` take a signed head

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#audit-exports-key-and-complete-take-a-signed-head).

<span id="join-tokens-are-bounded-and-waiting-nodes-too">

Join tokens are bounded, and waiting nodes too

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/call-compatibility#join-tokens-are-bounded-and-waiting-nodes-too).

<span id="key-commands-get-a-minimal-environment">

Key commands get a minimal environment

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#key-commands-get-a-minimal-environment).

<span id="key-files-other-users-can-read-are-refused">

Key files other users can read are refused

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#key-files-other-users-can-read-are-refused).

<span id="erasures-are-recorded-in-the-ledger-or-refused">

Erasures are recorded in the ledger or refused

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#erasures-are-recorded-in-the-ledger-or-refused).

<span id="the-erasure-ledger-on-a-volume-of-its-own">

The erasure ledger on a volume of its own

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#the-erasure-ledger-on-a-volume-of-its-own).

<span id="subjects-are-matched-by-their-normal-form">

Subjects are matched by their normal form

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#subjects-are-matched-by-their-normal-form).

<span id="nodes-look-only-for-secrets-a-guess-would-not-find">

Nodes look only for secrets a guess would not find

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/data-migrations#nodes-look-only-for-secrets-a-guess-would-not-find).

<span id="secrets-come-from-files-or-the-environment-never-arguments">

Secrets come from files or the environment, never arguments

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#secrets-come-from-files-or-the-environment-never-arguments).

<span id="what-a-plugins-mcp-servers-say-is-held-to-more">

What a plugin's MCP servers say is held to more

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/sandbox-changes#what-a-plugins-mcp-servers-say-is-held-to-more).

<span id="each-calls-quota-charge-carries-the-calls-id">

Each call's quota charge carries the call's ID

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/model-and-budget-changes#each-calls-quota-charge-carries-the-calls-id).

<span id="upgrade-nodes">

Upgrade nodes

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/nodes#upgrade-nodes).

<span id="upgrade-the-runtime">

Upgrade the runtime

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/nodes#upgrade-the-runtime).

<span id="the-runtimes-shutdown-order">

The runtime's shutdown order

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-shutdown#the-runtimes-shutdown-order).

<span id="lost-harness-sessions-start-again-and-session-calls-idempotency-keys">

Lost harness sessions start again, and session calls' idempotency keys

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-shutdown#lost-harness-sessions-start-again-and-session-calls-idempotency-keys).

<span id="tenant-entries-need-a-runtime-from-the-same-release">

Tenant entries need a runtime from the same release

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-changes#tenant-entries-need-a-runtime-from-the-same-release).

<span id="strict-budgets-count-more-of-a-call">

Strict budgets count more of a call

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-changes#strict-budgets-count-more-of-a-call).

<span id="timings-on-model-calls-and-per-second-prices-from-the-send">

Timings on model calls, and per-second prices from the send

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-changes#timings-on-model-calls-and-per-second-prices-from-the-send).

<span id="failure-kinds">

Failure kinds

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-changes#failure-kinds).

<span id="a-sessions-streams-are-sockets">

A session's streams are sockets

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-shutdown#a-sessions-streams-are-sockets).

<span id="upgrade-the-node-binary">

Upgrade the node binary

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/nodes#upgrade-the-node-binary).

<span id="releases-and-their-signatures">

Releases and their signatures

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/release-verification#releases-and-their-signatures).

<span id="switch-release-channels">

Switch release channels

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/release-verification#switch-release-channels).

<span id="roll-a-runtime-out-in-steps">

Roll a runtime out in steps

</span>

: [open guide](https://docs.oz0.ai/fleet/upgrades/runtime-rollouts#roll-a-runtime-out-in-steps).

<span id="back-up">

Back up

</span>

: [open guide](https://docs.oz0.ai/fleet/backups#back-up).

<span id="restore">

Restore

</span>

: [open guide](https://docs.oz0.ai/fleet/backups/restore#restore).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.
