Prepare storage and identity migrations

For: cluster operators.

Check database ownership, role grants, certificates and key configuration before restarting servers.

Upgrading to encryption by default, retention in Temporal and the key checks

• Payloads are encrypted by default in a cluster. A cluster's servers start with payload encryption on unless they run with --encrypt-payloads=false (the Helm chart's temporal.encryptPayloads is true by default now). From the first upgraded server's start, new payloads reach Temporal encrypted with each tenant's key; the plaintext ones already in Temporal's histories stay readable, and running jobs carry on.
  Nodes' runtimes learn it within a minute, during which the edge refuses their plaintext writes and Temporal retries them. A decision on an approval sent to one of their jobs in that minute waits for the job, up to two minutes, instead of failing: Temporal refuses updates to a job whose workflow tasks keep failing, and the edge sends the decision again until the job takes it (a server from before this release answered unavailable, and the decision had to be given again).

Upgrade every edge before you rely on it: an edge still on the old release writes plaintext. Point any Temporal UI of your own at the edge's codec endpoint (Temporal's UIs), or it shows ciphertext. Dev mode does not change.

• Temporal's histories follow each tenant's retention. A tenant with a retention rule gets its namespace's retention set to it within minutes of the upgrade; a tenant without one keeps what its namespace keeps (7 days unless you changed it), raised only to 184 days when it has an agent marked high-risk. On Temporal Cloud, oz0 doctor says what to set. Jobs that closed before keep the expiry they had. See how long histories are kept.
• Doctor checks the keys and backups a recovery needs. In a cluster it warns until an offline copy of the master key (and of the secrets key, with the roles apart) is confirmed, and while the newest backup is older than the newest tenant payload key. Clear them with oz0 server master-key confirm-backup, oz0 server secrets-key confirm-backup and a backup (below).

An edge's payload role needs the job files' scopes

With the roles apart and an edge's own PostgreSQL role for payloads (OZ0_PAYLOAD_DB or --payload-db-file), the edge now also keeps each job file's scope, which job tree may read it, in the table file_scopes of that database. Grant the role that table before you upgrade the edges, as the payload and artifact tables: GRANT SELECT, INSERT, UPDATE, DELETE ON file_scopes TO oz0_payloads; (the upgrade's migration makes the table). Without it a job file is refused, and so is a job started under an ID you choose ("permission denied for table file_scopes"). An edge with --payload-store needs nothing.

The edges' database role makes nothing

On PostgreSQL, management now takes temporary tables from PUBLIC in its database at each start (and, before PostgreSQL 15, tables in the public schema), so that the role edges' statements run as can create neither, and it no longer lets that role set a sequence.

Management's user can take those rights when it owns the database and its public schema; before PostgreSQL 15 the public schema is the superuser's, so give it to management's user before you upgrade: ALTER SCHEMA public OWNER TO oz0;.

Until then a management server apart from the edges refuses to start, and one that runs every role starts without the internal API, both with the statement to run (what the database must allow).

An edge's statement that management does not know, such as a call of a function the platform's statements never make, is refused and logged on management once.

Sign-in by email needs a verified address

From this release, an identity provider with username_claim: email names an account by the address only when its ID token says email_verified: true, since most providers let people put any address in their profile and the address decides which account a first sign-in creates or links (Sign-in).

People already linked keep signing in. Microsoft Entra ID sends no such claim, so an Entra provider refuses every first sign-in after the upgrade until a cluster admin takes its addresses as verified: oz0 oidc set <id> --assume-email-verified, or assume_email_verified: true in the --oidc-config file (Microsoft Entra ID). A token that says false is refused whatever the setting.

An index on notification deliveries

On PostgreSQL the upgrade builds an index on the deliveries of notifications, so one approval's deliveries are read through it instead of the whole tenant's. Writes to that table wait while it builds: about 4 seconds per million deliveries on four cores. SQLite needs nothing.

The management name and signed calls

With the roles apart, management's certificate carries management.oz0 and the edges check it; management signs each call it hands an edge for a person or an operator whole, for that edge, once; and an edge's writes of key material (publishing the secrets key, re-sealing values, its signing keys and key sources) carry a proof of the master key. The temporal role's internode certificate carries temporal.oz0, which the edges check on its frontend. Upgrade management first, then the temporal role, then the edges soon after:

• Management issues its certificate at each start, so the upgraded management has the name, and keeps the edges' name: edges not yet upgraded still reach it.
• An upgraded edge does not reach a management that is not upgraded: it says the certificate is not management's.
• Until an edge is upgraded, management refuses its key-material writes (the edge's re-seal loops send them again once it is upgraded), and the edge refuses the console's runtime calls and plugin builds management hands it.
• A server started with --san management.oz0 (or *.oz0) refuses to start: only servers that run the manager role get the name.
• The temporal role issues its internode certificate at each start, so the upgraded temporal role carries temporal.oz0; an upgraded edge does not reach the frontend of a temporal role that is not upgraded (the handshake names the certificate).
• Each edge that was there before the upgrade re-seals the data subjects' keys and payload keys no edge sealed yet, for a day from its first start on this release at most; an edge added after the upgrade, or one whose data directory was lost, refuses them from its start.
  If every edge lost its data directory, or the cluster's keys were sealed before it had edges apart, run oz0 server secrets-key reseal-older on one edge's machine and restart that edge: it re-seals them for one more day. oz0 doctor warns until the edges have recorded that every key is sealed by an edge.
• An edge's read cache from before the upgrade is removed at its first start, and filled again from management's answers.
• Each server signs its row in the servers table once upgraded, and upgraded servers take only signed rows: until an edge is upgraded, management hands it no calls, the other edges leave it out of the nodes' edge lists and the fence asks it nothing.
• An edge's node reports carry only the node's facts, after its session: an edge not yet upgraded sends a node's facts before its session, which management refuses once (the next connection sends them again).

Key commands get a minimal environment

Key commands (--master-key-command, --secrets-key-command) and the CA's signing command (--ca-signer-command) no longer run with the server's whole environment, which holds its database URL and other credentials. They get what any program needs (PATH, HOME, the locale, the time zone, the temporary directory, the system's certificates) and the variables you name in OZ0_KEY_COMMAND_ENV: names separated by commas, each a variable or a prefix ending in *. A command that read anything else stops finding it, and the server does not start, nor does the Helm chart's init Job. Before you upgrade, set for example:

In the Helm chart, put it in server.extraEnv. What a command writes to stderr now goes only to the server's log, never into an error or doctor.

Key files other users can read are refused

The server refuses a key file (--master-key-file, --secrets-key-file) that other users can read or write, or that its group can write, and says which file and how to fix it: chmod 600 FILE. Group read stays allowed, for a Kubernetes Secret mounted with fsGroup. docker compose makes its master key 600 at every start. Doctor warns while a .plain copy of a wrapped key is left beside it.

Secrets come from files or the environment, never arguments

A command line is readable by every user of the machine (ps, /proc), stays in shell history and is printed by supervisors and orchestrators, so no command takes a secret as an argument any more. Orchestrator Zero is pre-release, so the old flags are gone without a deprecation period; a command that passes one stops with unknown flag:

The Helm chart already passed the database URLs from Secrets as environment variables; its otlp.secretHeaders are now mounted as files rather than expanded into the server's arguments, and otlp.headers go in OZ0_OTLP_HEADERS (a value with a comma or a percent sign belongs in otlp.secretHeaders). Docker Compose passes OZ0_DB in the environment as before. Change your own unit files and scripts the same way.