Run a cluster
For: cluster operators.
Prepare PostgreSQL, network access and the master-key backup before starting the servers. For separate management and edge machines, use the separate-role procedure.
This procedure creates a cluster of combined orchestrator-zero servers sharing PostgreSQL. Each server process runs management, edge and embedded Temporal, which is the default layout. Servers find each other through the database; there is no seed list to maintain.
What you need
- PostgreSQL 13 to 16, reachable from every server. The user in the URL should be allowed to create databases; otherwise create the three databases yourself (see below).
max_connectionsof at least 100 per server. Temporal opens a connection pool per service and store.- The
orchestrator-zerobinary on every server, and a copy of the master key file once it exists. - Open ports between servers: 7234 to 7236 and 7239 (Temporal gRPC), 6933 to 6935 and 6939 (Temporal membership). Nodes need 7233 and 7443; operators need 8443.
Create the cluster once
export OZ0_DB='postgres://oz0:<password>@db.internal:5432/oz0?sslmode=require'
oz0 server init --cluster-name prod-eu --history-shards 1024 \
--master-key-out /etc/orchestrator-zero/master.key
server init creates three databases, named after the one in the URL (oz0, oz0_temporal and oz0_temporal_visibility), applies Temporal's schema migrations and ours, and creates:
- The master key file. Back it up and keep it secret. Every server needs a copy; without it the cluster's CAs, secrets and payload keys cannot be unsealed, encrypted histories can never be read again, and the cluster is lost. Keep one copy offline, apart from the database's backups, and record that you did:
oz0 server master-key confirm-backup --master-key-file /etc/orchestrator-zero/master.key(it records a check value, never the key). Until thenoz0 doctorwarns (master key backup); see what doctor checks. - Two certificate authorities, sealed with the master key and stored in the database: the cluster CA (nodes, operators and the edge's ports) and the internode CA (only Temporal's traffic between servers). The cluster CA's key can live in an HSM or a KMS instead; see in this guide.
- The first tenant,
default, with a Temporal namespace of the same name. - The first operator,
admin, whose certificate and CLI context go to~/.config/orchestrator-zero/<cluster>/. - The console's first account, also
admin, with a generated password thatserver initprints once. Sign in athttps://<server>:8443; see Console.
Choose the history shards carefully
The number of Temporal history shards can never change. 1024 suits a cluster that may grow to thousands of state transitions per second; small clusters can use 64 or 128.
Start the servers
On each server, with the same database URL and a copy of the master key:
export OZ0_DB='postgres://...'
oz0 server start --master-key-file /etc/orchestrator-zero/master.key --advertise 10.0.0.11
--advertise is the address other servers and nodes reach this server on. Add --san edge.example.com for every extra name nodes use to reach the edge, such as a load balancer. oz0 server start --help lists the flags most deployments set first, under Common flags, and the others after them.
A cluster encrypts every payload with its tenant's key before it reaches Temporal, so Temporal's databases hold only ciphertext of what jobs read and wrote; --encrypt-payloads=false turns that off, which the server and oz0 doctor warn about. Each tenant's Temporal namespace keeps histories as long as the tenant's retention says. See Temporal backends.
Point the CLI at the cluster
export OZ0_CONTEXT=~/.config/orchestrator-zero/prod-eu/context.json
oz0 cluster info
cluster info shows the cluster's name, its CA hash and the servers that are up.
Operator certificates
An operator's certificate is valid for 90 days. Commands warn on stderr when it has less than 15 days left. Renew it while it works:
oz0 operator renew
The CLI makes a new key, the cluster signs a certificate for it, and both replace the ones in the context's directory. The CLI checks the new certificate before it writes anything (it is for the new key, the cluster's CA signed it for this operator of this cluster, and it has not expired), so the pair in use stays as it was when the answer is not right.
The new pair is written beside the old one and then moved into place; if that stops halfway (a crash, a full disk), the next oz0 command finishes it, so the key and the certificate always belong together. Another oz0 command that runs during a renewal waits for it, and operator renew reports success only when the renewed pair is the one in place. The audit log records it as operator.renew.
An operator whose certificate expired or was lost gets new credentials on a server's machine, from the database and the master key. The command also adds an operator the cluster does not have yet:
oz0 server operator issue admin --master-key-file /etc/orchestrator-zero/master.key \
--operator-dir ~/.config/orchestrator-zero/prod-eu --advertise oz0.example.com
A context already in --operator-dir keeps its addresses; a new one points at --advertise. Whoever can run it is trusted like a server, since it reads the master key. For a dev server, use --dev --data-dir <dir>.
An operator's certificate that was lost or stolen stops working when you disable the operator, on a server's machine. Every route that takes an operator's certificate asks the same check, so it is refused everywhere: at once on the management port (the admin API, the console's sign-in, the operators' MCP server, Temporal's UI and its codec, and the calls management hands an edge), and within seconds on every edge (the runtime API, the Temporal port, the model gateway, the codec endpoint and the edges' status for each other).
A refused certificate is never taken as someone signed out. An expired certificate is refused the same way. None is renewed or issued for a disabled operator until you enable it again. Both are in the audit log as operator.disable and operator.enable:
oz0 server operator disable ci-bot # with OZ0_DB set, or --db-file FILE
oz0 server operator enable ci-bot
To give the operator a new certificate after a theft, enable it and run server operator issue; the stolen one still works until it expires, so keep the operator disabled until then, or issue the new credentials under a new name.
Keys stay in memory
Each server issues itself certificates from the cluster's CAs at start and renews them every day, in memory. No server key is written to disk; holding the master key is what makes a machine a server.
The cluster CA's key in an HSM or a KMS
The cluster CA's key can live outside the servers, in an HSM, a cloud KMS or Vault, so that no server and no backup ever holds it. The servers then sign through a signing command you provide: a small program or script that speaks two verbs.
| Command | Input | Output |
|---|---|---|
COMMAND public-key | none | the key's public half, as a PEM PUBLIC KEY block |
COMMAND sign sha256 | a 32-byte SHA-256 digest on stdin | the ECDSA signature, ASN.1 DER or the 64 raw bytes of r and s, on stdout |
The key must be ECDSA P-256. The command runs without a shell and with a small environment of its own, plus the variables you name in OZ0_KEY_COMMAND_ENV (as key commands do), so a PIN or a token reaches it the way you choose and the rest of the server's environment does not; a non-zero exit fails the signature, and what it wrote to stderr goes to the server's log.
Each run may take 30 seconds; after that the command is killed, on Linux and macOS together with any processes it started, such as the other half of a pipeline. Every signature is checked against the public key before it is used.
A wrapper around pkcs11-tool --sign --mechanism ECDSA works with any PKCS#11 token; one around your KMS's CLI works with a cloud KMS. The command signs when a server starts, when a node joins or renews its certificate, when an operator renews, and when a node is purged.
Create a new cluster this way:
oz0 server init --cluster-name prod-eu --master-key-out /etc/orchestrator-zero/master.key \
--ca-signer-command /usr/local/bin/oz0-hsm-sign
and start every server, and run server operator issue, with the same --ca-signer-command. A server started without it refuses to start. cluster info says where the key is. The internode CA, which only Temporal's traffic between servers uses, stays sealed in the database.
Move an existing cluster's key out in four steps:
# 1. Write the key to a file only you can read, and import it into the HSM or KMS.
oz0 server ca export-key --out ca-key.pem --master-key-file /etc/orchestrator-zero/master.key
# 2. Delete the file, then restart every server with --ca-signer-command.
# 3. Check that the command signs as the CA, and delete the sealed copy from the database.
oz0 server ca forget-key --ca-signer-command /usr/local/bin/oz0-hsm-sign \
--master-key-file /etc/orchestrator-zero/master.key
Both steps are in the audit log, as cluster.ca-key-exported and cluster.ca-key-forgotten. Backups taken afterwards hold no CA key, and servers restored from one start with the signing command.
The master key and the secrets key can be kept wrapped in the same way, with the key that unwraps them in an HSM or a KMS: see Key commands.
Next
- Join nodes
- High availability
- Upgrades, backups and restores
- Install on Kubernetes: the same cluster with the Helm chart