Restore a cluster

For: cluster operators.

Prepare the restore

A restore goes into an empty database: a new PostgreSQL database, or a new dev data directory. Before it writes anything, it checks every file in the archive against its digest, and checks that the master key unseals the backup's CAs. Then it inserts every row in one transaction and migrates to the binary's schema. A restore that fails leaves a dev data directory as it found it.

Add --erasure-ledger FILE (once per management server's ledger) so the restore applies again every erasure made after the backup was taken, before any server takes a job; it reports how many, and any ledger line that does not verify with the master key, which it never applies. Without it, a restore brings back the data of people erased since the backup (personal data).

Restore

server restore restores the archive's databases. If the deployment used a separate SQL payload database or an S3 payload bucket, restore that store with its own tools before starting the servers. Use a recovery point consistent with the management database so artifact rows and job payload references resolve. The archive does not contain those separately stored bytes. See what to back up.

oz0 server restore oz0-backup.tar.gz --master-key-file /etc/orchestrator-zero/master.key \
  --db-file /etc/orchestrator-zero/db-url
oz0 server restore oz0-backup.tar.gz --master-key-file .oz0/dev/master.key --dev --data-dir .oz0/restored

With a master key wrapped by a key command, give server restore the same --master-key-command; a dev restore copies the wrapped file into the new data directory as it is.

After a restore:

• start the servers with the same master key;
• with the roles apart, delete registry.snapshot from each edge's --data-dir before you start it: an edge refuses a registry older than the snapshot it holds, and counts as behind until it applies one;
• nodes keep their certificates and reconnect when they reach the edge at the address they know;
• operators get new credentials with server operator issue. A dev server issues its own when it starts;
• accounts keep their passwords, and sign in again.

A cluster restored without Temporal's databases has everything but the jobs that were running and their recent histories. A dev backup has those too. make e2e-backup backs up a dev server while a job waits for a person, restores it into a new data directory, and the job finishes there once it is approved.