Encrypt Temporal payloads

For: cluster operators.

Configure payload encryption and preserve its keys before relying on encrypted histories.

Encrypted payloads

In a cluster (the embedded or an external Temporal on PostgreSQL), every payload is encrypted with its tenant's key before it reaches Temporal, unless the servers start with --encrypt-payloads=false; with Temporal Cloud always. Dev mode stays as it was: clear unless --encrypt-payloads. A cluster that runs with --encrypt-payloads=false logs a warning at start, and oz0 doctor warns (payload encryption) until it runs without the flag.

What is encrypted: inputs, results, memos, the summaries of each step, approvals, and the messages and stack traces of the errors your code raises. Temporal holds only ciphertext (AES-256-GCM, encoding: binary/oz0-tenant) of what jobs read and write. The node encrypts what it writes; the edge encrypts what it writes and decrypts what it reads, so the console and the CLI show jobs as before.

What Temporal (Cloud) still sees in the clear, because Temporal needs it to run your jobs:

• workflow IDs and types, activity types, and task queue names, which carry agent and pool names;
• search attributes, such as routine names and triggers;
• the headers of workflows, activities and signals, such as the trace context the SDKs add;
• timers, timeouts and retry settings, and the sizes and times of everything;
• the short messages of failures Temporal and the SDKs make themselves (a timeout, "activity error", a nondeterminism error) and the reason the edge gives when it cancels or terminates a job.
• Data subjects. A job started with --data-subject is encrypted with the person's key first and then with the tenant's, so Temporal does not even see which person a payload belongs to. Erasing the person still makes their payloads unreadable (personal data).
• External Storage. Large payloads are stored on the edge as the codecs made them, encrypted too.
• Keys. Each tenant has its own keys, sealed with the master key and kept in the database; they are in every backup. Rotate with oz0 tenant rotate-payload-key <tenant> --reason "<why>", or the console's Rotate payload key (below): within a minute the edge and the nodes encrypt with a new key, and the old one still decrypts. tenant list --json counts each tenant's keys.
• Each key's life. The edge retires a key at seven days: the edge that next reads it after that retires it and makes the next one, as a rotation does, and the audit log keeps it as tenant.payload_key.rotate by system:payload-keys. Retiring needs management's database, so while management or its database cannot be reached, as for an edge apart from management, the edge goes on encrypting with that key, logs each failed try, and retires it on the first read after they are back.
  Every payload gets a random 96-bit nonce, which keeps a key safe for 2^32 encryptions (NIST SP 800-38D); to reach that within a week, a tenant would have to encrypt over 7,000 payloads a second, every second, so even an outage of weeks leaves a key far below it.
• Old runtimes. In encrypted mode the edge refuses what a node writes unencrypted, payloads and error messages alike, such as a runtime from before this feature, with a message that says so. A runtime learns that encryption was turned on within a minute; until then its writes are refused and Temporal retries them.
• Every field. The check for encryption covers every payload of a node's call wherever it sits, except search attributes and headers, which Temporal must read and no codec encodes.
  A payload passes it by its form, not by its label: an encrypted one holds only its encoding, its key's name and bytes long enough for a nonce and a tag, and a reference to External Storage is the two metadata keys, the edge's driver and one digest, which the edge reads before it takes it as a reference.

The edge holds no plaintext to compare with: whether the bytes under the encrypted label are ciphertext only the key shows, so this keeps an old or broken runtime out, not a node that sets out to write plain bytes.

Separately, a payload marked as one of Temporal's own system payloads, which carries a message whose other fields Temporal never encrypts, is refused in any field of any call, search attributes and headers included, from nodes and operators alike, and also when payloads are not encrypted.

Nodes and the CLI send none (the SDKs use them only for Nexus operations).

What reads a job's history on the edge (a held tenant's copy, an erasure, a replay) does not read search attributes as payloads, so no value Temporal took there can make a history unreadable to it.

• oz0 job run. An operator's raw workflow start through the edge's Temporal port is encrypted by the edge with the tenant's key, and its answers are decrypted for it: operators are cluster admins, who may see every tenant's content.
• Turning it on. A cluster upgraded from a release where it was off encrypts new payloads from its next start and keeps reading the plaintext ones its histories already hold; they stay plaintext in Temporal until those histories expire.

Encryption status and key rotation

Every member of a tenant can see how its data is encrypted: in the console under Settings, Encryption, with oz0 tenant encryption <tenant> (--json for scripts), or through the admin API's GetEncryptionStatus. It shows:

Each line that is not as it should be says what to do. Most fixes are an operator's, on a server's machine, with the database and the key files: oz0 server master-key confirm-backup, oz0 server secrets-key confirm-backup, oz0 server backup (or oz0 server backup record after a backup with PostgreSQL's own tools), a key command, or a server flag. The status says so, and the console shows the command to copy. A dev server skips the backup lines and shows plaintext payloads as how a dev server starts, not as a problem.

What a tenant's members see of the cluster's keys is yes or no: where each comes from, whether its offline copy is confirmed. The payload key's ID, the key command's holder and errors, the names of the servers that do not encrypt, who confirmed a copy and when the newest backup was taken are shown to cluster admins only, and nobody sees a key, a check value, another tenant or a data subject. Readers and up may read the status; cluster admins may read any tenant's.

Rotating the payload key. A tenant admin rotates it in the console (Rotate payload key…, which asks why) or with oz0 tenant rotate-payload-key <tenant> --reason "<why>" (--decision-ref for a case number; the CLI takes $OZ0_REASON when --reason is not given). What it does:

• new payloads are encrypted with a new key: the edge makes it within 30 seconds, and nodes take it within a minute;
• payloads already in Temporal stay readable: the retired key still decrypts them, and jobs that run across the rotation carry on;
• nothing is re-encrypted, so a key that leaked before the rotation still opens what it encrypted, and no key is deleted while the tenant exists;
• a database backup taken before the rotation does not hold the new key: take one afterwards, and the status's database backup line turns green.

The edge also rotates each key by itself at seven days, once management's database can be reached (see each key's life above); a rotation by hand resets that week.

The audit log keeps each rotation as tenant.payload_key.rotate, with who rotated, the retired key, the reason and the decision reference. Rotation needs no second admin: it only tightens.