Keep plaintext secrets on edges

For: cluster operators.

Only edges hold the private secrets key. Review resealing and recovery before changing that key.

Secrets management cannot open

The first edge that starts with --secrets-key-file publishes the key's public half. From then on management seals every tenant secret to it, and the edges seal the existing ones to it within seconds; management stores them and can no longer open them.

The keys of data subjects, which encrypt the payloads of their jobs, and the tenants' payload keys, which encrypt jobs' payloads in Temporal, move to the same key.

A backup holds all three sealed to a key that is not in it, so keep a copy of the secrets key with the master key: without it, the secrets, the data subjects' keys and the payload keys are lost, and with the keys every payload they encrypt.

Once a cluster has a secrets key, every server that runs the edge role needs it, including servers that run all three roles. A cluster that never makes one goes on sealing with the master key.

The keys of data subjects and the tenants' payload keys an edge seals carry a tag only a holder of the secrets key can make, so management, which knows the key's public half, cannot seal one an edge would take; and an edge seals nothing to a published key that is not its own, of either flavour (it refuses, logs an error and records secrets.key.refused in the audit log with the two keys' IDs, once per key), so changing the published key gets nobody a key or a secret.

Whether an edge takes a key no edge sealed is decided by its own data directory, not by anything management answers.

An edge that was there before (its data directory holds its registry snapshot from an earlier start) re-seals the older keys for one day from its first start at most, and finishes sooner once a round finds none left; it notes its state in its data directory (subject-keys-edge-sealed, payload-keys-edge-sealed), so a restart does not lengthen the day.

A new edge, or one that lost its data directory, refuses such keys from its start.

From then on an edge refuses, and logs as an error, any key no edge sealed; a key that opens with no key is passed over and counted in the log, and holds no round back.

The edge whose round finds every key sealed by an edge also records it in the cluster's database, with a MAC only a holder of the secrets key can make, and oz0 doctor warns until both kinds of keys are recorded.

If every edge lost its data directory, or the cluster's keys were sealed before it had edges apart, run oz0 server secrets-key reseal-older on one edge's machine (with its --data-dir) and restart it: it re-seals them for one more day, and opens keys sealed with the master key meanwhile, so do it while management is one you trust.

A secret may also be a reference to a secret store, which only edges run (--secret-stores, which a server without the edge role refuses). Management stores the reference, checks it against the stores the edges publish, and records the versions edges read; it never runs a store, and never reads a reference's value.

The secrets key file can hold the key wrapped by a key command, so the key that unwraps it stays in an HSM or a KMS: make it that way with oz0 server secrets-key create --command ..., or wrap an existing one with oz0 server secrets-key wrap, and start every edge with --secrets-key-command. An edge apart from management has no database, so the wrap commands run there without checking the key against the cluster; the edge reports where its keys come from through its link, and oz0 doctor shows it.