Key commands

For: cluster and tenant administrators.

Keep an offline recovery copy before changing how the servers unwrap their keys. The protocol below lets an HSM, KMS or secret engine protect keys at rest.

Every server reads the master key from a file, and every server that runs the edge role reads the secrets key from another (Run the roles separately). By default these files hold the keys in plain base64, so whoever copies a server's disk, its Kubernetes Secret or a copy of the file holds the cluster.

A key command changes that. The file then holds the key wrapped: encrypted under a wrapping key that lives somewhere else, such as an HSM, a cloud KMS or a secrets engine. At start, the server asks the command to unwrap it. A stolen file is useless without the wrapping key, and each unwrap is a request your HSM or KMS can authorize, log and refuse.

What it does not change: the server still holds the unwrapped key in memory while it runs, as it always did. A key command protects the key at rest, not in use.

How a server runs the command

Give the command with --master-key-command (or OZ0_MASTER_KEY_COMMAND) and --secrets-key-command (or OZ0_SECRETS_KEY_COMMAND). The server runs it:

• without a shell, with a small environment of its own: PATH, HOME, USER, LOGNAME, the locale (LANG, LC_ALL, LC_CTYPE), TZ, TMPDIR, SSL_CERT_FILE and SSL_CERT_DIR (and on Windows what a program needs to run there), plus the variables you name in OZ0_KEY_COMMAND_ENV, a comma-separated list of names or prefixes ending in *, such as OZ0_KEY_COMMAND_ENV=AWS_PROFILE,AWS_REGION or VAULT_*. That is how a PIN, a token or cloud credentials reach it; the rest of the server's environment, its database URL among it, does not. The CA's signing command gets the same environment;
• for at most 30 seconds per run; after that it is killed, on Linux and macOS together with any processes it started;
• once per key at every start, and once every 10 minutes to ask whether it still reaches its wrapping key.

Every command that reads a key goes through it: server start in every mode and role, server init, server restore, server operator issue, server provisioner issue, server ca export-key and forget-key, and master-key confirm-backup and secrets-key confirm-backup. Each takes the same flag.

A server refuses to start when:

• the key file, plain or wrapped, can be read or written by other users, or written by its group (make it chmod 600, or 640 for a group only the server is in, as a Kubernetes Secret mounted with fsGroup is);
• the file is wrapped and no command is given (the message names the command that wrapped it);
• a command is given and the file is plain (wrap it first, or leave the flag out);
• the command fails or takes too long (the message says so; what the command wrote to stderr, cut short, is in the server's log, never in the message, which doctor and the database keep);
• the command answers with another key than the one the file names.

The protocol

The command gets one argument, the verb, and reads a JSON document on stdin. Each document names the protocol, the verb, the kind of key (master-key or secrets-key) and the key's ID.

{"protocol": "oz0-keycmd-v1", "verb": "unwrap", "kind": "master-key", "key_id": "mk-4f1c2a9b0e7d3365",
 "wrapped": {"key": "projects/acme/keys/oz0-master", "ciphertext": "..."}}

• unwrap runs at every start. Print the key and nothing else on stdout; never print it on stderr, which goes to the server's log.
• wrap runs only when you move a key under the command, rotate its wrapping key, or make a new key with server init --master-key-command or secrets-key create --command. What the document holds is up to the command: the server stores it as it is and hands it back to unwrap.
• describe says what holds the wrapping key, such as a KMS key's name or a token's label, and what wraps with it, such as AES-256-GCM. Never a secret. Exit non-zero when the wrapping key cannot be reached: doctor shows it.

Bind each wrapped key to its kind and ID, for example as a KMS's encryption context or as AES-GCM's associated data, so that a master key's document can never be unwrapped as the secrets key. The server checks the key's ID after every unwrap anyway.

Writing your own

A key command is a few lines around your HSM's or KMS's tool or SDK. A sketch in Python, where encrypt and decrypt stand for the calls your wrapping key's holder offers:

#!/usr/bin/env python3
import base64, json, sys


def encrypt(plaintext: bytes, context: bytes) -> dict: ...  # your HSM or KMS
def decrypt(document: dict, context: bytes) -> bytes: ...


req = json.load(sys.stdin)
if req["protocol"] != "oz0-keycmd-v1":
    sys.exit("unknown protocol")
context = f"{req['kind']}/{req.get('key_id', '')}".encode()
verb = sys.argv[1]
if verb == "describe":
    print(json.dumps({"holder": "my-kms key oz0-root", "algorithm": "AES-256-GCM"}))
elif verb == "wrap":
    print(json.dumps(encrypt(base64.b64decode(req["key"]), context)))
elif verb == "unwrap":
    print(base64.b64encode(decrypt(req["wrapped"], context)).decode())
else:
    sys.exit(f"unknown verb {verb}")

scripts/fakekeycmd in the repository is a complete one for tests: it wraps with AES-256-GCM under a key in a plain file, which is fine for a test and pointless for anything else.

The key file

A wrapped key file is a JSON document with a header and the command's document:

{
  "format": "oz0-wrapped-key-v1",
  "kind": "master-key",
  "key_id": "mk-4f1c2a9b0e7d3365",
  "command": "/usr/local/bin/oz0-key-command",
  "holder": "my-kms key oz0-root",
  "algorithm": "AES-256-GCM",
  "wrapped_at": "2026-10-09T12:00:00Z",
  "wrapped": {"...": "what the command printed"}
}

The key ID names the key without revealing it. command says which command wrapped the key; the server never runs a command named in a file, only the one you give it.

Move a cluster's keys under a command

On a server's machine, with the key file the servers use:

# 1. Wrap the key. The plain file is kept as master.key.plain, and the round trip is checked first.
oz0 server master-key wrap --command /usr/local/bin/oz0-key-command \
  --master-key-file /etc/orchestrator-zero/master.key
# 2. Copy the wrapped file to every server, and restart each with the command.
oz0 server start --master-key-file /etc/orchestrator-zero/master.key \
  --master-key-command /usr/local/bin/oz0-key-command ...
# 3. Once they run, shred the plain copy. You are asked again for the offline copy.
oz0 server master-key wrap --forget-plain --confirm-backup --command /usr/local/bin/oz0-key-command \
  --master-key-file /etc/orchestrator-zero/master.key --note "safe 2, envelope 7"

The secrets key moves the same way with oz0 server secrets-key wrap --secrets-key-file ... and --secrets-key-command on the edges.

With the database (OZ0_DB, --db-file or --dev) each step checks the key against the cluster (the master key must unseal it; the secrets key must be the one the cluster seals to) and writes an audit entry: key.wrapped, key.plain_forgotten. An edge apart from management has no database: there the commands run without it, and say so.

Until step 3, every server that starts through the command finds the plain copy beside its key file, says so in its log, and oz0 doctor's key source check warns with the step that shreds it.

Overwriting a file does not erase it everywhere: on SSDs and copy-on-write file systems the old blocks can survive. Treat a machine that held the plain file as having held the key.

A new cluster can start wrapped, so the key is never on disk in the clear:

oz0 server init --master-key-out /etc/orchestrator-zero/master.key --master-key-command /usr/local/bin/oz0-key-command ...
oz0 server secrets-key create --out /etc/orchestrator-zero/secrets.key --command /usr/local/bin/oz0-key-command

Rotate the wrapping key

A new KMS key, a new key version, a new alias, new credentials or a new endpoint: rewrap.

oz0 server master-key wrap --rewrap --command /usr/local/bin/oz0-key-command \
  --master-key-file /etc/orchestrator-zero/master.key

It unwraps through --from-command (by default --command), wraps the same key again, checks the round trip, keeps the old document as master.key.prev and writes the new one in place (key.rewrapped in the audit log). The key itself does not change, so nothing sealed under it changes, and your backup confirmations stay valid. Copy the new file to every server; servers that still have the old one keep working as long as the old wrapping key can unwrap it.

The way out

unwrap-to-file moves a key back to a plain file, so the command, and whatever holds its wrapping key, can be taken away:

oz0 server master-key unwrap-to-file --command /usr/local/bin/oz0-key-command \
  --master-key-file /etc/orchestrator-zero/master.key
oz0 server secrets-key unwrap-to-file --command /usr/local/bin/oz0-key-command \
  --secrets-key-file /etc/orchestrator-zero/secrets.key

It unwraps once more and writes the plain key, mode 0600, in place of the wrapped file, which is kept as master.key.wrapped; or at --out, which must not exist. Then start the servers without the command. It needs only unwrap from the command.

Keep an offline copy

The offline copy of the master key is still your last way back: if the wrapping key is lost, a wrapped file cannot be opened. If you made the key under a command and never had it in plain, write one copy to offline media with unwrap-to-file --out, then record it with oz0 server master-key confirm-backup.

Kubernetes

The chart takes masterKey.command and, with roles.separate, roles.edge.secretsKey.command. The Secrets then hold the wrapped files, and the command must be in the server image or on a volume, like caSigner.command.

Doctor

oz0 doctor has two checks:

• master key source: which running servers read the master key from a plain file and which through a key command, with what the command says holds its wrapping key;
• secrets key source: the same for the edges' secrets key.

Each warns when a server's last unwrap failed (recorded when the server could reach the database) or when its command could not reach the wrapping key at its last check. A server whose command stops answering keeps running on the key it unwrapped, but will not start again until the command answers. The fix says how to move the key back to a plain file.

Next

• Run a cluster: the cluster CA's signing command, the same idea for the CA's key
• Upgrades, backups and restores
• Doctor