Use an external secret store

For: cluster and tenant administrators.

Run a store process on each edge that needs it. Management stores references while edges read values.

Secrets in your own store

A secret may live in a secret store you already run, such as OpenBao, HashiCorp Vault or a cloud's secret manager, instead of in the cluster's database. The secret is then a reference, and only an edge reads the value, through a store process the edge runs. Sealed secrets stay the default, and every consumer of a secret (the gateway's keys, plugins, connections, webhooks, the secret scan) reads a reference the same way it reads a sealed value.

oz0 secret stores --tenant acme
# STORE  EDGES  ANSWERING  VERSIONS  PREFIXES
# bao    2      2          yes       oz0/acme/

oz0 secret set ANTHROPIC_API_KEY --tenant acme --ref 'store:bao/oz0/acme/llm#key'
# Secret ANTHROPIC_API_KEY set for tenant acme as a reference to store:bao/oz0/acme/llm#key; edges read it from the store.

A reference is store:<store>/<path>[#<field>][@<version>]: the store's name, the path in it, a field of a structured secret (a KV key, a JSON member), and a version to pin. It is a name, never a credential, and holds no value.

Stores run on the edges

Each edge reads its stores from server start --secret-stores FILE (only the edge role reads it; management never runs a store or reads a reference's value, so roles apart keep holding):

stores:
  - name: bao
    command: [/usr/local/bin/my-store, --config, /etc/oz0/bao.yaml]
    env: {BAO_ADDR: "https://bao.internal:8200"}
    pass_env: [BAO_TOKEN]
    timeout: 10s
    cache_ttl: 60s
    tenants:
      "*": ["oz0/{tenant}/"]
      acme: ["shared/acme-llm"]
    hosts: [api.anthropic.com, github.com]

Process environment and configuration permissions

A store is a process the edge starts with the command argv (no shell) and keeps running. It gets only PATH, HOME, TMPDIR and the like, the env you give it, and the pass_env variables copied from the server's environment; never the server's OZ0_* variables. Credentials reach it as files or variables it reads; the server never sees them: env is for settings such as an address, never a credential (use pass_env or a file the store reads). The file must not be writable by group or others, nor readable by others when it sets any env.

Tenant path limits

A tenant may read only the path prefixes its entry (or "*"'s, where {tenant} is its ID) lists; a prefix ending in / covers what is under it. "*" must contain {tenant}, so no prefix is shared by every tenant by accident. The edge checks this on every read, against its own file, whatever the database says. Only the tenant's admins may set a reference, and only to a store and path a live edge offers that tenant (oz0 secret stores).

Destination hosts

hosts lists where the store's references may be sent by a sink that sends a secret out (a plugin source's token, a provider's key), as --host does for a sealed secret and in its forms (a wildcard over a public suffix is refused when the edge reads the file); without it, its references go only where a built-in provider's key goes. Only this file sets them, so management cannot widen them. An exported reference (secret export-to-sealed) keeps its store's hosts, sealed with the value.

Protocol limits

The protocol, oz0-secret-store/1, is JSON lines on the process's stdin and stdout with three operations: get, versions and health. Values are at most 64 KB, a line 256 KB, each call bounded by timeout (1 to 60 s); a store that misbehaves is restarted, after 1 s and then up to a minute. What it writes to stderr goes to the edge's log, cut short, 20 lines a minute: a store must not write values there.

Cache and failures

A value read is kept in the edge's memory for cache_ttl (5 s to 1 hour), never on disk or in the database. A read that fails is a missing secret, as an unset one is, never the value. oz0 secret list shows FAILING: ... with the store's not_found, denied or invalid (such as not_found: ...) or the edge's refusal.

Anything else shows only as not available on that edge: a store's unavailable (an address that does not answer, a credential file with the wrong mode, a cloud account's refusal), a process that does not run or answer. That detail is the operator's, and it is in the edge's log. oz0 secret stores says the same of a store's health.

Implement a store

Any program that speaks the protocol is a store: pkg/secretstore in the source has the messages and Serve, which makes a Go program a store from three methods; scripts/fakesecretstore is the test store the end-to-end tests use. A store's not_found, denied and invalid messages reach the tenant's admins, so they name only the path, field and version asked for; Serve answers a backend's plain Go error as unavailable with a fixed message and writes its text to stderr, which reaches the edge's log.

Rotation in the store

Every edge reads each reference of its stores again once a cache_ttl. When the store reports a new version, the edge records it as a new version of the secret, set by store:<name>: plugins that read it restart with the new value, as after a secret set, and secret list and the Secrets tab show the store's version, its time and its expiry, if it has one (reminders and the overdue finding follow it).

The first read of a new reference counts as a change only when the store changed the value after the reference was set, or does not say when. A store without versions gets one from a keyed hash of the value, which management cannot compute. Recording what an edge read is audited as secret.store.changed, with both versions and no value.

• Rollback of a secret whose store changed it pins the store's previous version (...@<version>), as long as the store keeps it; a second rollback unpins it. A reference you set over a sealed value (or the reverse) rolls back within 24 hours as any set does.
• With management away, edges keep reading from their stores; a store's new version reaches nodes when management is back.

Taking a store away

oz0 secret export-to-sealed has an edge read each of the tenant's references from its store and replace it with the value, sealed as any secret is, if the secret still holds that reference. It works for any store:

oz0 secret export-to-sealed --tenant acme --dry-run   # read each one, change nothing
oz0 secret export-to-sealed --tenant acme             # or --store bao for one store's
# SECRET             REFERENCE                       STATUS    VERSION  DETAIL
# ANTHROPIC_API_KEY  store:bao/oz0/acme/llm#key      exported  9

Each export is audited (secret.exported, with the reference, never the value), and the reference is kept for 24 hours, so secret rollback brings it back. The CLI's context must reach an edge that runs the store; the tenant's admins may, as people or with an operator certificate, never with an app's key. Once nothing references a store, remove it from the edges' files.