--- title: "Use an external secret store" description: "Run a store process on each edge that needs it. Management stores references while edges read values." canonical_url: "https://docs.oz0.ai/admin/secrets/external-stores" --- # Use an external secret store > Run a store process on each edge that needs it. Management stores references while edges read values. For: cluster and tenant administrators. Run a store process on each edge that needs it. Management stores references while edges read values. ## Secrets in your own store A secret may live in a secret store you already run, such as OpenBao, HashiCorp Vault or a cloud's secret manager, instead of in the cluster's database. The secret is then a reference, and only an edge reads the value, through a store process the edge runs. Sealed secrets stay the default, and every consumer of a secret (the gateway's keys, plugins, connections, webhooks, the secret scan) reads a reference the same way it reads a sealed value. ```bash [Terminal] oz0 secret stores --tenant acme # STORE EDGES ANSWERING VERSIONS PREFIXES # bao 2 2 yes oz0/acme/ oz0 secret set ANTHROPIC_API_KEY --tenant acme --ref 'store:bao/oz0/acme/llm#key' # Secret ANTHROPIC_API_KEY set for tenant acme as a reference to store:bao/oz0/acme/llm#key; edges read it from the store. ``` A reference is `store:/[#][@]`: the store's name, the path in it, a field of a structured secret (a KV key, a JSON member), and a version to pin. It is a name, never a credential, and holds no value. ### Stores run on the edges Each edge reads its stores from `server start --secret-stores FILE` (only the edge role reads it; management never runs a store or reads a reference's value, so [roles apart](https://docs.oz0.ai/fleet/separate-roles) keep holding): ```yaml [secret-stores.yaml] stores: - name: bao command: [/usr/local/bin/my-store, --config, /etc/oz0/bao.yaml] env: {BAO_ADDR: "https://bao.internal:8200"} pass_env: [BAO_TOKEN] timeout: 10s cache_ttl: 60s tenants: "*": ["oz0/{tenant}/"] acme: ["shared/acme-llm"] hosts: [api.anthropic.com, github.com] ``` #### Process environment and configuration permissions A store is a process the edge starts with the `command` argv (no shell) and keeps running. It gets only `PATH`, `HOME`, `TMPDIR` and the like, the `env` you give it, and the `pass_env` variables copied from the server's environment; never the server's `OZ0_*` variables. Credentials reach it as files or variables it reads; the server never sees them: `env` is for settings such as an address, never a credential (use `pass_env` or a file the store reads). The file must not be writable by group or others, nor readable by others when it sets any `env`. #### Tenant path limits A tenant may read only the path prefixes its entry (or `"*"`'s, where `{tenant}` is its ID) lists; a prefix ending in `/` covers what is under it. `"*"` must contain `{tenant}`, so no prefix is shared by every tenant by accident. The edge checks this on every read, against its own file, whatever the database says. Only the tenant's admins may set a reference, and only to a store and path a live edge offers that tenant (`oz0 secret stores`). #### Destination hosts `hosts` lists where the store's references may be sent by a sink that sends a secret out (a plugin source's token, a provider's key), as `--host` does for a sealed secret and in its forms (a wildcard over a public suffix is refused when the edge reads the file); without it, its references go only where a built-in provider's key goes. Only this file sets them, so management cannot widen them. An exported reference (`secret export-to-sealed`) keeps its store's hosts, sealed with the value. #### Protocol limits The protocol, `oz0-secret-store/1`, is JSON lines on the process's stdin and stdout with three operations: `get`, `versions` and `health`. Values are at most 64 KB, a line 256 KB, each call bounded by `timeout` (1 to 60 s); a store that misbehaves is restarted, after 1 s and then up to a minute. What it writes to stderr goes to the edge's log, cut short, 20 lines a minute: a store must not write values there. #### Cache and failures A value read is kept in the edge's memory for `cache_ttl` (5 s to 1 hour), never on disk or in the database. A read that fails is a missing secret, as an unset one is, never the value. `oz0 secret list` shows `FAILING: ...` with the store's `not_found`, `denied` or `invalid` (such as `not_found: ...`) or the edge's refusal. Anything else shows only as not available on that edge: a store's `unavailable` (an address that does not answer, a credential file with the wrong mode, a cloud account's refusal), a process that does not run or answer. That detail is the operator's, and it is in the edge's log. `oz0 secret stores` says the same of a store's health. #### Implement a store Any program that speaks the protocol is a store: `pkg/secretstore` in the source has the messages and `Serve`, which makes a Go program a store from three methods; `scripts/fakesecretstore` is the test store the end-to-end tests use. A store's `not_found`, `denied` and `invalid` messages reach the tenant's admins, so they name only the path, field and version asked for; `Serve` answers a backend's plain Go error as `unavailable` with a fixed message and writes its text to stderr, which reaches the edge's log. ### Rotation in the store Every edge reads each reference of its stores again once a `cache_ttl`. When the store reports a new version, the edge records it as a new version of the secret, set by `store:`: plugins that read it restart with the new value, as after a `secret set`, and `secret list` and the Secrets tab show the store's version, its time and its expiry, if it has one (reminders and the overdue finding follow it). The first read of a new reference counts as a change only when the store changed the value after the reference was set, or does not say when. A store without versions gets one from a keyed hash of the value, which management cannot compute. Recording what an edge read is audited as `secret.store.changed`, with both versions and no value. - Rollback of a secret whose store changed it pins the store's previous version (`...@`), as long as the store keeps it; a second rollback unpins it. A reference you set over a sealed value (or the reverse) rolls back within 24 hours as any set does. - With management away, edges keep reading from their stores; a store's new version reaches nodes when management is back. ### Taking a store away `oz0 secret export-to-sealed` has an edge read each of the tenant's references from its store and replace it with the value, sealed as any secret is, if the secret still holds that reference. It works for any store: ```bash [Terminal] oz0 secret export-to-sealed --tenant acme --dry-run # read each one, change nothing oz0 secret export-to-sealed --tenant acme # or --store bao for one store's # SECRET REFERENCE STATUS VERSION DETAIL # ANTHROPIC_API_KEY store:bao/oz0/acme/llm#key exported 9 ``` Each export is audited (`secret.exported`, with the reference, never the value), and the reference is kept for 24 hours, so `secret rollback` brings it back. The CLI's context must reach an edge that runs the store; the tenant's admins may, as people or with an operator certificate, never with an app's key. Once nothing references a store, remove it from the edges' files. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.