Deliver and scan plugin secrets

For: cluster and tenant administrators.

Declare the secrets each plugin needs and review file-descriptor and scan behaviour by platform.

Secrets for plugins

A plugin lists the tenant secrets its processes need in its manifest:

requires:
  secrets: [JEV_API_KEY]

oz0 secret set JEV_API_KEY --from-file ~/.secrets/jev-api-key

When the edge sends a node its plugins, it unseals the secrets each one declares and sends them along, over the node's mTLS connection, to the nodes that run that plugin and to no others. The node keeps them in memory, never on disk, and starts the plugin's tools, hooks and harnesses with each one as an environment variable of the same name. Other plugins' processes do not get them.

• A missing secret: plugin install warns, and nodes report the plugin failed with the command that sets it, keeping a previous version running if they had one.
• Rotation: after secret set or secret rollback, every node that runs a plugin declaring the secret gets the new value within seconds and restarts its runtime, as for a new version. Running jobs carry on.
• Trust: a secret is in the environment of the plugin's processes on every node that runs it. Install plugins that declare secrets as carefully as the secrets themselves, or bind the secret to the host it goes to: with a credential binding the processes hold a placeholder, and the node's egress proxy puts the value on requests to that host only.

Secrets on a file descriptor

An environment variable stays readable in /proc/<pid>/environ by every process of the node's user for the process's whole life, even after the process removes it. A plugin whose servers can read a file descriptor instead asks for one:

requires:
  secrets: [JEV_API_KEY]
  secrets_from: fd

from oz0 import secrets  # pip install oz0-client: the module needs only the standard library

secrets.load()  # first, before the server starts other processes
key = secrets.require("JEV_API_KEY")

The node starts each server through its own host-exec, which hands it one sealed in-memory file of JSON ({"JEV_API_KEY": "..."}) on an inherited descriptor named by OZ0_SECRETS_FD, and no secret in its environment.

load() reads it once, closes it and forgets the variable; load(harden=True) also makes the process undumpable, so that no other process of the user can read its memory. A plugin in another language reads the descriptor the same way.

This works on every Linux and macOS node, sandboxed or not; a Windows node has no descriptor to hand, so there the server gets its secrets in its environment, which load() falls back to.

plugin lint warns when a plugin asks for ANTHROPIC_API_KEY or OPENAI_API_KEY: with the gateway's key, a plugin could call the provider directly, unmetered and outside every budget. Agents and harnesses reach models through the gateway without it.

What is scanned

The edge sends each node salted fingerprints of the tenant's secrets that pass its guessability check, in their original form and encoded as base64, hex and percent-encoding. It never sends the values in this set. Values shorter than 16 bytes and values built from predictable words, dates, keyboard runs or other password patterns are omitted because someone holding the fingerprints could recover them by guessing. Setting such a secret reports that the nodes will not scan for it.

A node refuses a tool call whose arguments carry one, and the egress proxy of a sandboxed process refuses a plain-text request that carries one, but for the process's own plugin's secrets. Every secret that passes this check is scanned for, whichever plugin reads it. Settings are not scanned because they are not secrets.

Every byte is scanned: a tool call's arguments up to 16 MiB of text (a call with more is not made), and a plain-text request's whole body, which the proxy scans as it forwards it and cuts off before any byte of a secret leaves. While the fingerprints have not reached a node, or its copy cannot be read, neither scan can be done, so neither is skipped: the node makes no tool call whose arguments hold text, and the proxy sends no plain-text request, each with a reason that says the fingerprints are missing or unreadable.