Node identity and tenant secrets

For: cluster operators and plugin authors.

Identity

• Every node has its own key, created on the machine when it joins. The private key never leaves the node, and the agent runtime never holds it: the supervisor keeps it and runs a local proxy that adds the node's certificate to outgoing calls.
• Joining is pinned. The join token, or the --ca-hash flag, carries the hash of the cluster's CA. The node checks the server's certificate chain against it before it sends anything.
• Certificates last 24 hours and renew themselves over the control stream. A node that was offline can renew a certificate that expired at most seven days ago.
• Operators have certificates too. server init creates the first operator and writes a CLI context with its certificate.
• Server certificate keys stay in memory. Each server issues itself certificates from the cluster's CAs at startup and renews them daily. The cluster's master key and the edge's optional secrets key may be supplied from files on disk; protect those files separately. Holding the master key authorizes a server to join the cluster.

Two ways in

Tokens carry a tenant, labels, an expiry and a maximum number of uses. They are bounded and never unlimited: one use and 24 hours unless asked otherwise, at most 1,000 uses and 30 days, checked when a token is made and again each time it is used. A token joins nodes only to its own tenant and cluster, and a node's ID comes from its own key. Revoking a token stops new joins; nodes that already joined stay.

A join without a token needs no credential, so what it can leave behind is bounded: at most 1,000 nodes wait for approval in a region and at most 20 of them from one address (each edge also holds one IPv6 /48 to 20 new keys), a waiting node keeps its place for 24 hours from its first ask, and each client (an IPv6 one by its /64; behind trusted proxies, the client they name) may send only a few join requests a second.

• A node's identity files are written whole. The node checks renewed credentials before it writes anything (a certificate the cluster CA signed for its own key, naming its own node, cluster and tenant), writes each file through a temporary one it renames, and never takes a new CA on renewal; what a node is, is what its certificate says.
• The CA issues only certificates that are valid. It refuses a lifetime of nothing and signs nothing once its own certificate has expired, and it loads only a CA certificate with a P-256 key. With a key behind a signing command, the signature runs outside any database transaction, and what the command prints stays in the server's log.

Cutting access

The edge authorizes every node call against its registry. In a combined server, the registry reloads when local changes are made and polls the database every two seconds. A separate edge receives registry updates through management's watch stream; it does not connect to management's database.

A node must be accepted, use its own tenant's namespace, call only the Temporal methods a worker needs and poll only its own queues. It answers only tasks handed to it by the edge.

Every task token is sealed for that node, and an answer with any other token is refused. A node therefore cannot complete, fail or heartbeat another node's task. The edge also derives sticky queues and worker names from the node's identity.

A long poll is checked again every second while it waits and when its task comes, so a node blocked while it waits gets no task.

oz0 node block reports how many servers applied it. On combined servers that can reach the database, it takes effect within two seconds: the control stream closes, Temporal calls are refused and the node stops its runtime. Work follows its recovery rules. node unblock lets the node back in; it notices within ten seconds.

A combined server cut off from the database while others still reach it fences itself, so it cannot keep serving a node blocked after the cut. Separate edges receive changes through management's watch stream and enforce registry-age and fencing limits.

oz0 node purge goes further: the node gets a command signed with the cluster CA's key on its next control stream, and deletes its whole data directory (its identity and keys, plugins, runtimes, the managed Python, its tools, caches and logs) and its binary (purge a node).

Secrets

Sealing and role separation

Secrets are sealed with AES-GCM under the cluster's master key and bound to the cluster, the tenant and the name, so a sealed value copied to another tenant does not open.

A cluster whose roles run apart can seal them instead with HPKE to a secrets key that only the edges hold, so that management stores them without being able to open them.

Only an edge, with the master key's proof, publishes that key, and an edge seals nothing to a published key it does not hold.

The data subjects' keys and payload keys an edge seals carry a tag management cannot make.

An edge apart from management takes only keys an edge sealed, decided by its own data directory rather than by anything management answers: a new edge from its start, and an edge that was there before once it re-sealed the older keys, within a day of its first start at most, unless an operator opens another day on the edge's own machine.

Secret administration

Values are never shown. The admin API seals on the way in and never returns a value; secret list shows names and the hosts each may go to. Changes are recorded in the audit log, without the value; oz0 audit reads it.

Host bindings

A secret goes only where it is bound. Every sink that sends a tenant secret to a host (a plugin source's token, a workspace export's push and pull request, a provider's key and the tokens it mints, a webhook's signature) sends it only to a host sealed with its value, which changes only with the value, and a credential only over https (plain http only to the same machine): through these sinks, neither a role that may name a secret nor a compromised management can send it elsewhere (Secrets).

A plugin's declared secrets are a grant to the plugin: its processes get their values, so install only plugins whose code you trust with the secrets they list.

Provider keys

Only the edge unseals. Provider keys are added to model calls in the edge's LLM gateway. Nodes get a local LLM proxy and a per-start token instead, and no plugin may ask for a model provider's key (the edge withholds one from a plugin's declared secrets and its tools' arguments alike), so no provider key is on a node.

Secret inputs

No secret on a command line. The CLI, the server and the node take every secret they need (a database URL with its password, the payload database's, a trace exporter's headers, a join token, a password, the identity provider's client secret) from a file, standard input or an environment variable, never as an argument, which every user of the machine reads in the process list and which shell history and supervisors keep.

A test walks every command of both binaries and fails on a flag that would take one, and another checks that the Helm chart's containers carry none in their arguments.

Support bundles

Support bundles are made here and sent nowhere. A bundle never holds the node's or the CLI's key, holds what jobs read and wrote only with --content, and replaces every secret its redactor recognises by shape or name; on a node, also every value of the tenant's secrets the node holds fingerprints of (support bundle).

Canary checks

Checked in every end-to-end run. Every end-to-end test run makes canary secrets of its own, puts one in the environment of every process it starts and, when it ends, looks for its canaries, raw and encoded, by the fingerprints below: in its logs and artifacts and, while its server still answers, the audit log and the jobs' answers.

The runs of the core paths (a job through the edge, sandbox tier 1, a backdoored model) also set canary tenant secrets, one bound to a host nothing may reach, and look in the server's data folder (its databases and stored artifacts), in the nodes' own files, in the server's and the nodes' support bundles (which must replace each by its fingerprint), in every model call the stand-in provider got and in what the stand-in services recorded.

A run that finds one fails, naming the canary and the place, never the value; an error the check itself reads is scanned too, and shown only with every canary replaced.

A test fails for an end-to-end script that does not run the check, and a script's later exit trap, in whatever form, runs after the check rather than instead of it.

It finds the run's own canaries in those places, not every secret everywhere: a value encrypted, compressed or split is not found.

Secrets in calls and requests

A secret could still leave in text a model writes. Every node therefore gets salted SHA-256 fingerprints of the tenant's secret values and their common encodings, never the values themselves.

Values that could be guessed from a fingerprint are excluded: fewer than 16 bytes, or less than an estimated 64 bits of strength. This includes combinations of words in any language or script, names, keyboard runs, sequences, interwoven patterns, dates and repeats, as found in passwords and pass phrases.

The node refuses a tool call whose arguments contain a fingerprinted value (Tool policy). It also refuses a plain-text HTTP request through a sandbox's egress proxy when the URL, headers or body contain one. Every byte is scanned. If the node has no readable fingerprint file, it refuses both kinds of request instead of passing them unscanned.

A sandboxed tool server may send its own plugin's secrets, which is what they are for.

HTTPS through the proxy is not read, and neither is a /tcp tunnel, so a value inside TLS or such a tunnel is not found; the audit log records secrets.refused for tool calls and for the platform's built-ins, and the proxy's denials say which secret.

A secret bound to the host it goes to is not given to the process: a credential binding gives the plugin's servers a placeholder instead, and the egress proxy terminates TLS for that host alone, with a CA of its own made per sandbox and constrained to the bound hosts, puts the value in the binding's header, refuses the placeholder anywhere else, refuses a switch of protocols, forwards no interim answer, and scrubs the value from the answer.

The scrub compares text, so it finds the value as sent and in its common encodings: a bound host that changes the value's form and sends it back gets it to the process, so bind only to a host you trust with it.

A model can write a {{secret:NAME}} placeholder into an argument a tool marks, which the node fills in inside the call and scrubs from its result.

With sandbox tier 1 or a container that is a boundary; on tier 0 and for sandbox: host tools it keeps the value out of the process's environment and files, but the process could still read it from the node user's other processes.

Secrets in the working directory

A harness session reads what its working directory holds, at every tier: the sandbox shows a session its own job's folders, so a .env file or a key a repository carries is readable inside them. So before a session starts, the node scans its working directory (a changeset's base, unpacked there), and after the session ends it scans it again (a repository the session cloned itself), for three kinds of file:

• .env files: .env and .env.*, the names a plugin's bundle leaves out, but not .env.example, .env.sample or .env.template, which repositories commit on purpose;
• private keys: a file whose first 64 KiB hold -----BEGIN ... PRIVATE KEY-----, the shape a plugin setting is refused for;
• the tenant's own secrets: a file whose first 64 KiB hold the exact value of one of the tenant's secrets, by the same fingerprints the egress proxy scans with. A secret the tenant never stored, or one too short or too guessable to fingerprint, is not known to it; only the first two kinds find those.

The scan is bounded: it follows no link, skips .git, node_modules, .venv and __pycache__, reads at most 64 KiB of each file and 64 MiB in all, and stops at 20,000 files, 32 folders deep or after 10 seconds, saying scan bounded: and why on the job.

What it finds is a finding, never a refusal: the job runs on, and its page, oz0 job get and the audit log (run.workspace_secret, one entry per kind and path) show how many files of each kind and where, relative to the working directory, never a value and never which secret matched.

Nothing is hidden or removed: a secret in the workspace is the tenant's to remove, from the repository or the job's input, and to rotate if it was real.

Pending joins on an exposed edge

The pending-approval limits reduce abuse, but multiple source addresses can still fill a region's 1,000 waiting places and refuse further tokenless joins. The extra IPv6 /48 count is local to each edge and resets on restart.

Use join tokens when the edge's join endpoint is reachable from networks you do not trust. Token-based joins do not use those pending places.