--- title: "Trust boundaries and compromise" description: "Trust boundaries and compromise for cluster operators and plugin authors." canonical_url: "https://docs.oz0.ai/concepts/threat-model/trust-boundaries" --- # Trust boundaries and compromise > Trust boundaries and compromise for cluster operators and plugin authors. For: cluster operators and plugin authors. ## The parts ```text people, apps, Temporal's UIs outside senders (webhooks, events) the hub, Git forges, registries │ session, API key, operator cert │ a secret or signature │ signed index, pinned digests ▼ ▼ ▼ ┌──────────────┐ internal API (server cert + MAC) ┌──────────────┐ gRPC ┌──────────────────┐ │ management │◄──────────────────────────────────►│ edges │◄──────►│ Temporal backend │ │ master key │ registry watch stream │ master key │ │ embedded, own, │ │ CA, accounts │ │ secrets key │ │ or Temporal Cloud│ └──────────────┘ │ provider keys│ └──────────────────┘ └──────┬───────┘ │ node certificate (24 h), one tenant ┌──────▼───────────────────────────────────────┐ │ node: supervisor, runtime, plugins, sessions │ │ tier 1: sandbox and egress proxy per session │ └──────────────────────────────────────────────┘ ``` By default, management and edge run in one `orchestrator-zero` process. The same binary can [run these roles in separate processes](https://docs.oz0.ai/fleet/separate-roles). Put those processes on separate machines when you need the host boundary described here. ## What each compromise gives
Who is taken What they can do What limits it
A person's session, or their password and second factor Details Details
A role that may name a secret without reading it ( plugins.install , workspace.write , models.manage , notifications.manage , or a scoped key with one) Details Details
A tenant API key Its role in its one tenant, on the runtime API and the operators' MCP endpoint; never the admin API Keys expire after 90 days unless made otherwise; revoking one takes effect within 2 s; its last use is recorded
An MCP client's access token Its person's own permissions in one tenant, on that tenant's MCP endpoint alone, for at most 10 minutes: listing and starting the agents they may start, and answering what they may answer Its only audience is the tenant's /mcp ; it is an HMAC under a key of the tenant's, checked with the grant, the account and the membership on every call; revoking the grant refuses it within 2 s, and what acts later for a job the client started (a long-waiting approval's re-check, Ask's hand-over) acts through the grant as it stands then, never with a cluster role
An MCP client's refresh token New access tokens, until the grant ends (at most 90 days) Bound to its client; rotated at each use, and a rotated one used again revokes the grant; stored only as a hash; the person or an admin of the tenant revokes it, narrowing the tenant's client allowlist revokes the grants of the clients it no longer allows, and removing the person's account or role in the tenant revokes theirs
A program on the person's computer listening on a loopback port The authorization code meant for a loopback client the person approves, which it cannot redeem without the client's PKCE verifier PKCE S256 only; codes last a minute and work once (a second use revokes the grant); the consent page warns that the answer goes to the person's own computer; a tenant can turn loopback clients off and list named clients instead
An operator certificate Everything: it is a cluster admin 90 days; server operator disable refuses it on every route that takes one: at once on the management port (the admin API, the console's sign-in, Temporal's UI and its codec) and within seconds on every edge
A provisioner's certificate Make, disable and remove the accounts of its identity providers' people; give them roles through the group rules a cluster admin wrote, and directly only in the tenants that allowed it, up to the role each allowed (owner only where one of the tenant's owners allowed it) 90 days; only the account calls of the admin API take it; server provisioner disable refuses it at once; a tenant's owners see and withdraw its allowance ( tenant provisioner ); every call is in the audit log
Management, apart from the edges Details Details
An edge Everything: it holds the master key, the secrets key and the provider keys, and talks to Temporal Nothing inside the cluster: treat edges as its most sensitive machines, and run as few as you need
A server certificate without the master key Details Details
The Temporal backend Read job histories and send tasks to nodes' queues Tenant payload encryption limits forgery and content access. See encryption scope and key rotation .
A join token Join nodes to its one tenant of its one cluster, as many as it has uses left, until it expires Bounded and never unlimited: one use and 24 hours by default, at most 1,000 uses and 30 days, checked again at each use (an older unbounded token is refused); a node's ID comes from its own key, so a token never yields another node's identity; revoking it stops new joins
Anyone who reaches an edge's join port without a token Ask to join as a node waiting for approval, which gets no certificate until an operator accepts it Pending joins have per-region, address and time limits. See join limits and exhaustion .
A node Details Details
A plugin's process Details Details
A harness session, such as an agent with a shell At tier 1, its own job's directory and the hosts the tenant, its plugin and its agent allow; a private network only where the tenant's list allows it Shell and write sessions require tier 1 unless a tenant admin permits tier 0; outside-started jobs always require tier 1. See session boundaries .
A remote agent's provider session What its built-in tools may do, each only after the job's yes The provider is set to ask before every built-in call, and gets no MCP server; our tools run on our nodes through the job's tool path; its keys reach the adapter in the session request, not its environment, and the adapter is not dumpable
A model, steered by text it read Details Details
An identity token lifted from a tool call Details Details
A plugin index, or a hub listing Offer a version An index must verify against root keys an admin trusted, is versioned and expires; a version installs from a source pinned to its content digest, and OCI sources by digest only
A release folder copied in, such as an offline folder or a dist folder, or whoever can write it Details Details
A tenant admin who may manage connections but not secrets Details Details
### Compromised Temporal backend The Temporal backend can read histories and send tasks to node queues. Tenant payload encryption is on by default in a cluster unless `--encrypt-payloads=false`, opt-in with `--encrypt-payloads` in dev mode, and always on Temporal Cloud. Temporal still reads search attributes and headers. The edge proxy refuses unencrypted node payloads in other fields. It also refuses payloads marked as Temporal system payloads in any field of any call. A compromised backend can replay an encrypted payload but cannot create a new one. The edge retires each payload key at seven days, or on its first read after management's database becomes reachable again. This is well below the capacity allowed by random nonces. ### Unauthenticated join requests Without a token, someone reaching an edge's join port can ask to join and wait for approval. No certificate is issued until an operator accepts the node. At most 1,000 nodes wait per region and 20 per address. A place lasts 24 hours from the first request. Each client may send a few join requests per second; IPv6 clients count by `/64`, and trusted proxies identify their clients. Each edge also limits an entire IPv6 `/48` to 20 pending keys it has seen. Fifty addresses on different networks can still fill a region's list. Use token-based joins when that is a concern. ### Harness session isolation Sessions with shell or write tools run at tier 1 regardless of who starts them. A tenant admin may opt a node out and allow tier 0, which the job displays. Jobs started by outside input always require tier 1 and receive no plugin secrets. The session's LLM proxy key reaches only model routes and charges its own job. Hooks and approvals see calls that pass through the platform's tool path or the harness's permission hook. Native actions a harness does not report are outside those checks; see [harness limits](https://docs.oz0.ai/concepts/threat-model/harness-tools). The supported session channels have their own [isolation controls and limits](https://docs.oz0.ai/concepts/threat-model/harness-tools#a-sessions-tools-and-its-harness). ### Compromised personal credentials #### What they can do What their roles allow, in the tenants they are a member of #### What limits it Tenant roles and the content permission limit access. The tenant's [label-policy walls](https://docs.oz0.ai/govern/taint-and-labels/label-policies#walls) apply per job on every path that shows content. Sessions expire, and an admin can end them with `user sessions end`. A session alone cannot change the account's sign-in: changing or removing its password, or setting up a second factor, requires the current password and any existing second factor. Every change is audited. Every server counts password and code guesses before checking them. Password guesses count per client and per name; behind a reverse proxy, the client it names counts once the server trusts it through `--trusted-proxy`. Code guesses count per account. ### A role that may name a secret without reading it (`plugins.install`, `workspace.write` details #### What they can do Use a tenant secret where it is meant to go: a plugin source's token for its Git host or registry, a provider's key at its provider, a webhook's signing key for its webhook #### What limits it The platform's own sinks (a plugin source's token, a workspace export's push and pull request, a provider's key and the tokens it mints, a webhook's signature) send a secret only to the hosts sealed with its value, which only a holder of `secrets.write` sets, with the value: through them such a role cannot send a secret to a host of its own. Nor does it learn which secrets the tenant has: without `secrets.read`, every refusal, warning, delivery record, output error and webhook answer reads the same for a secret that is missing, of the other use or bound elsewhere. What it does learn is that a secret works where it is bound, by using it there. A plugin's declared secrets are not covered: a plugin that lists a secret in `requires.secrets` gets its value in its processes, so installing it hands that secret to the plugin's code; the install screen names every secret a plugin asks for, and no plugin may ask for a model provider's key ### Compromised management #### What they can do The master key opens the CA's key, allowing management to issue any certificate and sign forwarded calls as any person. It can ask an edge to build a plugin with a tenant's Git token, but the edge sends the token only to a bound host. A compromised management cannot redirect a secret through the platform's sinks: the hosts are sealed with the value, which management cannot open. It can rebind a secret only by replacing it with a value it chose. A value set by an edge, such as a connection's refreshed or saved token, keeps only hosts confirmed by an edge, never hosts supplied by management. Management can still install a plugin that asks for a secret, giving its processes that value. Management also holds every account and the audit log. #### What limits it It cannot open tenant secrets, data subjects' keys or (with the secrets key) payload keys: they are sealed to a key only edges hold, and an edge seals nothing to a published key it does not hold, and audits the refusal (`secrets.key.refused`). Whether an edge takes a data subject's key or a payload key no edge sealed is decided on the edge, by its own data directory, never by anything management serves: a new edge, or one that lost its data directory, refuses such keys from its start, and an edge that was there before re-seals the older ones for one day from its first start at most, then refuses them too. So management cannot place a key it knows, except within that day after an upgrade, or after an operator opened another day on an edge's own machine. It never dials Temporal, so it reads no job content directly. With [`--ca-signer-command`](https://docs.oz0.ai/fleet/cluster), the CA's key is outside it ### Compromised server certificate #### What they can do Through management's internal API, read and write the tables edges may (nodes, usage, quotas, jobs' records, sealed values, API key hashes, accounts without their password hashes), and send the buffered writes every edge sends for the cluster (metering, audit entries). With those writes it can still replace a tenant secret's sealed value with one of its own, or destroy sealed values, data subjects' keys and payload keys. It can change a node's row, its state and tenant among them, since edges write it for joins and renewals: so it can accept a pending node into a tenant, whose next join then gets that tenant's certificate. And it can report for a node whose session it claims #### What limits it PostgreSQL's grants to the edges' own role; no sign-in data, sessions, password hashes, second factors or schema. It cannot open a sealed value or redirect one (a secret goes only to the hosts sealed with it), set or replace the key tenant secrets are sealed to, publish a signing key or report a key source for a server (each needs a proof of the master key), nor plant a data subject's key or a payload key an edge would take (edges take only keys sealed with a tag of the secrets key, apart from the older keys an edge re-seals within its one day after an upgrade). A node's buffered reports are applied only for the edge that last claimed its session, and change only its facts. A server's row in the servers table counts only with its proof of the master key, so a certificate cannot draw management's calls, artifacts or nodes to an address of its own, and management takes a call's answer only from the server the row names. It cannot answer an edge as management, or as the temporal role: their certificates carry names the CA gives only to them. It cannot have an edge act for a person or an operator, or build with a tenant's token: management signs each such call, whole, for one edge, once, and the edge checks the signature before it reads the call's body. It cannot bootstrap a new edge ### Compromised node #### What they can do Its tenant's work on the queues it serves: it can answer tasks falsely, read its tenant's payloads, artifacts, payload keys and data subjects' keys, and spend its tenant's model budget #### What limits it It acts only in its own tenant's namespace, polls only its assigned queues and calls only worker methods; it answers, heartbeats and shuts down only the tasks and queues the edge handed it, its workflows signal and cancel only the jobs they started and start no Nexus operation, and its activities publish only to their own job's live stream; its certificate lasts 24 hours and blocking it stops it within 2 s. It holds no provider key and reaches no admin or runtime API. An event source's answer counts only where the sender's signed delivery, checked by the edge, agrees with its bindings. A compromised node cannot: - Act as a chat person who sent or clicked nothing, or treat a message not addressed to the app as one that was. - Move a message to another thread or change its text or submitted form values. - Decide a form from a click the person did not make. - Fire a routine past `allow_senders`. - Produce more events than the delivery contains, or combine values from different events. - Wake another job's wait with an event absent from the delivery. Of a delivery whose signature only its plugin checks, nothing counts as verified ### Compromised plugin process #### What they can do At tier 0, the node's user. At tier 1, a job's tool server has its job's directories, and the network only through the egress proxy's allowlist; an installed plugin's own servers run in a sandbox of their own held to the same allowlist, unless the node's operator chose `--plugin-sandbox auto`, which keeps the host's network, or exempted the plugin, whose `sandbox: host` or `sandbox: own` tool then runs as the node's user #### What limits it Plugins are installed by tenant admins, pinned to a commit or digest, and gated by their evals; each process gets only its plugin's declared secrets and a minimal environment, and what its manifest sets in that environment reaches neither the runner of its sandbox nor a port of the node's. On a strict node none of its code runs at install: an exempted plugin's build, and every plugin's on `auto`, `off` or tier 0, runs as the node's user. At tier 1 its plugin's own entries reach public addresses only, even through a name whose DNS its author controls: only the tenant's list reaches the company's private networks ### A model steered by outside text #### What they can do What the agent's tools and outputs allow #### What limits it The controls include approvals, hooks, contracts, budgets, delegation limits and kill switches. Routine outputs are checked against their schema, redacted and stripped of mentions and links to other hosts. Outside text reaches a model labelled as data. [Taint](https://docs.oz0.ai/govern/taint-and-labels) follows what a job reads through children, colleagues, boards and files. Calls that can send data out wait for a person once the job is tainted, except a harness session's own built-ins. The edge takes the runtime's word for labels. `evidence verify` checks recorded tool-call labels against each other, the job's start, joins recorded by the runtime and its children. A quarantine child returns only a bounded shape of at most 4 KiB. [Label policies](https://docs.oz0.ai/govern/taint-and-labels/label-policies#label-policies) withhold tool results above an agent's clearance from its model, although the plugin already read them. At the egress proxy they narrow session, terminal and job-sandbox access by the job's level. They refuse model providers by level at the gateway and before a remote session reaches its provider; the node's word can only raise those labels. Walls separate compartments for a job's tool results and for people on every content path: steps, chats, feeds, live streams, approvals, exports and private audit parts. Overflowed compartments count as every compartment. ### Stolen tool identity token #### What they can do Act as the agent (or, for a delegated token, as the person with the agent) at the one service its audience names, until it expires #### What limits it It lives at most five minutes and names one audience, the node and the call; a node gets one only for a call the edge handed it, with that call's binding (until the binding expires, at most the tool's time limit plus a minute, even after the call returned), and a revoked or parked job gets none. Only the agent's own plugin's connections get a token that names the agent: another plugin's tool the agent was granted gets none. The secrets key derives every tenant's issuer keys: a leak of it is a leak of every tenant's outside trust, as it already is of every stored key; replacing it, or an emergency rotation of one tenant's issuer, cuts the old keys off ### Untrusted release folders #### What they can do Offer node binaries, runtimes and a release's order to new nodes #### What limits it An edge serves a release only when its manifest verifies against the release keys built into the server, matches its flavour and belongs to the cluster's channel. It must be no older than a release the edge previously served, whose record survives restarts, or the version management told nodes to run. A node's own reported version does not set that boundary. A cluster admin can authorize one step back to a named version. Each file is served only with the digest in the signed manifest. `dist pull` refuses another channel or an older release without a reason, and nodes verify the signature again before installing. ### Connection administration without secret access #### What they can do Point a connection's params, such as an API's address, somewhere else #### What limits it The addresses a connection's credentials go to are pinned when it is connected: the edge signs, mints, refreshes and hands out a token only while they are as pinned; a token set by hand goes only to the plugin's own address or to the hosts the secret is bound to; a secret pasted on a step's page goes to another address only once whoever pastes it confirms it; a step's sign-in goes on only once the person who signs in confirmed, in their own browser, every address it goes to that is not the plugin's own (the host they sign in at included), and its callback is taken only in the browser that page sent, so a provider's link passed on to someone else is refused; a step's sign-in, paste or check goes on only to the addresses its page showed, and changing one of them while a flow is under way ends the flow; and making credentials anew for another address means running the flow again from its first step ## Who may call what Every RPC and every HTTP route on the management, edge and metrics ports has a row in the authorization matrix: the kinds of callers it takes, the least role in the tenant it acts in, and whether it needs the content permission. The matrix is defined in `server/internal/authmatrix/matrix.go`. Reading the source currently requires repository access; public source downloads are coming soon. A test starts a server, reads every service and route it serves, fails on one without a row, and calls each with 15 kinds of caller: none, accounts with each role, another tenant's admin, a cluster reader and a cluster admin, the operator's certificate, API keys with each role, another tenant's admin key, a node's certificate and a server's. A second check calls every call without a tenant as a reader of one tenant and fails when an answer names another. On PostgreSQL a third runs management and an edge apart and checks every route they serve the same way. A row that acts in the tenant of an object it names by ID (a node, a key, a rollout, a kill switch, a receipt, a budget request) is called with a real object of the tenant, so each caller is answered as for that object, and the test fails for such a row no call named an object for. The edge's **Temporal proxy**, the port nodes use most, has a row for each of Temporal's WorkflowService and OperatorService methods. A node may call only the worker methods (polling, answering tasks, heartbeats) and one signal, in its own tenant's namespace and on the task queues assigned to it, and answer a task only with the token the edge sealed for it; an active operator may call any; servers, accounts, API keys and callers without a certificate are refused. A test runs every method through the proxy's authorizer as 13 kinds of caller, and fails when a Temporal upgrade adds a method without a row. The following rules apply by interface. ### Admin API The **admin API** takes accounts and operator certificates; API keys only through the operators' MCP endpoint. ### Runtime API The **runtime API** (Connect, HTTP, MCP and A2A) takes accounts on the management port, API keys and operator certificates on both ports; never nodes or servers. An MCP client's OAuth access token works only on its tenant's MCP endpoint on the edge, as its person. ### MCP authorization server The edge's **authorization server for MCP clients** (`/oauth//...` and the two metadata documents) is public, as OAuth needs: authorize checks the client and its redirect before it sends anyone anywhere, registration and token requests are limited per address, and the consent itself is a signed-in person's call on the console. ### Node routes The **node routes** of the edge (model calls, usage, payloads, artifacts, keys, events, boards, permissions) take an admitted node of its own tenant; some also an operator, who names the tenant. A node acts through them only for the jobs it runs: a route that names a job takes it from the call binding of the task the edge handed the node, or from a step the job's workflow scheduled on it, never from what the request says, so a node cannot ask another job's workflow about a tool call, charge another job, its owner's quota or its team's, or write another job's audit entries. The runtime's own asks about a harness session's tool calls are marked by a copy of the binding that only the runtime holds, never by a header. Every node route declares where its job comes from, and a test fails for one that declares nothing. This holds in full when the edge requires call bindings (`--call-binding required`, or the default `auto` for every node whose runtime carries them). The default `auto` goes by what each node reports of its own runtimes: from a node that reports an older runtime without bindings, the edge takes calls without one, but on every node route, for model calls, connection tokens and live-stream publishes alike, only for the very job whose task it handed that node in the last 30 minutes (and a harness session's asks only from the node that got the session's latest task), so even a node that lies about its runtime acts only for jobs it runs. A node that once showed bindings is marked in its stored status and never taken without one again until management tells it to run another runtime or binary; every other node's calls still need their binding. Pin `--call-binding required` once the fleet has upgraded. A child's admission, asked from the parent's workflow task, is taken without a binding only from the node that got the parent's latest workflow task. ### Webhooks Webhooks take only the secret or signature of what they trigger; every other credential gets the same answer as none. A signed delivery is named by what its signature covers, so the same delivery sent again, whatever its headers say, starts nothing new; a source's replay window holds only for a timestamp the sender signed. ### Internal API The **internal API** takes servers' certificates, and builds and forwarded callers also management's MAC. The statements a server sends run as the edges' own [database role](https://docs.oz0.ai/fleet/separate-roles/database-rights#how-management-runs-edges-statements), which reaches only the edges' tables, reads sequences without setting them, can make no table (not even a temporary one), starts every session with a 30-second statement limit and is reset between statements, and runs at most 4 statements at once for any one server; management refuses a statement whose form, functions or tables it does not know before the database sees it, and PostgreSQL's catalogs and other schemas alike. ### Revocations during a database outage Revocations hold while the database does not answer. The edge reads a job's authority envelope before it lets the job's model call, per-call token, child start, tool call, session or continue-as-new through; when it cannot read it, whether the database or management is slow or busy, or the database of a server that runs every role does not answer, it holds the call (`503`, or the task offered again) rather than pass it unchecked, and a job, a scheduled fire included, whose envelope cannot be minted and stored does not run. Every edge stores a job's envelope before the job goes on, and with the roles apart an edge that finds none asks Temporal what the job's run names, so no edge takes a job whose envelope is on its way for one without. With the roles apart, while an edge gets no answer from management at all, or management has no connection to its database (when no revocation can be recorded), the fence lets the calls of jobs whose envelope it does not hold go on, and only what would widen a job's authority, or send a call where its labels forbid, waits. An edge apart from management that refuses a registry snapshot (one older than it holds, or one whose signature fails) reports it to management, which does not count the edge as having applied that version, and the edge falls behind and stops admitting nodes until it applies one ([kill switches](https://docs.oz0.ai/govern/kill-switches#while-the-database-does-not-answer)). ### Public routes Health, the CA certificate, the install script and the node downloads are public; so is `/metrics` on its own plain-HTTP listener, which you should bind to a private address. ## Input from outside The parsers that read what someone outside a cluster can send are fuzzed: plugin manifests and their overlays, flows and their CEL, Claude Code plugins and Agent Plugins, bundles and archives, OCI manifests and registries' challenges, plugin indexes and listing files, webhook and event signatures, event filters, MCP requests, the codec endpoint, the internal API's statement check, and what a node unpacks. The node runtime's harness protocol, ACP's JSON-RPC, flow expressions and the event text a model reads have property tests. `make test` runs each for a few seconds; `make fuzz` longer. What such input may cost is bounded before the work that grows with it: a YAML document's size before it is parsed, its keys and its aliases before it is decoded; a JSON Schema's size before it is parsed, what it holds before it compiles, how far its references fan out before it is used, and what a value would cost before the validator starts; and what the answer says of either. See [the limits](https://docs.oz0.ai/concepts/limits-and-scaling/limits#the-limits). An [outside A2A agent](https://docs.oz0.ai/build/apps/remote-agents/a2a#outside-a2a-agents)'s answers are input from outside too: whatever its card says, its answer taints the job (source `a2a`), so the [taint rule](https://docs.oz0.ai/govern/taint-and-labels) asks before the job's next outbound call. Its card is checked before it is used (a signature wherever its key can be found, the interface and the key on the card's own origin; a definition that sets `signature: required`, `key` or `jwks` refuses a card it cannot verify, and the audit log records how each card was checked), its questions wait for a person's approval, a request for credentials cancels the task, and nothing it sends can choose where the job's data or the plugin's token goes: only the card's origin, which installing the plugin granted. ## Dependencies `make audit` runs govulncheck over the Go modules (the server's and the Go SDK's), uv's audit over each Python lock (the Python SDK's included), and npm's audit over the console's, the docs' and the TypeScript SDK's locks. A finding fails it unless `scripts/audit/exceptions.txt` accepts it. Each accepted advisory identifies the affected dependency, explains why its vulnerable path is not exposed in this product, and sets a date for review. Read the current exception entry for its scope; an exception for build tooling does not establish that another copy of the dependency is safe at runtime. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.