Temporal
For: cluster operators and plugin authors.
Orchestrator Zero uses the open-source Temporal engine to keep agent runs durable across crashes, restarts and lost nodes. Temporal records workflow history and schedules the activities that do the work.
| Temporal feature | What it gives your agents |
|---|---|
| Durable execution | A crash, deploy or lost node does not erase workflow history; Temporal replays that history and each in-flight activity follows its retry or recovery rules |
| Activities | Temporal retries model calls; tool calls report their result and retry only when read-only or idempotent under a key their server honours |
| Task queues | Each job lands on a node with the right agent and labels |
| Child workflows | Agents delegate to agents on other machines |
| Timers and signals | Runs can wait for a long time without holding a machine |
| History | Every step of every job is recorded and can be inspected |
You never use Temporal directly: the edge talks to it for the nodes, and nodes cannot tell which backend is behind it.
Backends
| Backend | What it is | Status |
|---|---|---|
| Embedded | Temporal's services inside the orchestrator-zero binary, on the same PostgreSQL (SQLite in dev mode) | Works today; the default |
| Your own cluster | A Temporal cluster your organization already runs (--temporal-backend external) | Works today |
| Temporal Cloud | Temporal's hosted service, with a namespace per tenant (--temporal-backend cloud) | Works today, against a stand-in; not yet tried against a real account |
A cluster can also reach several named backends at once, with each tenant on one of them, and move a tenant from one to another while its jobs run: new jobs start on the new backend and running ones finish on the old (more than one backend).
Nodes go through the edge with every backend. Temporal Cloud does not check certificate revocation, so the edge holds the Cloud credentials and blocks nodes itself, on their next call. With Temporal Cloud every payload is encrypted with the tenant's key before it leaves the node or the edge. See Temporal backends for setup, keys, namespaces and the codec endpoint.
A node answers only its own tasks, with every backend alike. The edge hands each task to its node with the task's token sealed for that node, and takes an answer, a heartbeat or a failure only with a token sealed for the node that sends it, so no node completes another's work, even in its own tenant.
A node cannot complete an activity by its workflow and activity IDs, take or answer Nexus tasks, or name another node's queue when its worker shuts down or reports. The edge also derives the names of each node's sticky queues (the per-worker queues Temporal uses for cached workflows) from the node, so no node can poll another's.
A workflow task's commands may signal or cancel only the jobs that job started, and may not start Nexus operations. The one signal a node sends itself, an activity's publish to its job's live stream, carries the call binding of that activity, so a node publishes only to the jobs whose work it runs.
The edge passes on only the parts of a node's request it knows and checked, and only what acts on the node's own worker: a node cannot set a task queue's dispatch rate for the other nodes on it, and its workers join only the platform's Worker Deployment.
When a node is blocked, turns unhealthy, starts to drain or reaches its drain's deadline while its long poll waits, the edge ends the poll as soon as it learns of the change, so the next task goes to another node. A task Temporal matched to the poll meanwhile is not handed to a blocked node; a node that is only unhealthy or draining gets it and runs it, rather than leave it waiting for its timeout.
When you upgrade the edges, a task a node held from before the upgrade cannot be answered with its old token: Temporal offers it again after its timeout (ten seconds for a workflow task; an activity's heartbeat or start-to-close timeout). While some edges are upgraded and others not, the same happens to a task handed out by an edge of one version and answered through one of the other. Upgrade the nodes' runtimes with the edges: an edge that requires call bindings refuses the live stream publishes of an older runtime, whose jobs then run without live output.
Things to know about embedded Temporal
- History shards are fixed at creation.
server init --history-shardssets them once; 1024 suits a cluster that may grow large, 64 or 128 suit small ones. - Ports must be 32767 or lower, because Temporal's PostgreSQL schema stores them in a small integer column. The defaults are fine.
- A tenant has at most 5,000 agents and flows by default, across all its plugins. Orchestrator Zero caps agent and workflow queue pools at 50 by default with embedded Temporal, and 33 with Temporal Cloud.
plugin installrefuses an install that would exceed either configured limit. A plugin's agents share one queue per set of labels they require, and Temporal registers a new runtime version's queues one at a time, so the queues, not the agents, are what cost time. See limits and scaling. - Upgrades go one Temporal minor version at a time.
server startapplies Temporal's schema migrations, and ours, under a PostgreSQL lock.
Security assurance
Review status, operational security limits and how to report a vulnerability.
Limits and scaling
How many agents and task queues a tenant may have, what a large plugin and a tenant of many small plugins cost, what 10,000 agents in one tenant measured, and what a fleet of nodes under load measured.