Concepts

Architecture

One server binary, its management and edge roles, and the nodes that run jobs.

For: cluster operators and plugin authors.

Orchestrator Zero has a server binary, orchestrator-zero (short name oz0), and a node binary, orchestrator-zero-node (oz0-node). Start the server with oz0 server start. By default, that one process runs management, the edge and embedded Temporal together. There is no separate management or edge binary.

Nodes run agents and tools. They connect to the server's edge role. You can run the server and a node on the same machine, or put nodes on other machines near the systems and hardware their jobs need.

One server process by default

oz0 server start

Management

Console, admin API, registry and sealed configuration

People and operators connect on 8443

Edge

Runtime API, model gateway, artifacts and node connections

Apps use 7443; nodes use 7233 and 7443

Embedded Temporal

The edge reaches this workflow engine to keep jobs durable.

Nodes connect out to the edge over mTLS.

Node processes: oz0-node

Run the agent runtime and plugins on laptops, servers, VMs or GPU machines.

This boundary shows the default process, not a required machine layout. Cluster mode stores management and Temporal state in PostgreSQL; dev mode uses SQLite. The edge reaches configured model providers, and tools reach their configured systems. You can separate server roles or replace embedded Temporal with an external backend.

The planes

The three planes describe responsibilities and trust boundaries: management, edge and nodes. They do not mean three machines or three server executables. Temporal is the durable workflow engine behind the edge.

PlaneResponsibilityTalks to
ManagementConsole, administration and cluster configurationPostgreSQL, the edge
EdgeRuntime APIs, model gateway and node connectionsNodes, apps, Temporal, configured outside services, management
NodesAgent runtime and plugin processesThe edge; configured systems through the applicable egress controls

Management holds sign-in, the admin API, tenants, people, teams, operators, the node registry and the certificate authority. It also holds plugin and artifact records, the workspace, sealed secrets, routines, usage and the audit log.

The edge serves the Temporal proxy, runtime API, MCP and A2A, webhooks, event deliveries, LLM gateway, plugin builds and artifact downloads. It keeps a control stream open to each node. Calls to model providers, Git and OCI registries use the configured access rules.

Choose where the roles run

DeploymentWhat runs
Default serverManagement, edge and embedded Temporal in one oz0 server start process
Combined clusterMultiple server processes sharing PostgreSQL; each runs the default roles
Separate rolesThe same binary started with selected --roles, on the same or separate machines
External TemporalYour Temporal cluster or Temporal Cloud replaces embedded Temporal

Management does not have to run separately. Separate the roles when you need independent scaling, network placement or key isolation. The flag calls the management role manager: for example, --roles=manager, --roles=edge or --roles=temporal. See Run the roles separately for the required connections and keys.

An edge running separately reaches management's tables through its internal API, rather than connecting to management's database. It keeps the watched registry on disk and buffers writes that can wait, such as metering. Its payloads and artifacts need their own database access or object storage.

Temporal is built into the server binary, so the default setup needs no separate Temporal installation. Setting the server's default --temporal-backend to an external backend stops it from starting embedded Temporal. You can also choose a different Temporal backend for an individual tenant; that does not change which roles the server starts.

Design rules

  1. Nodes need no inbound network ports. A node connects out to the edge on port 7233 (Temporal, mTLS) and 7443 (join, renewals, the control stream, artifacts and the LLM gateway). Its local proxies serve processes on that node, so it works behind NAT and firewalls.
  2. Every node call is authorized. The edge checks each call against its node registry. Combined servers refresh it on database notifications and every two seconds. Separate edges follow management's watch stream; outage rules bound use of a stale copy.
  3. Gateway provider keys stay central. They are sealed in management and unsealed in the edge; nodes call a local proxy instead of receiving those keys. Plugin secrets have delivery paths that depend on the platform and execution mode: see credential injection.
  4. Management never needs Temporal. When the roles run separately, running jobs, schedules and the LLM gateway can continue through a management outage, within the edge's cache and buffer limits. Stopping a combined server stops all roles in that process.
  5. The core is neutral. No models, tools or services are built in; they come from plugins.

A job, end to end

A client starts a job

oz0 run acme-reviewer "..." or an app calls StartJob on the edge's runtime API. The edge starts an AgentWorkflow for acme-reviewer on its plugin's task queue, such as pool.acme-github, in the tenant's Temporal namespace.

A node picks up the job

Every node that has the agent's plugin, and the labels it requires, polls that queue, which the plugin's agents share (routing). The first one free takes the job and becomes its home node.

Model calls go through the gateway

The agent's model calls leave the node through its local LLM proxy, reach the edge's gateway, get the tenant's key added and go to the provider. The gateway records tokens and cost under the job.

Tools run on the home node

Tool calls are Temporal activities on node.<home>, so they run where the job runs. The agent's own steps can run on any node that has the agent.

Children run anywhere

When the agent delegates with oz.delegate, the child is a job of its own on any node that has the child agent. Its cost counts toward the parent's tree.

The answer comes back

The workflow completes, the CLI or app gets the answer, and job get shows the cost of every branch.

When something is down

The role-specific cases below assume the other roles and their storage remain available. A stopped combined server loses management, edge and embedded Temporal together; other servers provide failover. See High availability.

DownWhat happens
ManagementSeparate edges can continue runtime work within their cache and buffer limits. Joins, installs and secret changes wait.
An edgeIts nodes move to another server's edge within seconds. Run at least two servers.
All edgesNodes cannot reach Temporal or models. Work pauses without losing state and resumes when an edge is back.
TemporalWorkflow history is durable; after a failure Temporal replays it and each in-flight activity follows its retry or recovery rules.
PostgreSQLManagement stops. The edge keeps authorizing nodes from its cached registry. In embedded mode Temporal stops too.
A nodeWorkflow history stays durable, but in-flight work follows its retry or recovery rules. A lost harness session may start fresh from job input or wait for approval (jobs and durability).

Read more

Copyright © 2026