---
title: "Architecture"
description: "One server binary, its management and edge roles, and the nodes that run jobs."
canonical_url: "https://docs.oz0.ai/concepts/architecture"
---
# Architecture

> One server binary, its management and edge roles, and the nodes that run jobs.

For: cluster operators and plugin authors.

Orchestrator Zero has a server binary, `orchestrator-zero` (short name `oz0`), and a node binary, `orchestrator-zero-node` (`oz0-node`). Start the server with `oz0 server start`. By default, that one process runs management, the edge and embedded Temporal together. There is no separate management or edge binary.

Nodes run agents and tools. They connect to the server's edge role. You can run the server and a node on the same machine, or put nodes on other machines near the systems and hardware their jobs need.

<architecture-diagram>



</architecture-diagram>

## The planes

The three planes describe responsibilities and trust boundaries: management, edge and nodes. They do not mean three machines or three server executables. Temporal is the durable workflow engine behind the edge.

<table>
<thead>
  <tr>
    <th>
      Plane
    </th>
    
    <th>
      Responsibility
    </th>
    
    <th>
      Talks to
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Management
    </td>
    
    <td>
      Console, administration and cluster configuration
    </td>
    
    <td>
      PostgreSQL, the edge
    </td>
  </tr>
  
  <tr>
    <td>
      Edge
    </td>
    
    <td>
      Runtime APIs, model gateway and node connections
    </td>
    
    <td>
      Nodes, apps, Temporal, configured outside services, management
    </td>
  </tr>
  
  <tr>
    <td>
      Nodes
    </td>
    
    <td>
      Agent runtime and plugin processes
    </td>
    
    <td>
      The edge; configured systems through the applicable egress controls
    </td>
  </tr>
</tbody>
</table>

Management holds sign-in, the admin API, tenants, people, teams, operators, the node registry and the certificate authority. It also holds plugin and artifact records, the workspace, sealed secrets, routines, usage and the audit log.

The edge serves the Temporal proxy, runtime API, MCP and A2A, webhooks, event deliveries, LLM gateway, plugin builds and artifact downloads. It keeps a control stream open to each node. Calls to model providers, Git and OCI registries use the configured access rules.

## Choose where the roles run

<table>
<thead>
  <tr>
    <th>
      Deployment
    </th>
    
    <th>
      What runs
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Default server
    </td>
    
    <td>
      Management, edge and embedded Temporal in one <code>
        oz0 server start
      </code>
      
       process
    </td>
  </tr>
  
  <tr>
    <td>
      Combined cluster
    </td>
    
    <td>
      Multiple server processes sharing PostgreSQL; each runs the default roles
    </td>
  </tr>
  
  <tr>
    <td>
      Separate roles
    </td>
    
    <td>
      The same binary started with selected <code>
        --roles
      </code>
      
      , on the same or separate machines
    </td>
  </tr>
  
  <tr>
    <td>
      External Temporal
    </td>
    
    <td>
      Your Temporal cluster or Temporal Cloud replaces embedded Temporal
    </td>
  </tr>
</tbody>
</table>

Management does not have to run separately. Separate the roles when you need independent scaling, network placement or key isolation. The flag calls the management role `manager`: for example, `--roles=manager`, `--roles=edge` or `--roles=temporal`. See [Run the roles separately](https://docs.oz0.ai/fleet/separate-roles) for the required connections and keys.

An edge running separately reaches management's tables through its internal API, rather than connecting to management's database. It keeps the watched registry on disk and buffers writes that can wait, such as metering. Its payloads and artifacts need their own database access or object storage.

Temporal is built into the server binary, so the default setup needs no separate Temporal installation. Setting the server's default `--temporal-backend` to an external backend stops it from starting embedded Temporal. You can also choose a different [Temporal backend](https://docs.oz0.ai/fleet/temporal-backends) for an individual tenant; that does not change which roles the server starts.

## Design rules

1. Nodes need no inbound network ports. A node connects out to the edge on port `7233` (Temporal, mTLS) and `7443` (join, renewals, the control stream, artifacts and the LLM gateway). Its local proxies serve processes on that node, so it works behind NAT and firewalls.
2. Every node call is authorized. The edge checks each call against its node registry. Combined servers refresh it on database notifications and every two seconds. Separate edges follow management's watch stream; [outage rules](https://docs.oz0.ai/fleet/separate-roles/outages) bound use of a stale copy.
3. Gateway provider keys stay central. They are sealed in management and unsealed in the edge; nodes call a local proxy instead of receiving those keys. Plugin secrets have delivery paths that depend on the platform and execution mode: see [credential injection](https://docs.oz0.ai/admin/credential-injection).
4. Management never needs Temporal. When the roles run separately, running jobs, schedules and the LLM gateway can continue through a management outage, within the edge's [cache and buffer limits](https://docs.oz0.ai/fleet/separate-roles/outages). Stopping a combined server stops all roles in that process.
5. The core is neutral. No models, tools or services are built in; they come from plugins.

## A job, end to end

<steps level="4">

#### A client starts a job

`oz0 run acme-reviewer "..."` or an app calls `StartJob` on the edge's runtime API. The edge starts an `AgentWorkflow` for `acme-reviewer` on its plugin's task queue, such as `pool.acme-github`, in the tenant's Temporal namespace.

<span id="a-node-picks-it-up-and-becomes-the-agent">



</span>



#### A node picks up the job

Every node that has the agent's plugin, and the labels it requires, polls that queue, which the plugin's agents share ([routing](https://docs.oz0.ai/concepts/routing)). The first one free takes the job and becomes its **home node**.

#### Model calls go through the gateway

The agent's model calls leave the node through its local LLM proxy, reach the edge's gateway, get the tenant's key added and go to the provider. The gateway records tokens and cost under the job.

#### Tools run on the home node

Tool calls are Temporal activities on `node.<home>`, so they run where the job runs. The agent's own steps can run on any node that has the agent.

#### Children run anywhere

When the agent delegates with `oz.delegate`, the child is a job of its own on any node that has the child agent. Its cost counts toward the parent's tree.

#### The answer comes back

The workflow completes, the CLI or app gets the answer, and `job get` shows the cost of every branch.

</steps>

## When something is down

The role-specific cases below assume the other roles and their storage remain available. A stopped combined server loses management, edge and embedded Temporal together; other servers provide failover. See [High availability](https://docs.oz0.ai/fleet/high-availability).

<table>
<thead>
  <tr>
    <th>
      Down
    </th>
    
    <th>
      What happens
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <strong>
        Management
      </strong>
    </td>
    
    <td>
      Separate edges can continue runtime work within their cache and buffer limits. Joins, installs and secret changes wait.
    </td>
  </tr>
  
  <tr>
    <td>
      <strong>
        An edge
      </strong>
    </td>
    
    <td>
      Its nodes move to another server's edge within seconds. Run at least two servers.
    </td>
  </tr>
  
  <tr>
    <td>
      <strong>
        All edges
      </strong>
    </td>
    
    <td>
      Nodes cannot reach Temporal or models. Work pauses without losing state and resumes when an edge is back.
    </td>
  </tr>
  
  <tr>
    <td>
      <strong>
        Temporal
      </strong>
    </td>
    
    <td>
      Workflow history is durable; after a failure Temporal replays it and each in-flight activity follows its retry or recovery rules.
    </td>
  </tr>
  
  <tr>
    <td>
      <strong>
        PostgreSQL
      </strong>
    </td>
    
    <td>
      Management stops. The edge keeps authorizing nodes from its cached registry. In embedded mode Temporal stops too.
    </td>
  </tr>
  
  <tr>
    <td>
      <strong>
        A node
      </strong>
    </td>
    
    <td>
      Workflow history stays durable, but in-flight work follows its retry or recovery rules. A lost harness session may start fresh from job input or wait for approval (<a href="https://docs.oz0.ai/concepts/jobs">
        jobs and durability
      </a>
      
      ).
    </td>
  </tr>
</tbody>
</table>

## Read more

- [Jobs and durability](https://docs.oz0.ai/concepts/jobs)
- [Routing](https://docs.oz0.ai/concepts/routing)
- [Security model](https://docs.oz0.ai/concepts/security)
- [Temporal backends](https://docs.oz0.ai/concepts/temporal)


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.
