---
title: "Run the roles separately"
description: "Prepare database rights, create the secrets key and start management, edges and Temporal separately."
canonical_url: "https://docs.oz0.ai/fleet/separate-roles"
---
# Run the roles separately

> Prepare database rights, create the secrets key and start management, edges and Temporal separately.

For: cluster operators.

Management, edge and embedded Temporal are roles of the same `orchestrator-zero` binary, also called `oz0`. By default one `oz0 server start` process runs all three.

You can run them as separate processes instead, on separate machines: management in an internal network, two or more edges near the nodes, and Temporal on its own. This is optional: management can stay in the combined process. The management flag value is `manager`.

Nodes use the separate `oz0-node` binary and keep the same connections to the edge whichever layout you choose.

<table>
<thead>
  <tr>
    <th>
      Role
    </th>
    
    <th>
      What it runs
    </th>
    
    <th>
      Needs
    </th>
    
    <th>
      Ports
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        manager
      </code>
    </td>
    
    <td>
      Admin API, console and sign-in, registry, CA, rollouts' records, cost reports
    </td>
    
    <td>
      PostgreSQL (<code>
        OZ0_DB
      </code>
      
       or <code>
        --db-file
      </code>
      
      ), the master key
    </td>
    
    <td>
      8443 (<code>
        --management-port
      </code>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        edge
      </code>
    </td>
    
    <td>
      Temporal proxy, runtime API, webhooks, LLM gateway, artifacts, control stream, the platform's own loops (rollouts, quality gates, schedules, summaries)
    </td>
    
    <td>
      Management's URL, the master key, the secrets key, its own PostgreSQL role or an S3-compatible bucket for payloads and artifacts, a data directory, a Temporal frontend
    </td>
    
    <td>
      7233 and 7443
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        temporal
      </code>
    </td>
    
    <td>
      The embedded Temporal services
    </td>
    
    <td>
      PostgreSQL (<code>
        OZ0_DB
      </code>
      
       or <code>
        --db-file
      </code>
      
      ), the master key
    </td>
    
    <td>
      The <code>
        --temporal-*-port
      </code>
      
       flags
    </td>
  </tr>
</tbody>
</table>

Management never connects to Temporal and cannot open tenant secrets. An edge never connects to management's database: it follows management's registry over a stream, and reaches management's tables through management's internal API, which allows an edge only the tables it needs.

Prepare database rights, create the secrets key and start management, edges and Temporal separately.

## Choose a task

<table>
<thead>
  <tr>
    <th>
      Task
    </th>
    
    <th>
      Guide
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Prepare database rights
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/separate-roles/database-rights">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Keep plaintext secrets on edges
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/separate-roles/secrets">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Operate through role outages
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/separate-roles/outages">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Use Temporal UI with separate roles
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/separate-roles/temporal-ui">
        Open guide
      </a>
    </td>
  </tr>
</tbody>
</table>

## Start the roles

Create the cluster as usual with `server init`, then make the secrets key, which only the edges hold:

```bash [Terminal]
oz0 server secrets-key create --out /etc/oz0/secrets.key
# Wrote the secrets key to /etc/oz0/secrets.key (mode 0600).
# Its public half: oz0-box-v1:...
```

Give the edges a place for payloads and plugin artifacts: an [S3-compatible bucket](https://docs.oz0.ai/fleet/payload-storage) (`--payload-store`), or a PostgreSQL role of their own that may touch only the payload and artifact tables and the job files' scopes (`file_scopes`, which say which job tree may read each file). Name the role anything but `oz0_edge_<database>`, which management keeps for itself:

```sql [psql]
CREATE ROLE oz0_payloads LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE oz0 TO oz0_payloads;
GRANT USAGE ON SCHEMA public TO oz0_payloads;
GRANT SELECT, INSERT, UPDATE, DELETE ON payloads, blobs, file_scopes TO oz0_payloads;
```

Then start the roles:

The database URLs are in files readable only by the servers' user (`/etc/oz0/db-url` holds
`postgres://oz0:PASSWORD@db:5432/oz0`, `/etc/oz0/payload-db-url` the payload role's), or in `OZ0_DB` and `OZ0_PAYLOAD_DB`:
never on the command line, where every user of the machine can read them.

```bash [Terminal]
# Temporal
oz0 server start --roles=temporal --name temporal-1 \
  --db-file /etc/oz0/db-url --master-key-file /etc/oz0/master.key --advertise 10.0.1.5

# Management
oz0 server start --roles=manager --name mgmt \
  --db-file /etc/oz0/db-url --master-key-file /etc/oz0/master.key --advertise 10.0.1.4

# Each edge
oz0 server start --roles=edge --name edge-a --advertise 10.0.2.10 \
  --management-url https://10.0.1.4:8443 \
  --temporal-address 10.0.1.5:7234 \
  --payload-db-file /etc/oz0/payload-db-url \
  --master-key-file /etc/oz0/master.key --secrets-key-file /etc/oz0/secrets.key \
  --data-dir /var/lib/oz0
```

<table>
<thead>
  <tr>
    <th>
      Flag
    </th>
    
    <th>
      Role
    </th>
    
    <th>
      What it does
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        --roles
      </code>
    </td>
    
    <td>
      all
    </td>
    
    <td>
      The roles this process runs, comma-separated. <code>
        manager,edge,temporal
      </code>
      
       (the default) is one server, as before.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --management-url
      </code>
    </td>
    
    <td>
      edge
    </td>
    
    <td>
      Management's address. Repeat it for several management servers: the edge, and its registry stream, move to the next one when one does not answer.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --temporal-address
      </code>
    </td>
    
    <td>
      edge
    </td>
    
    <td>
      The Temporal frontend of a <code>
        --roles=temporal
      </code>
      
       server, reached with the cluster's internode TLS.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --payload-db-file
      </code>
    </td>
    
    <td>
      edge
    </td>
    
    <td>
      A file holding the edge's own connection for payloads and plugin artifacts (or <code>
        OZ0_PAYLOAD_DB
      </code>
      
      ). Optional with <code>
        --payload-store
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --payload-store
      </code>
    </td>
    
    <td>
      edge
    </td>
    
    <td>
      An S3-compatible bucket for payloads and plugin artifacts instead: see <a href="https://docs.oz0.ai/fleet/payload-storage">
        Payload storage
      </a>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --secrets-key-file
      </code>
    </td>
    
    <td>
      edge
    </td>
    
    <td>
      The key tenant secrets are sealed to. Give it to every edge, and to no server that runs only management.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --data-dir
      </code>
    </td>
    
    <td>
      edge
    </td>
    
    <td>
      Where the edge keeps its registry snapshot, the writes waiting for management, its read cache and the cluster record. Use a volume that survives restarts.
    </td>
  </tr>
</tbody>
</table>

The console's runtime calls, and plugin builds, go from management to an edge that is not fenced, signed with a key derived from the master key over the whole call and the edge it goes to, and each signature works once, within a minute; a server certificate alone cannot make an edge act for someone.

Nodes join through any edge, as before. Every edge's certificate carries the name `edge.oz0`, and each node learns every edge's address, so run at least two.

Management's certificate also carries `management.oz0`, a name the cluster CA gives only to servers that run the manager role, and the edges check it whenever they call management, so no other server can answer them as management. In the same way the temporal role's internode certificate carries `temporal.oz0`, which the internode CA gives only to it, and the edges check it on the frontend they dial. A server whose `--san` names it refuses to start. Management issues its certificate at each start, so an upgrade gives it the name; upgrade management before the edges (see [Upgrades](https://docs.oz0.ai/fleet/upgrades/storage-and-identity#the-management-name-and-signed-calls)).

Management checks buffered writes against the server that sent them:

- A write about a server's own rows, such as its key sources or signing key, must come from that server's certificate.
- A node's reports, including facts, heartbeats, models, drain and the end of its session, are applied only for the edge that last reported that session. They never change the node's state, tenant or certificate. The edges' database role still writes node rows for joins and renewals.
- A write setting the cluster's key material must carry proof of the master key. This covers the secrets key's publication, a value re-sealed to it, a signing key and the record that every key is sealed by an edge. Every edge holds the master key; a certificate alone does not prove it.

Each server signs its row in the servers table with a key derived from the master key, and the servers apart from each other take only rows whose signature fits: management hands calls, artifacts and plugin builds only to an edge with such a row, the edges give nodes only those edges' addresses, and the fence asks only those peers.

Management and the edges also check that the server answering at an address is the one its row names. During an upgrade a server of the earlier version writes no signature, so the others leave it out until it is upgraded; upgrade management, then every edge promptly.

Readiness: management answers `GET /readyz` on port 8443 with 200 while its database answers; an edge answers `GET /readyz` on port 7443 with 200 unless it is fenced.

## Previous section links

<span id="how-management-runs-edges-statements">

How management runs edges' statements

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/database-rights#how-management-runs-edges-statements).

<span id="what-the-database-must-allow">

What the database must allow

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/database-rights#what-the-database-must-allow).

<span id="secrets-management-cannot-open">

Secrets management cannot open

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/secrets#secrets-management-cannot-open).

<span id="what-happens-when-something-is-down">

What happens when something is down

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/outages#what-happens-when-something-is-down).

<span id="what-an-edge-answers-while-management-is-away">

What an edge answers while management is away

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/outages#what-an-edge-answers-while-management-is-away).

<span id="snapshots-and-long-outages">

Snapshots and long outages

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/outages#snapshots-and-long-outages).

<span id="limits">

Limits

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/outages#limits).

<span id="temporals-ui">

Temporal's UI

</span>

: [open guide](https://docs.oz0.ai/fleet/separate-roles/temporal-ui#temporals-ui).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.