Run the roles separately

For: cluster operators.

Management, edge and embedded Temporal are roles of the same orchestrator-zero binary, also called oz0. By default one oz0 server start process runs all three.

You can run them as separate processes instead, on separate machines: management in an internal network, two or more edges near the nodes, and Temporal on its own. This is optional: management can stay in the combined process. The management flag value is manager.

Nodes use the separate oz0-node binary and keep the same connections to the edge whichever layout you choose.

Management never connects to Temporal and cannot open tenant secrets. An edge never connects to management's database: it follows management's registry over a stream, and reaches management's tables through management's internal API, which allows an edge only the tables it needs.

Prepare database rights, create the secrets key and start management, edges and Temporal separately.

Choose a task

Start the roles

Create the cluster as usual with server init, then make the secrets key, which only the edges hold:

oz0 server secrets-key create --out /etc/oz0/secrets.key
# Wrote the secrets key to /etc/oz0/secrets.key (mode 0600).
# Its public half: oz0-box-v1:...

Give the edges a place for payloads and plugin artifacts: an S3-compatible bucket (--payload-store), or a PostgreSQL role of their own that may touch only the payload and artifact tables and the job files' scopes (file_scopes, which say which job tree may read each file). Name the role anything but oz0_edge_<database>, which management keeps for itself:

CREATE ROLE oz0_payloads LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE oz0 TO oz0_payloads;
GRANT USAGE ON SCHEMA public TO oz0_payloads;
GRANT SELECT, INSERT, UPDATE, DELETE ON payloads, blobs, file_scopes TO oz0_payloads;

Then start the roles:

The database URLs are in files readable only by the servers' user (/etc/oz0/db-url holds postgres://oz0:PASSWORD@db:5432/oz0, /etc/oz0/payload-db-url the payload role's), or in OZ0_DB and OZ0_PAYLOAD_DB: never on the command line, where every user of the machine can read them.

# Temporal
oz0 server start --roles=temporal --name temporal-1 \
  --db-file /etc/oz0/db-url --master-key-file /etc/oz0/master.key --advertise 10.0.1.5

# Management
oz0 server start --roles=manager --name mgmt \
  --db-file /etc/oz0/db-url --master-key-file /etc/oz0/master.key --advertise 10.0.1.4

# Each edge
oz0 server start --roles=edge --name edge-a --advertise 10.0.2.10 \
  --management-url https://10.0.1.4:8443 \
  --temporal-address 10.0.1.5:7234 \
  --payload-db-file /etc/oz0/payload-db-url \
  --master-key-file /etc/oz0/master.key --secrets-key-file /etc/oz0/secrets.key \
  --data-dir /var/lib/oz0

The console's runtime calls, and plugin builds, go from management to an edge that is not fenced, signed with a key derived from the master key over the whole call and the edge it goes to, and each signature works once, within a minute; a server certificate alone cannot make an edge act for someone.

Nodes join through any edge, as before. Every edge's certificate carries the name edge.oz0, and each node learns every edge's address, so run at least two.

Management's certificate also carries management.oz0, a name the cluster CA gives only to servers that run the manager role, and the edges check it whenever they call management, so no other server can answer them as management. In the same way the temporal role's internode certificate carries temporal.oz0, which the internode CA gives only to it, and the edges check it on the frontend they dial. A server whose --san names it refuses to start. Management issues its certificate at each start, so an upgrade gives it the name; upgrade management before the edges (see Upgrades).

Management checks buffered writes against the server that sent them:

• A write about a server's own rows, such as its key sources or signing key, must come from that server's certificate.
• A node's reports, including facts, heartbeats, models, drain and the end of its session, are applied only for the edge that last reported that session. They never change the node's state, tenant or certificate. The edges' database role still writes node rows for joins and renewals.
• A write setting the cluster's key material must carry proof of the master key. This covers the secrets key's publication, a value re-sealed to it, a signing key and the record that every key is sealed by an edge. Every edge holds the master key; a certificate alone does not prove it.

Each server signs its row in the servers table with a key derived from the master key, and the servers apart from each other take only rows whose signature fits: management hands calls, artifacts and plugin builds only to an edge with such a row, the edges give nodes only those edges' addresses, and the fence asks only those peers.

Management and the edges also check that the server answering at an address is the one its row names. During an upgrade a server of the earlier version writes no signature, so the others leave it out until it is upgraded; upgrade management, then every edge promptly.

Readiness: management answers GET /readyz on port 8443 with 200 while its database answers; an edge answers GET /readyz on port 7443 with 200 unless it is fenced.

Previous section links

How management runs edges' statements: open guide.

What the database must allow: open guide.

Secrets management cannot open: open guide.

What happens when something is down: open guide.

What an edge answers while management is away: open guide.

Snapshots and long outages: open guide.

Limits: open guide.

Temporal's UI: open guide.