Fleet

High availability

Run two or more servers so nodes and jobs keep going when one stops.

For: cluster operators.

Plan server redundancy around the failures below. Fencing and freshness checks determine which edge may continue serving.

This guide uses the default combined layout with embedded Temporal: each orchestrator-zero server runs management, edge and Temporal in one process. Run at least two. When one stops, its nodes move to another server's edge within seconds; interrupted work follows its retry or recovery rules. You can also run management, the edges and Temporal as separate processes; see Run the roles separately.

How nodes fail over

  • Every edge certificate carries the extra name edge.oz0, so one TLS configuration fits every edge of the cluster.
  • When a node connects, the edge sends it the list of every server's edge that reported in the last 30 seconds, and sends it again when the list changes.
  • The node keeps that list and always tries the address it joined through first, then the others.
  • A server that shuts down closes its control streams at once and gives Temporal calls in flight two seconds, so its nodes move quickly.
  • In a cluster with regions, the list names only the edges of the node's own region: a node never fails over to another region's edge, which would refuse it. Run two or more edges in each region.

When a server loses PostgreSQL: fencing

Each edge keeps a copy of the registry: nodes, tenants, operators, plugins and API keys. It reads it again at once when anything in it changes (PostgreSQL tells every server through LISTEN/NOTIFY), and every 2 seconds in any case. A change such as a block takes effect within 2 seconds on every edge that can reach the database; in tests it takes 10 to 40 milliseconds.

An edge that cannot read the registry for 10 seconds asks the other servers' edges, over mutual TLS with the servers' own certificates, how long ago they read it. What it does next depends on the answers:

StateWhenNew nodes, renewals, plugin downloadsNodes it already servesReady (/readyz)
freshIt read the registry in the last 10 sYesYesYes
behindIt has not, and is still askingNo: nodes try another edgeYesYes
fencedAnother edge read the registry in the last 5 s and has been fresh for at least 10 s, or no edge has answered for 30 sNoNo: it ends their control streams with "reconnect elsewhere", answers 503 to every other call on the edge port (model calls, traces, payloads, the runtime API), sends GOAWAY on its Temporal port, and leaves the Temporal ringNo (503)
fail-staticEvery edge that answers is behind too, or there is no other serverNoYes, from its copyYes, with an alert

So a server that alone has lost the database steps aside: its nodes reconnect to another edge within a second or two and send their model calls and other calls there too, and the other servers' Temporal takes over its share of the work. When PostgreSQL is down for every server at once, no server fences itself: they all keep serving from their copies, jobs that do not need the database carry on, and blocks wait until it is back. A server with no peers never fences itself.

When the database comes back, the fenced server reads the registry, turns fresh, serves again and joins the Temporal ring. Nodes stay where they moved until their connection fails. After an outage that hit every server, the servers come back a second or two apart; a peer must have been fresh for 10 seconds before an edge fences itself for it, so the last one back does not.

A read of the registry gives up after 5 seconds. In a PostgreSQL failover, the old primary's connections can die without either side noticing, and a query on one would wait for the operating system to give up on it, some 15 minutes. The edge drops such a connection and the idle ones beside it, and reads again over new connections to the new primary, so it does not fall behind.

The registry has two connections of its own, so a busy server cannot hold its reads up either. The edge's other copies (kill switches, tool policies, model providers and routines) and the server's heartbeat give up after 5 seconds the same way.

Check a server's state with its readiness endpoint, which load balancers can use, or with the CLI:

Terminal
curl -k https://10.0.0.12:7443/readyz
# fenced: the registry is 14s behind while peers are not; nodes go to other edges

oz0 cluster edges
# Registry version  1791001239707
# Applied           applied on 2 of 3 edges; server-c behind for 40 s
#
# EDGE      STATE  VERSION        LAST READ  NOTE
# server-a  fresh  1791001239707  1s ago
# server-b  fresh  1791001239707  0s ago
# server-c  fresh  1791001239650  40s ago    behind for 40s

The state an edge reports is the last one it could write; a server cut off from the database cannot say it is fenced, so the CLI and the console's Overview show it as behind with how long it has been silent. Every state change is logged (edge fenced: ..., edge fresh again: ...) and counted in the metrics oz0_edge_state and oz0_edge_state_changes_total.

Where a change applied

Every change to the registry bumps its version, and each edge records the version it applied. node block and node unblock wait up to 3 seconds (--wait) for every edge and say where the change applied:

Terminal
oz0 node block node-1a2b --reason "lost laptop"
# Node node-1a2b blocked.
# Block applied on 2 of 3 edges; server-c behind for 40 s.

The console says the same when you block a node, and the Overview page lists the edges. An edge that stopped on purpose is not counted, nor one that has been silent for more than 10 minutes.

A reload that looks wrong is not applied

An edge refuses a registry that looks broken rather than act on it: one with no tenants, or one in which more than half of the accepted nodes, and at least three, lose access at once. It keeps its last good copy, logs an error, counts it in oz0_registry_reloads_refused_total, and shows it in cluster edges and on the Overview page. If the change is meant, a cluster admin confirms it, and the edges apply it within seconds:

Terminal
oz0 cluster confirm-registry

A confirmation covers the registry as it is when you confirm. A later change that looks wrong, such as a second mass block a minute after, is refused again and needs its own confirmation.

Behind a load balancer

If nodes reach the edges through a load balancer or a DNS name, give that name to every server with --san, so the edge's certificate is valid for it:

Terminal
oz0 server start --master-key-file /etc/orchestrator-zero/master.key \
  --advertise 10.0.0.11 --san edge.example.com

Join nodes with --server https://edge.example.com:7443. They keep trying that address first, so the load balancer stays in charge.

Work that runs on one server at a time

Some work must run on exactly one server: a plugin's quality gate, each rollout, and starting the rollouts that wait for their turn. A server takes a lease in the database before it does such work:

  • A lease lasts 15 seconds and its holder renews it every 5. Its expiry comes from the database's clock, so servers whose clocks disagree still agree on who holds it.
  • When a server stops, pauses or loses its connection to the database, its leases expire and another server takes them over within about 15 seconds. If that happens in the middle of one of its writes, PostgreSQL ends the stalled transaction 15 seconds later, so the takeover comes within about 30 seconds at worst. The takeover raises the lease's epoch. A server that only tries to take a lease reads it without locking it, and takes it in one statement that waits for no lock, so a server that hangs while it tries never costs the holder its lease.
  • Every write the holder makes to the database checks its epoch in the same transaction. A server that paused, for a garbage collection or a frozen virtual machine, and wakes up after its lease was taken over cannot write over its successor's work: the write is refused and the server stops that work. Before a call to Temporal, which cannot check an epoch, such as a rollout's ramp, the server checks that it still holds the lease by its own clock too.
  • A server that loses a lease logs lease: lost; stopping its work. A gate that stops this way runs again from the start on the server that took it over; a rollout goes on from the step it was saved at.

See who holds what, on the database's clock (cluster admins only):

Terminal
oz0 cluster leases
LEASE             HOLDER    EPOCH  EXPIRES  ACQUIRED
gate:4f1c…        server-b  3      in 12.4s 2m10s ago
rollout:9d2e…     server-a  1      in 9.1s  40s ago
rollouts          server-a  2      in 13.8s 1h2m ago

Connections to PostgreSQL

Give every server a direct connection to PostgreSQL, or one through PgBouncer in session mode. Servers take PostgreSQL advisory locks while they migrate the schema and set up Temporal, and those belong to a session, which PgBouncer's transaction and statement modes do not keep. Leases themselves need no session: each check is a statement in the transaction it guards.

Do not let a pooler hold connections open for a server that has gone: an advisory lock waits for the connection to close, which with Linux's default TCP keepalive takes about two hours. A lease expires on the database's clock whatever its holder's connection does. While the holder writes, it holds a row lock on its lease, and each such transaction may run or sit idle for at most the lease's 15 seconds (statement_timeout and idle_in_transaction_session_timeout, set for that transaction only) before PostgreSQL ends it and frees the lease.

What still depends on one thing

  • PostgreSQL. Every server uses it. Run it with your usual high-availability setup. If it is down for every server, management stops; the edges keep authorizing nodes from their cached registry (fail-static, above), blocks wait until it is back, and with embedded Temporal, jobs pause. If it is down for one server only, that server fences itself and the others carry on.
  • A region. With regions, a tenant's jobs run only in its region: when that region's edges, Temporal or payload store are down, its tenants' jobs wait until it is back. No other region takes over.
  • The master key. Every server needs a copy. Store it like any other critical key.
  • The cluster CA's key, when it lives in an HSM or a KMS instead of the database (see Create a cluster): servers sign with it when they start and when nodes join or renew, so every server must reach the signing command.
Copyright © 2026