Management over a WAN
For: cluster operators.
Use the measurements and limits below when placing management far from its edges. Registry delivery, cached reads and buffered writes respond differently to latency.
With the roles apart, management does not have to sit next to its edges. It can run in another data centre, another country or another cloud, with the edges, their nodes and Temporal together on the other side. Nodes never talk to management, so the network between management and the edges is the only one that gets longer.
This page says what that network must allow, what it may be like, and what gets slower. We test it with make e2e-wan, which runs the split layout with every connection between management and its edges through a proxy that adds latency and loss: 80 ms round trip with 1 % loss, then 300 ms with 5 %.
What crosses the network
| From | To | What | Port |
|---|---|---|---|
| Each edge | Management | The registry's watch stream, the edge's reads and writes of management's tables, buffered metering, the bootstrap | Management's port (8443) |
| Management | Each edge | Plugin builds, the console's calls to the runtime API, Temporal's UI, artifacts | The edge API port (7443), or --advertise-to-management |
| People and the CLI | Management | The console, the admin API | Management's port |
Everything else stays on the edges' side: nodes to edges, edges to Temporal, edges to each other (for fencing), and edges to their payload storage. Management's own PostgreSQL stays next to management.
All of it is HTTPS with the cluster's certificates; the edges verify management and management verifies the edges. The edges check management's own name, management.oz0, which only management's certificate carries, whatever address they reach it at, and management checks that the edge answering at an address is the server whose signed row in the servers table names it, so a load balancer or a tunnel in between must pass TLS through rather than end it. Nothing needs a VPN, but the edges must reach management's port, and management must reach each edge's API port.
While management cannot be reached, an edge answers reads from its read cache for at most 24 hours (the kill switches for at most an hour), and never the reads that decide who may get in (keys, accounts, memberships, grants, revocations): see What an edge answers while management is away. A slow network is not an outage: a read that answers late is a read that answers.
When management reaches an edge at another address
An edge tells management and its nodes the same address for its API: --advertise and the edge API port. When management reaches the edge another way, through NAT, a load balancer or a tunnel, give that address to the edge:
oz0 server start --roles=edge --management-url https://mgmt.example.net:8443 \
--advertise 10.0.0.5 --advertise-to-management edge-a.example.net:7443 ...
Nodes keep using 10.0.0.5:7443; management dials edge-a.example.net:7443.
What it tolerates
Measured with make e2e-wan on one machine, two edges, three nodes and a fake model API. Times are from the start of the step to the end, and include the CLI's own calls to management across the same network.
| Step | 80 ms, 1 % loss | 300 ms, 5 % loss |
|---|---|---|
| Two edges start and are ready | 2.3 to 2.8 s | |
| An edge restarts and is ready | 8.7 to 9.6 s | |
| A node joins through an edge (token from management, join) | 0.8 to 2.2 s | 10 to 17 s |
| A plugin installs until the nodes run its agent | 1.7 to 1.9 s | 16 to 20 s |
| A job with two model calls, started to finished | 2.8 to 3.3 s | 18 to 29 s |
| One CLI call to management | 0.4 to 0.5 s | 1.3 to 1.7 s |
| A block, until the node's edge ends its stream | 0.4 to 0.5 s | 1.1 to 1.9 s |
| A block, until the CLI says it applied on 2 of 2 edges | 0.7 to 0.8 s | 2.2 to 3.9 s |
| Management cut off from one edge, until that edge fences itself | 7 to 10.5 s | |
| The cut heals, until that edge is fresh again | 3 to 4 s |
The ranges are three runs of the test on a busy machine; the first job after the nodes start takes longer at either profile, while their workers warm up.
Nothing failed at either profile: every job finished, every model call was metered exactly once, no edge fenced itself or called management unreachable while the network was only slow, and an edge cut off from management fenced itself, as it should, and was fresh again seconds after the cut healed.
What gets slower, and why
An edge sends each read and write of management's tables as one request, so a step costs one round trip for each statement it makes, and more when a packet is lost.
A job's own work (model calls, tools, Temporal) runs on the edges' side at full speed; what waits on the network is its bookkeeping: its owner and quota at the start, the budget and quota checks before each model call (read side by side, so one round trip), and its summary at the end.
At 300 ms and 5 % loss a job with two model calls takes 20 to 30 seconds instead of three.
Answers from management of more than about one packet go compressed, which halved what crosses the network.
An edge spreads its requests over four connections to management, each request on the one with the fewest under way, so a lost packet holds up the requests on one connection, not every request of the edge: at 300 ms and 5 % loss that took a node's join from about 30 to 8.5 seconds.
What does not get slower: running jobs never wait for management to answer, metering is buffered when management is slow or away, and nodes keep working. What an edge buffers goes to management in batches it can take across a slow link (at most 8 MiB, halved when management cannot take one), is applied once by its ID, and is never sent once it is older than management remembers applied writes; see the outbox's limits.
Limits
- A block is not within 2 seconds at every distance. A block takes effect once management's snapshot reaches the edge: one way across the network after management records it, plus a retransmission when a packet is lost, plus the CLI's own call to management.
At 300 ms and 5 % loss we measured 1 to 2 s to the node's edge, and up to 4 s until the CLI reports 2 of 2 edges, because it asks management across the same network. - Fencing needs a fresh registry every 10 s. Management sends a keepalive every 5 s on a connection of its own, so an edge falls behind only when its stream is silent for 10 s. A network that loses more than about 10 % of packets for minutes on end will make edges fence now and then; their nodes move to an edge that is fresh.
- An edge that stops tells management so, for up to 10 seconds. It tries three times within 10 s, so a lost answer does not leave management counting it. If none arrives (management is away, or the network is worse than that), management counts the edge, and
oz0 node blockreports it behind, until it has been silent for 10 minutes. - Management and its edges run the same version. The edges send SQL statements, so a management of one version cannot serve edges of another.
- Each statement's transaction is held across the network. A transaction an edge opens lasts as long as its statements' round trips, and management holds at most six of one edge's and ten of all edges' at once; it ends one that is idle for 30 s. Taking a lease holds none: it is a look and one statement.
Trying it
make e2e-wan needs PostgreSQL (make pg-up) and takes about four minutes. WAN_PROFILES="80ms:0.01 300ms:0.05" changes the profiles (round trip and loss; jitter is a tenth of the round trip), and WAN_KEEP_LOGS=<dir> keeps the processes' logs. The proxy, scripts/wanproxy, needs no root: it models what loss does to a TCP stream (a retransmission, the bytes behind it waiting, the sender's window halving) instead of dropping packets. scripts/wanproxy/calibrate.sh compares it with Linux's tc netem where unprivileged namespaces are allowed.