Operate through role outages

For: cluster operators.

Review tested outage outcomes and the limits of caches, snapshots and buffered writes.

What happens when something is down

We test each of these with make e2e-roles: one management, two edges, one Temporal and two nodes on PostgreSQL. make e2e-roles-backends runs the same layout with two Temporal backends, a tenant on each, an S3-compatible bucket instead of a payload database, and a tenant moved while a long job runs.

Management unavailable

Running and scheduled jobs carry on.

Apps start new jobs through the edges with their API keys; a start under a job ID the app chooses is refused as unavailable until management is back, since management keeps whether the ID is a conversation's turn, whose files the job would read, so leave the ID out.

A job whose authority envelope an edge does not hold (one started meanwhile through another edge) runs its model calls and its own tools through it, and waits for management to start a child job, continue as new or call another plugin's tool; a scheduled fire runs when its first task reaches an edge during the outage, and otherwise waits for management.

An edge that gets slow or busy answers from management is not cut off from it: what needs an envelope it does not hold waits, and a start, child start or fire whose envelope management does not take is refused as unavailable and tried again.

Joins, renewals, installs and the console wait.

An edge that restarts starts from the snapshot and the cluster record in its data directory.

What the edges record meanwhile (model calls and their cost, quota charges, audit entries, nodes' heartbeats) waits in their outboxes, on disk, and reaches management once it is back, each record once.

The outage test recorded this result: A running job finished, the routine's schedule fired, the app started three jobs and was refused one under its own ID, an edge restarted, and every model call was metered once afterwards.

One edge unavailable

Its nodes reconnect to another edge within seconds and carry on.

The outage test recorded this result: The node moved to the other edge and ran work there.

PostgreSQL, for management unavailable

Management stops serving (/readyz answers 503). Edges serve from their cache, buffer metering, and catch up when it is back.

With payload encryption on, the default, each edge goes on with the tenants' payload keys it already opened, for up to 24 hours; a tenant whose key the edge never opened starts no job until the database is back.

With the embedded Temporal on the same PostgreSQL, Temporal stops too, and everything waits until it is back.

The outage test recorded this result: The test cuts only management's connection: a job ran through an edge, and the edges caught up within seconds of the database's return.

Management, for one edge only unavailable

That edge falls behind, sees that another edge is fresh, and fences itself; its nodes move. A block recorded meanwhile applies on the other edges at once, and on that edge as soon as it reaches management again. A job that edge started meanwhile waits on the other edges until its authority envelope reaches management.

The outage test recorded this result: The edge fenced itself 6 to 11 s after the cut, runtime API calls made through management went to the other edge, and the block applied there seconds after it reconnected.

While management is away, an edge cannot see what the other edges spend:

• limits on jobs at once and jobs a day are not checked;
• monthly quotas and budgets are checked against what the edge last read;
• a job's owner is recorded when management is back.

A job's cost shows 0 until then. The edge logs this once a minute.

The tool policy is written only by management. When a node first lists a plugin version's functions, the edge records the listing and leaves a note for management, which within a few seconds gives a function whose type rose its stricter default back and narrows the plugin's suggested rules to what the node lists (updates and rising types). While management is away the edge cannot record the listing; it tries again at the node's next heartbeat, and the policy stays as the install left it, so nothing is loosened meanwhile.

What an edge answers while management is away

An edge keeps management's last answer to each read in its read cache, on disk, and answers from it while management cannot be reached, for at most 24 hours: past that a read fails as one never made, since no node certificate issued before the outage is still valid. Each answer from the cache is logged with its age and counted in oz0_edge_read_cache_reads_total (by outcome) and oz0_edge_read_cache_answer_age_seconds.

Some reads are never answered from the cache, because an earlier answer could let in someone management has shut out since: API keys and apps, join tokens, accounts, their sign-in identities and OIDC issuers, operators, memberships, teams' members, role sets, chat channels' members, grants (MCP and plugin uses, plugins' read grants and the results targets they name), credential bindings, chat shares, stream tickets, link codes, nodes' standing (accepted or blocked), and the authority revocations new jobs are stamped from.

While management is away an edge authorizes a key or a node only from its signed registry snapshot: a key made or revoked since it was taken waits for management, and a plugin's read grant is refused until management answers.

Live state (servers, leases, nodes' connections, and which authority envelope a job runs on) is not cached either.

Kill switches and the models' vettings are: the edge reloads them whole every few seconds and keeps the last ones when management is away, so an edge that restarts meanwhile starts with the last ones it read. Kill switches are answered for at most an hour, and an edge that has not read them since it started starts no new job (the refusal says so); jobs running go on under the switches it knows.

Nothing that writes is answered from the cache, nor an answer that carried an error.

Snapshots and long outages

Management signs each registry snapshot with the cluster CA, through the CA signing command when the key is outside the server. A snapshot is good for an hour, and management sends a new one every 15 minutes. When management stays away longer, an edge keeps serving from the snapshot it holds and logs an error every hour. Node certificates are what limit the outage: they last 24 hours, and only an edge that can reach management renews them. An edge that starts while management is away accepts a snapshot that expired less than 24 hours ago.

An edge refuses a snapshot whose signature fails, and one older than the snapshot it holds (a replay could unblock a node). It logs the refusal as an error and reports it to management, whose registry status names it (edge-b refused it (...)) and does not count that edge as having applied the version; and it counts as behind, not fresh, until it applies a snapshot, so it stops admitting nodes and renewing their certificates, and fences itself when another edge is fresh.

After you restore management's database from a backup, its registry is older than what the edges hold, so every edge apart refuses it: stop each one, delete registry.snapshot from its --data-dir, and start it again, and it takes management's registry as it is now.

Limits

• The outbox holds up to 1,000,000 writes or 512 MiB per edge, and one write of at most 16 MiB. Past that, new writes are refused and logged.
• Management remembers what it applied for 30 days. An edge drops writes older than 29 days, when it starts and while it runs, rather than risk applying one twice, and management refuses one that old too.
• What the outbox reports as on disk, such as the intent entry a tool call writes before it goes on, is in a file whose sync succeeded, and whose entry in its directory was synced too. When a sync or a write fails, the writes since are written again into a new file, and until that works such a write fails instead of being taken for kept.
  While the disk keeps failing, the edge tries again after a wait that doubles up to 30 seconds. A file the edge cannot read in full when it starts is moved to outbox/damaged under its data directory, with every write in it that can be read kept and the lines that cannot counted in the log.
• Buffered writes go to management in batches of at most 256 writes and 8 MiB, halved when management cannot take one.
  A write management itself says it cannot take, sent alone (too large for it, or not JSON to it), is moved to outbox/dead.jsonl and logged, so it holds up none of the writes behind it; only that write, on management's word about it.

Each is synced into that file before it leaves the outbox; when the file is new, the directory is synced too, and tried again at every sync until that succeeds.

A write refused with no such word on it (a body that could not be read, or a proxy in front of management that answers 400 or 413, such as a limit on a body's size) is kept, never set aside: it waits apart and is sent again on its own, after a wait that doubles from a second up to five minutes, while the writes after it go on in batches and reads go on as before.

Later rows of the same job wait with it, so one job's rows still reach management in the order they were sent.

Up to 16 writes wait apart with waits of their own.

Writes refused past those wait apart too, so none of them holds up the writes behind it; they are sent again one at a time, in turn, every five minutes, and the oldest takes a wait of its own when one of the 16 leaves.

After a write is set apart, the next batch is a single write and batches double back to full ones, so a later write over the same limit costs one refusal.

When such refusals come one after another with no write taken between them, the edge sends the smallest write waiting on its own: when even that one is refused, as a proxy that refuses every write would, delivery pauses (from a quarter of a second, doubling up to five seconds) until a write is taken, while reads go on.

A write waiting apart is logged as an error after ten minutes, then every hour, until it is taken or is older than management remembers applied writes (29 days).

oz0_edge_outbox_held_writes, oz0_edge_outbox_held_oldest_seconds and oz0_edge_outbox_held_past_limit_writes count them for an alert; when they climb, raise the limit on a body's size of the proxy in front of management to 32 MiB, what management reads in one request, and the writes waiting apart reach management within five minutes.

Each write's result is matched to it by its ID, and a write management refuses is logged with its ID.

• The internal API passes SQL statements, so management and its edges must run the same version.
• Management may run far from its edges, across a slow or lossy network: see Management over a WAN for what it needs and what gets slower.
• Management apart from Temporal serves Temporal's console through an edge: /temporal/ on the management port signs you in as before (admins only) and hands each request to a live edge, with who you are signed by management, and the edge runs the UI against its Temporal backend. Management still never connects to Temporal. With Temporal Cloud the page says where the Cloud UI is, as one server's does.