Manage nodes
For: cluster operators.
Inspect a node before changing its state. Use draining for maintenance, blocking to refuse access and purging to remove its local installation.
See the fleet
oz0 node list # every node
oz0 node list --state pending # pending, accepted, rejected, blocked or purging
oz0 node list --tenant acme
oz0 node show node-7f3a01b2c4d5 # one node in detail
Add --json to any of them for scripts.
A node's states
| State | Meaning |
|---|---|
pending | Joined without a token and waiting for an operator |
accepted | Allowed in; it connects and runs work |
rejected | Refused at approval |
blocked | Shut out; it keeps its files and can be unblocked |
purging | Removing itself from its machine on its next control stream, then forgotten |
An accepted node can also be draining or stopping (below). node list adds it to the state, such as accepted, draining, accepted, drained: safe to stop or accepted, stopping, and node show has a Drain line with when it began, its deadline, who drained it and what still runs there.
Block and unblock
oz0 node block node-7f3a01b2c4d5 --reason "lost laptop"
oz0 node unblock node-7f3a01b2c4d5
An edge normally applies a block within two seconds, provided it can read the registry and accepts the reload. It then closes the node's control stream and refuses its Temporal calls; the node's runtime stops. Jobs it was running continue on other nodes where they can. The reason goes to the audit log.
An edge keeps its previous registry when a reload looks broken, including a mass block. A cluster admin must review and confirm an intended change before that edge applies it. Check the command's acknowledgement for every edge.
The command waits up to 3 seconds (--wait) and says where the block applied, such as Block applied on 2 of 3 edges; edge-c behind for 40 s., naming each edge that has not applied it: an edge that cannot reach the database applies it when it can again, and until then is fenced if other servers are fine. Unblocking needs nothing on the node; it notices within ten seconds.
Drain a node
Draining a node takes it out of the work without stopping it, before a patch, a reboot or an upgrade:
oz0 node drain node-7f3a01b2c4d5 --deadline 2h --reason "kernel patch"
oz0 node show node-7f3a01b2c4d5
oz0 node undrain node-7f3a01b2c4d5
In the console, a cluster admin finds Drain in a node's Actions, which asks for the deadline and a reason, and Undrain once it drains. The Nodes page can show only the nodes that drain or stop.
While a node drains:
- It takes no new job and starts no new harness session, within a few seconds of the drain. The tenant's other nodes take them. On a tenant's only node, new work and the model steps of framework agents wait until the node takes work again. A session whose start was already on its way to the node when the drain began can still start there in the drain's first minute; it then runs as any other.
- What runs there finishes. A harness session that runs goes on to its end. The jobs whose home is the node keep running their tool calls there, wherever their model steps run.
- At the deadline it takes nothing at all. The node stops its runtimes, and what still ran there is lost; the node runs nothing until it is undrained. The deadline is two hours by default, from a minute to seven days. Drain again to move it.
node show says drained … safe to stop once no session, tool call or other step runs there and no job homed there is still open. The edge asks the cluster's workflow service which of those jobs still run, every ten seconds. The audit log records node.drain and node.undrain with who did it, and node.drained once, when nothing runs there any more. Draining and undraining are for cluster admins: a tenant's own admins can block or upgrade its nodes, but they ask a cluster admin to drain one.
Patch a node
To patch a node's machine and reboot it without losing work:
- Drain it with a deadline that covers its longest work:
oz0 node drain NODE --deadline 2h. - Wait for
safe to stopinoz0 node show NODE(or the badge on the Nodes page). Past the deadline it saysdrainedtoo, but what still ran there was lost. - Patch the machine and reboot it. The node's service starts it again; it stays drained.
- Undrain it:
oz0 node undrain NODE. It takes work again at once.
Stops drain first
A node also drains by itself when it is stopped: on SIGTERM, a service stop, a pod's termination or a node binary update. It takes no new work, waits for the sessions, tool calls and other steps running there to end, for at most its stop grace, and then stops its runtimes. A second signal stops it without waiting.
A stop waits only for what is in flight on the node, and needs neither the server nor the workflow service to answer, so a machine that shuts down its server first still stops at once when its node is idle.
A job homed on the node that runs nothing there at that moment, such as one waiting for a person's approval or for a model step on another node, does not hold the stop: its next tool call waits for the node to come back. Before you take a node away for good, drain it with oz0 node drain, which waits for those jobs too.
The stop grace is ten minutes by default: long enough for most tool calls and short sessions, short enough that no machine waits long to shut down. Set it with oz0-node run --stop-grace 5m (or OZ0_STOP_GRACE; at most an hour), and for a service with oz0-node service install --stop-grace 5m, which gives the service manager the same grace and a minute more. The runtime's own stop, at most 30 seconds, is inside the grace. A session longer than the grace is lost when the grace ends; drain the node with a deadline for planned work.
oz0-node drain --wait asks the node that runs on the machine to drain for a stop, without stopping it, and returns once nothing runs there or its grace is ending. The stop that follows waits only for what is left of the same grace. A Kubernetes pod's preStop hook runs it (Kubernetes).
Remove a node
oz0 node delete node-7f3a01b2c4d5
delete forgets the node. Its key may join again, as a new node. The node's files stay on its machine.
Purge a node
Purging makes a node remove itself from its machine. It cannot be undone:
oz0 node purge node-7f3a01b2c4d5 --yes
In the console, it is Purge in a node's Actions, which asks first.
- The node stops getting work at once, as a blocked node does.
- On its next control stream it gets a purge command signed with the cluster CA's key. A blocked node gets it too, and an offline node gets it when it comes back.
- The node checks the signature against the CA certificate it holds, and that the command names its own cluster and ID. A command that fails the check is refused, and the audit log records
node.purge.refused. - It answers the edge, which forgets the node (
node.purgedin the audit log), and then it stops its runtimes and plugins and deletes its whole data directory (its identity and keys, plugins and their environments, runtimes, the managed Python, its tools such as uv and Node.js, caches and logs; a link in it goes, not what it points to) and its binary, with the copies self-update keeps beside it. A directory that cannot go itself, such as a mount point, stays empty but for a marker that refuses a join with it.
To join again, the machine must be set up again. The node deletes its data directory only if it holds a node identity, and follows no symbolic links out of it. A node that never comes back stays purging; node delete forgets it.
Join tokens
oz0 token create --tenant default --ttl 24h --max-uses 5 --label team=data --description "data team laptops"
oz0 token list
oz0 token revoke <token-id>
A token is bounded: by default one use and 24 hours; --max-uses takes 1 to 1,000 and --ttl at most 720h (30 days). --ttl 0 and --max-uses 0 are refused: a token is never unlimited and never lasts forever.
A token made before these bounds with no use limit or no expiry admits nobody now; oz0 token list shows it as unbounded, and the join it refuses names the commands that replace it.
Revoking a token stops new joins; nodes that already joined with it stay. In the console, the tenant's admins find the tokens on the Join tokens tab of Fleet, Nodes (a cluster admin's are every tenant's), revoke one there, and make one with Add nodes.