---
title: "Manage nodes"
description: "List, inspect, accept, block, drain, purge and remove nodes, patch them without losing work, and manage join tokens."
canonical_url: "https://docs.oz0.ai/fleet/manage-nodes"
---
# Manage nodes

> List, inspect, accept, block, drain, purge and remove nodes, patch them without losing work, and manage join tokens.

For: cluster operators.

Inspect a node before changing its state. Use draining for maintenance, blocking to refuse access and purging to remove its local installation.

## See the fleet

```bash [Terminal]
oz0 node list                    # every node
oz0 node list --state pending    # pending, accepted, rejected, blocked or purging
oz0 node list --tenant acme
oz0 node show node-7f3a01b2c4d5  # one node in detail
```

Add `--json` to any of them for scripts.

## A node's states

<table>
<thead>
  <tr>
    <th>
      State
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        pending
      </code>
    </td>
    
    <td>
      Joined without a token and waiting for an operator
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        accepted
      </code>
    </td>
    
    <td>
      Allowed in; it connects and runs work
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        rejected
      </code>
    </td>
    
    <td>
      Refused at approval
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        blocked
      </code>
    </td>
    
    <td>
      Shut out; it keeps its files and can be unblocked
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        purging
      </code>
    </td>
    
    <td>
      Removing itself from its machine on its next control stream, then forgotten
    </td>
  </tr>
</tbody>
</table>

An accepted node can also be draining or stopping (below). `node list` adds it to the state, such as `accepted, draining`, `accepted, drained: safe to stop` or `accepted, stopping`, and `node show` has a `Drain` line with when it began, its deadline, who drained it and what still runs there.

## Block and unblock

```bash [Terminal]
oz0 node block node-7f3a01b2c4d5 --reason "lost laptop"
oz0 node unblock node-7f3a01b2c4d5
```

An edge normally applies a block within two seconds, provided it can read the registry and accepts the reload. It then closes the node's control stream and refuses its Temporal calls; the node's runtime stops. Jobs it was running continue on other nodes where they can. The reason goes to the audit log.

An edge keeps its previous registry when a reload [looks broken](https://docs.oz0.ai/fleet/high-availability#a-reload-that-looks-wrong-is-not-applied), including a mass block. A cluster admin must review and confirm an intended change before that edge applies it. Check the command's acknowledgement for every edge.

The command waits up to 3 seconds (`--wait`) and says where the block applied, such as `Block applied on 2 of 3 edges; edge-c behind for 40 s.`, naming each edge that has not applied it: an edge that cannot reach the database applies it when it can again, and until then is [fenced](https://docs.oz0.ai/fleet/high-availability#when-a-server-loses-postgresql-fencing) if other servers are fine. Unblocking needs nothing on the node; it notices within ten seconds.

## Drain a node

Draining a node takes it out of the work without stopping it, before a patch, a reboot or an upgrade:

```bash [Terminal]
oz0 node drain node-7f3a01b2c4d5 --deadline 2h --reason "kernel patch"
oz0 node show node-7f3a01b2c4d5
oz0 node undrain node-7f3a01b2c4d5
```

In the console, a cluster admin finds Drain in a node's Actions, which asks for the deadline and a reason, and Undrain once it drains. The Nodes page can show only the nodes that drain or stop.

While a node drains:

- It takes no new job and starts no new harness session, within a few seconds of the drain. The tenant's other nodes take them. On a tenant's only node, new work and the model steps of framework agents wait until the node takes work again. A session whose start was already on its way to the node when the drain began can still start there in the drain's first minute; it then runs as any other.
- What runs there finishes. A harness session that runs goes on to its end. The jobs whose home is the node keep running their tool calls there, wherever their model steps run.
- At the deadline it takes nothing at all. The node stops its runtimes, and what still ran there is lost; the node runs nothing until it is undrained. The deadline is two hours by default, from a minute to seven days. Drain again to move it.

`node show` says `drained … safe to stop` once no session, tool call or other step runs there and no job homed there is still open. The edge asks the cluster's workflow service which of those jobs still run, every ten seconds. The audit log records `node.drain` and `node.undrain` with who did it, and `node.drained` once, when nothing runs there any more. Draining and undraining are for cluster admins: a tenant's own admins can block or upgrade its nodes, but they ask a cluster admin to drain one.

## Patch a node

To patch a node's machine and reboot it without losing work:

1. Drain it with a deadline that covers its longest work: `oz0 node drain NODE --deadline 2h`.
2. Wait for `safe to stop` in `oz0 node show NODE` (or the badge on the Nodes page). Past the deadline it says `drained` too, but what still ran there was lost.
3. Patch the machine and reboot it. The node's service starts it again; it stays drained.
4. Undrain it: `oz0 node undrain NODE`. It takes work again at once.

## Stops drain first

A node also drains by itself when it is stopped: on SIGTERM, a service stop, a pod's termination or a node binary update. It takes no new work, waits for the sessions, tool calls and other steps running there to end, for at most its stop grace, and then stops its runtimes. A second signal stops it without waiting.

A stop waits only for what is in flight on the node, and needs neither the server nor the workflow service to answer, so a machine that shuts down its server first still stops at once when its node is idle.

A job homed on the node that runs nothing there at that moment, such as one waiting for a person's approval or for a model step on another node, does not hold the stop: its next tool call waits for the node to come back. Before you take a node away for good, drain it with `oz0 node drain`, which waits for those jobs too.

The stop grace is ten minutes by default: long enough for most tool calls and short sessions, short enough that no machine waits long to shut down. Set it with `oz0-node run --stop-grace 5m` (or `OZ0_STOP_GRACE`; at most an hour), and for a service with `oz0-node service install --stop-grace 5m`, which gives the service manager the same grace and a minute more. The runtime's own stop, at most 30 seconds, is inside the grace. A session longer than the grace is lost when the grace ends; drain the node with a deadline for planned work.

`oz0-node drain --wait` asks the node that runs on the machine to drain for a stop, without stopping it, and returns once nothing runs there or its grace is ending. The stop that follows waits only for what is left of the same grace. A Kubernetes pod's preStop hook runs it ([Kubernetes](https://docs.oz0.ai/fleet/kubernetes#node-pods-drain-before-they-stop)).

## Remove a node

```bash [Terminal]
oz0 node delete node-7f3a01b2c4d5
```

`delete` forgets the node. Its key may join again, as a new node. The node's files stay on its machine.

## Purge a node

Purging makes a node remove itself from its machine. It cannot be undone:

```bash [Terminal]
oz0 node purge node-7f3a01b2c4d5 --yes
```

In the console, it is Purge in a node's Actions, which asks first.

1. The node stops getting work at once, as a blocked node does.
2. On its next control stream it gets a purge command signed with the cluster CA's key. A blocked node gets it too, and an offline node gets it when it comes back.
3. The node checks the signature against the CA certificate it holds, and that the command names its own cluster and ID. A command that fails the check is refused, and the audit log records `node.purge.refused`.
4. It answers the edge, which forgets the node (`node.purged` in the audit log), and then it stops its runtimes and plugins and deletes its whole data directory (its identity and keys, plugins and their environments, runtimes, the managed Python, its tools such as uv and Node.js, caches and logs; a link in it goes, not what it points to) and its binary, with the copies self-update keeps beside it. A directory that cannot go itself, such as a mount point, stays empty but for a marker that refuses a join with it.

To join again, the machine must be set up again. The node deletes its data directory only if it holds a node identity, and follows no symbolic links out of it. A node that never comes back stays `purging`; `node delete` forgets it.

## Join tokens

```bash [Terminal]
oz0 token create --tenant default --ttl 24h --max-uses 5 --label team=data --description "data team laptops"
oz0 token list
oz0 token revoke <token-id>
```

A token is bounded: by default one use and 24 hours; `--max-uses` takes 1 to 1,000 and `--ttl` at most 720h (30 days). `--ttl 0` and `--max-uses 0` are refused: a token is never unlimited and never lasts forever.

A token made before these bounds with no use limit or no expiry admits nobody now; `oz0 token list` shows it as unbounded, and the join it refuses names the commands that replace it.

Revoking a token stops new joins; nodes that already joined with it stay. In the console, the tenant's admins find the tokens on the Join tokens tab of Fleet, Nodes (a cluster admin's are every tenant's), revoke one there, and make one with Add nodes.


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.
