--- title: "Node health" description: "See which nodes are connected, what they run and what state they are in." canonical_url: "https://docs.oz0.ai/fleet/nodes" --- # Node health > See which nodes are connected, what they run and what state they are in. For: cluster operators. Use findings and health scores to identify why a node cannot take work. Check plugin readiness alongside its connection state. ## What you can see today ```bash [Terminal] oz0 node list # every node with its state oz0 node show # one node in detail oz0 plugin list # each node's state with each plugin oz0 catalog # which nodes are ready to run each agent oz0 cluster info # the servers that are up ``` `cluster info` and the console's overview show every server's name and version to the members of a tenant. Where each one listens (its advertised address and endpoints) is the shared cluster's own layout, so only cluster admins and readers, and operators' certificates, see it; for a tenant's people the column shows `-`. An account that is in no tenant and holds no cluster role sees no servers. A server's name is its host's name unless `oz0 server start --name` sets one, and on a cloud host that name often spells the machine's private address: give servers names of their own when a tenant's people should not learn it. Nodes keep a control stream open to the edge and send heartbeats over it. A node that is blocked, rejected or deleted has its stream closed within two seconds. `node show` and the Nodes page also show each node's [sandbox tier](https://docs.oz0.ai/concepts/security/sandbox-tiers#sandboxes): `tier 1 (bubblewrap 0.9.0)`, or `tier 0` with the reason, such as bubblewrap missing. The catalog lists, for each agent, its nodes with tier 1 and whether its webhook jobs need them. ## Findings Beside its health score, a node can have findings: problems it reports that cost no points and do not change where work goes, each with what to do about it. The Nodes page shows them as a warning beside the score and, with the fix, in the node's dialog (the status filter has With findings); `oz0 node show` prints each as a `Finding` line, the API carries them in `Node.findings`, and [`oz0 doctor`](https://docs.oz0.ai/fleet/doctor) sums them up as `node sandboxes`.
Finding When The fix it shows
sandbox tier 0 The node runs tier 0 : shell and write harness sessions go to tier-1 nodes instead, unless a tenant admin opted this node out, and every job an outside event starts, whatever the opt-out For what the node found: install bubblewrap, load Ubuntu's own AppArmor profile ( sudo oz0-node sandbox setup , see Sandbox tier 1 on Linux ), rewrite an old service unit, a sysctl , drop --sandbox off ; on Windows, what its probes for Windows 11's process security environment found, and that no Windows build runs tier 1 yet ( Sandbox on Windows )
sandbox opt-out A tenant admin let the node run shell and write harness sessions without the tier-1 sandbox ( oz0 node sandbox-optout ), and it holds that: who, when and why. Jobs that ran such a session there show tier 0 and the node Give the node sandbox tier 1, then end the opt-out: oz0 node sandbox-optout <node> --remove --reason ...
floor queues The node has sandbox tier 1 or the opt-out, but its node binary is from before shell and write sessions ran on tier 1 by default, so the servers give it none of them: it would not record where such a session ran without a sandbox Upgrade the node's binary: oz0 node upgrade --binary <node> , or install the current oz0-node and restart it
bubblewrap advisory The node's bubblewrap has a known vulnerability: today CVE-2026-87766 Update bubblewrap; for a distribution's package, the state of its fix
gVisor The node's container runtime lists gVisor's runsc , but a test container the node ran under it at start failed, or a tool's container later failed under it and a second test container failed too: tools that ask for gVisor go to other nodes For what failed: a newer runsc, or the second runsc-oz runtime on old Docker (runsc refused the host-uds annotation, or the test container could not reach the node's socket); runsc install again (the engine does not know the runtime); the engine's runtime list (it ran another runtime); or a container under runsc by hand to read the error. Then restart the node
A tier-0 node still takes work, but no shell or write harness session unless it holds the [sandbox opt-out](https://docs.oz0.ai/fleet/nodes#the-sandbox-opt-out): Windows nodes run tier 0 by design, and a vulnerable bubblewrap still sandboxes far better than none. `oz0 doctor` warns only about findings you can act on (a tier-0 Linux or macOS node with a fix); a node run with `--sandbox off`, a Windows node, a node in a container and the advisory are named without a warning. For a node in a container (Docker, Podman, or a Kubernetes pod such as the Helm chart's), the fix says to run the node on the machine itself; a unit from before this release gets a fix that rewrites that unit, with its data directory and service name. The node reports what it found as facts (which fix applies, its package manager, its unit and data directory), and management words the fix itself: it never shows text a node sent, since on a tier-0 node an agent runs as the node's account and could report anything. A unit or data directory whose name does not contain `oz0` or `orchestrator-zero` is left out, and the fix then says which flags to give. Fixes that say `sudo` name the bare `orchestrator-zero-node`, which sudo finds where install.sh put it; run them as written, never with the copy of the binary in a node's data directory, which the node's account can replace. On the node, `oz0-node sandbox check` and `oz0-node doctor` give the full command with the node's own unit and data directory. ## The sandbox opt-out Every harness session whose agent can run a shell or write files runs on a node with sandbox tier 1, whoever started the job ([Which tier runs what](https://docs.oz0.ai/concepts/security/sandbox-tiers#which-tier-runs-what)). A job with no such node fails after 10 minutes and names the fixes; it never runs without a sandbox instead. Where a node cannot have tier 1 and must run such sessions all the same, a tenant admin lets that one node do so, with a reason: ```bash [Terminal] oz0 node sandbox-optout build-win-01 --reason "the Windows build box runs only our own repositories" oz0 node sandbox-optout build-win-01 --remove --reason "it has sandbox tier 1 now" # back to tier-1 nodes only, at once ``` It takes `governance.write` in the node's tenant and is a governance change: with [four eyes](https://docs.oz0.ai/govern/reviewed-changes) on for loosenings, setting it files a change request another admin approves, and the node is unchanged until then; ending it acts at once. Each change is audited as `tenant.sandbox_optout` with the node and the reason. While the node holds it: - it is a `sandbox opt-out` [finding](https://docs.oz0.ai/fleet/nodes#findings), and `oz0 doctor`'s `node sandboxes` line warns; - `oz0 node show` prints a `Sandbox opt-out` line (who, when, why), and the Nodes page shows opted out beside its sandbox and the same in its dialog, read-only; - each job that ran a shell or write session there shows tier 0 and the node on its page and in `oz0 job get`, and the audit log has a `run.sandbox.tier0` entry for it, which the [evidence bundle](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#evidence-bundles) carries. The entry says what allowed it (`allowed_by`): `optout`, with the opt-out in force then (who set it, when, why and the decision it follows), or `dev_server`; when the node no longer holds that opt-out by the time the entry is written, `unknown`, and the node's `tenant.sandbox_optout` entries before the job's end name it. The opt-out covers jobs a person or an app started. A job an outside event started (a webhook, a routine's fire, a chat message) never runs a shell there: it still needs a real tier-1 node. There is no tenant-wide opt-out. On the node, a session's shell has the node user's files, network and rights, as any program that user runs. ## Health scores The edge that holds a node's control stream scores the node at every heartbeat, every ten seconds, from 0 to 100: ```text score = 100 − resources − failures − latency − model calls − restarts − lost sessions ```
Penalty Up to Grows with
Resources 25 The busiest of CPU, memory, disk and GPU memory over 80%, smoothed over a few heartbeats
Failures 30 The share of the node's activities in the last 15 minutes that failed because of the node or a plugin, over 2%. A tool that answers with an error, a model that is overloaded or a job out of budget is not the node's failure.
Latency 15 How much slower the node runs a type of activity than the cluster's median, from 1.5 times; at least 20 runs
Model calls 10 The share of its model calls in 15 minutes that got 429 or 5xx, or never reached the provider, over 5%. A 429 that says the provider account's money or quota is spent ( SpendLimitReached ) is the account's, not the node's, and does not count
Restarts 20 Restarts of its runtime in 15 minutes; three cost everything
Lost sessions 25 Harness sessions lost with the node in 15 minutes; three cost everything, which makes the node degraded
State Score What happens
Healthy 80-100 Nothing
Degraded 50-79 The node takes half as many activities at once until it recovers. The audit log records node.degraded .
Unhealthy 0-49 The node gets no new tasks and drains: its polls come back empty, and what it runs finishes as usual. The audit log records node.unhealthy .
Offline : No heartbeat for 30 seconds. Its activities time out and run elsewhere.
A disk over 95% full, or a runtime that has been down for 30 seconds, makes a node unhealthy whatever its score. A state changes after two heartbeats in a row agree, and back to healthy takes three. ```bash [Terminal] oz0 node list # HEALTH: state and score oz0 node show # the reason and every penalty oz0 audit --action node. # when nodes changed state, and why ``` ```text Health unhealthy 75: disk 97% full Penalties resources 25.0, failures 0.0, latency 0.0, model calls 0.0, restarts 0.0 (scored 3s ago) ``` The console's *Nodes* page shows the same, with the penalties on the badge's tooltip. A tool with `affinity: node` runs only on its job's home node, so its calls wait while that node is unhealthy. Each server answers `GET /healthz` on its edge port (7443) and its management port (8443), for load balancers and monitoring. The edge port also answers `GET /readyz`: 503 while the server's edge is [fenced](https://docs.oz0.ai/fleet/high-availability#when-a-server-loses-postgresql-fencing), so a load balancer in front of the edges stops sending it work, and 200 otherwise, with the state in the `Oz0-Edge-State` header. ## Local models A node that runs model servers reports them in its heartbeats: each server's state (`ready`, `down` or `refused`, with the reason), the models it serves, its kind and version, how many calls it runs at once and the slots the node holds it to, the model APIs it serves, and how often the node started it again after it stopped answering (`oz0 node show` prints a `Model server` line for each). The Nodes page shows them when you open a node, and a ready server's models label the node `model.=ready`, which routes agents on `local/` to it. The node's log says each change (`model server model_server=ollama state=ready models=[qwen3-coder]`). See [Local models](https://docs.oz0.ai/fleet/local-models). ## Logs Servers and nodes log to stderr. Raise the level when you need detail: ```bash [Terminal] oz0 server start --log-level debug ... oz0-node run --log-level debug ... ``` Temporal's own logs are quieter by default; set `--temporal-log-level` on the server to see them. The edge tells the node its verdict on the control stream, and the node's log says what it does about it (`health verdict state=degraded ...`). Planned: logs that stream from nodes to the edge. Until then, read a node's log on its machine. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.