Doctor

For: cluster operators.

Run the check from the machine whose access you need to test. Follow the reported fix, then run it again to confirm the problem is resolved.

oz0 doctor checks the setup your CLI context points at, from the machine you run it on. oz0-node doctor checks a node from the machine the node runs on. Each check says what it found, and each one that is not ok says how to fix it.

oz0 doctor

ok    CLI context                  cluster dev, namespace default
ok    operator certificate         valid until 2027-01-05 (89 days)
ok    edge Temporal port           127.0.0.1:7233 takes connections (1ms)
ok    edge API port                127.0.0.1:7443 takes connections (0s)
ok    edge certificate             127.0.0.1:7443's certificate is from the cluster's CA, valid until 2026-11-06 (29 days)
ok    clock                        within a second or two of the server's
ok    management                   https://127.0.0.1:8443 answers
ok    database                     SQLite answered a ping in 0s
ok    migrations                   schema version 74, as this server expects
ok    servers                      1 up: dev
ok    cluster CA                   valid until 2036-10-04
ok    nodes                        1 of 1 online
ok    node certificates            every node's is renewed in time
ok    node sandboxes               1 of 1 nodes reported, every one at tier 1 with no known advisory
ok    temporal                     namespace default answered in 1ms
warn  model keys                   tenant default has no model key, so its agents' calls to cloud models are refused (MissingKey)
                                   fix: store one: printf %s "$ANTHROPIC_API_KEY" | oz0 secret set ANTHROPIC_API_KEY --tenant default; or serve a local model on a node (models.yaml, see Local models)
ok    disk                         /home/you/.config/orchestrator-zero: 366.4 GB free of 1006.9 GB (64% used)
17 checks: 16 ok, 1 warnings, 0 failed, 0 skipped

What it checks

From this machine:

• The CLI context and the operator certificate: that there is a context, and that its certificate is valid and not about to end.
• The edge's ports: that the Temporal port (7233) and the API port (7443) take connections from here. Nodes need both; operators need the API port.
• The edge's certificate: that the API port answers TLS with a certificate of the cluster's CA, for the address you use, that is not about to end.
• The clock: this machine's clock against the edge's. Certificates and Temporal's timeouts depend on it; more than 30 seconds apart fails.
• Disk: the free space where the context lives, or a server's --data-dir.

Management's own checks, through the admin API. Only cluster admins may run them; for anyone else doctor skips them, which does not count as a failure:

• The database and its migrations: that PostgreSQL (or SQLite in dev mode) answers, how fast, and that its schema is the one this server expects.
• The servers: which ones report in, and which have not for a minute.
• The cluster CA, and the nodes: how many are online, which are not healthy, and whether each node's certificate was renewed in time. A running node renews its own at two thirds of its life.
• The keys and backups a recovery needs (a cluster; a dev server skips them): master key backup warns until someone ran oz0 server master-key confirm-backup (without the master key everything sealed under it is lost, encrypted histories included); secrets key backup, with the roles apart, until oz0 server secrets-key confirm-backup recorded the key the cluster seals to; database backup while the newest recorded backup is older than the newest tenant payload key, whose histories a database loss would leave unreadable.
  payload encryption warns when an edge runs with --encrypt-payloads=false, and temporal retention when a namespace does not keep what its tenants' retention wants, with the command that fixes it (on Temporal Cloud, tcld namespace retention set).

keys sealed by the edges, with the roles apart, warns until an edge has recorded that every data subject's key and payload key is sealed by an edge; until then the edges that were there before still re-seal older keys, for a day from their start at most, and their logs count the keys they pass over because they open with no key.

master key source and secrets key source say which running servers read each key from a plain file and which through a key command, and warn when a server's last unwrap failed or its command cannot reach the wrapping key, or while the plain copy that wrapping keeps (<file>.plain) is still beside a server's wrapped key file.

erasure ledger reads the whole erasure ledger as a restore would and fails when a line does not verify, when the file cannot be read, or when it no longer holds the newest erasure the database says it recorded; erasures are refused until it is fixed.

A tenant's members see the same lines for their tenant, as yes or no without key IDs, holders or server names, under Settings, Encryption or with oz0 tenant encryption <tenant> (encryption status).

• Shared pools (while the cluster shares pools): shared pools warns about each accepted node that cannot poll them, because its supervisor predates them or because every runtime it starts or runs does, with the fix (oz0 node upgrade --binary or --runtime, or --queue-pools=plugin until then).
  fair dispatch names the backends that are an external Temporal cluster or Temporal Cloud, where a shared pool's plugins take turns only if fair dispatch is on there (matching.enableFairness, or the namespace's fairness): management cannot read that setting, so the line is a skipped check that says what to look at, and leaves the exit code alone (Upgrades).
• Harness sessions: how many harness sessions nodes lost in the last day and how many finished within a drain; it warns for each node that lost three or more, with what became of them (restarted, waiting for a person, failed) and what they spent, and suggests draining the node before planned work and looking at why it stops.
• Secrets past their expiry (secret_overdue): each tenant's secrets whose expiry has passed, with the fix: rotate them, or move the expiry. It warns; the secrets keep working.
• The nodes' sandboxes: which accepted nodes run sandbox tier 0, and which run a bubblewrap with a known advisory, from the nodes' findings.
  It warns only about what you can fix on a node: a Linux or macOS node at tier 0 with a fix. A node run with --sandbox off, a Windows node, a node in a container (the quickstart's, or a Helm chart's pod) and a bubblewrap advisory no distribution has fixed yet are named in the detail of an ok check, so doctor stays green, and exits 0, on a setup that is as good as it can be. When the nodes to fix share a fix, the check shows it; otherwise oz0 node show <node> has each one's.

The edge's checks for the tenant (--tenant), through the runtime API. Only the tenant's admins may run them; anyone else sees a skip:

• Temporal: that the tenant's namespace answers, on whichever backend it is (embedded, an external cluster or Temporal Cloud).
• Secrets past their expiry (secret_overdue): the tenant's own, as for cluster admins above.
• Model keys: each key the tenant has, checked with the provider's list of models; for a pooled anthropic or openai record, each member's key, at the member's endpoint. That spends no tokens. --skip-model-keys leaves it out. A record that signs in rather than sending its key in a header (OAuth2 or SigV4, see Models and keys) is not checked: its secret goes only to its token URL, or into a signature.

The tenant's tool policy (--tenant), through the admin API. Anyone who may read the tool policy may run it (the tenant's readers and admins); anyone else sees a skip:

• The tool policy (tool policy): the policy lint, one warning per finding, named tool policy rule <id>, each with the command that fixes it: a rule an earlier one hides, a pattern that matches no function, a condition on an argument the function does not take, a condition on tool.class without its host form, a shadow rule never turned on, and a rule unused for 90 days.
  With no finding it is one ok line that says how many rules it checked against how many functions. A warning here makes doctor exit 2, as any warning does; the policy works as it is.

The tenant's sandbox floor (--tenant), through the admin API, for anyone who may read the tenant's catalog and nodes:

• The sandbox floor (sandbox floor): whether the tenant's agents that can run a shell or write files have a node to run on. Their sessions run only on a node with sandbox tier 1, or one a tenant admin opted out; it warns when the tenant has such agents and neither, naming them and both fixes. It reads only what servers before that rule answer too, so run the new version's oz0 doctor against the cluster before you upgrade it. On a dev server it is ok, and says that the dev server runs them on any node.
• The nodes' sandboxes line above also names every node that holds the opt-out, and warns while one does.

Run it on a server too, with --data-dir, to check that server's disk. Point it at another edge address with --edge-api and --edge-temporal.

On a node

oz0-node doctor

ok    identity            node node-79186dd87b6d of tenant default in cluster dev
ok    node certificate    valid until 2026-10-08T05:33:04Z, renewed while the node runs
ok    edge Temporal port  edge.example.com:7233 takes connections (2ms)
ok    edge API port       edge.example.com:7443 takes connections (2ms)
ok    edge certificate    edge.example.com:7443's certificate is from the cluster's CA, valid until 2026-11-06 (29 days)
ok    clock               within a second or two of the server's
ok    disk                /home/you/.local/share/orchestrator-zero-node: 366.2 GB free of 1006.9 GB (64% used)
ok    sandbox             tier 1 (bubblewrap 0.9.0)
skip  local models        no /home/you/.local/share/orchestrator-zero-node/models.yaml: this node serves no local model
9 checks: 8 ok, 0 warnings, 0 failed, 1 skipped

It reads the node's data directory (--data-dir), so it works whether or not the node runs: that the machine has joined, that the edge's ports answer from here, the clock, the disk (a node over 95% full takes no work), the sandbox tier it can run, and each local model server in models.yaml, on loopback only: its kind, version, calls at once, slots and formats when it is ready, hung for one that takes the connection and does not answer, and refused with the reason for one the node will not use.

For an mlx server it hashes the pinned model's files first, waiting up to 3 minutes, as the running node does once.

A sandbox at tier 0 warns with the fix the node works out (on Ubuntu 24.04, sudo oz0-node sandbox setup: see Sandbox tier 1 on Linux), and a bubblewrap with a known advisory shows as bubblewrap advisory, as the findings do.

As in oz0 doctor, only what can be fixed on the node warns: the advisory, a container and Windows are ok with the detail.

A check that fails

A port that does not answer, for example:

fail  edge API port  127.0.0.1:7443 does not take connections: connection refused
                     fix: check that the server runs (oz0 server start, or oz0 dev), that it serves the edge's API on port 7443 (--edge-api-port), and that no firewall between here and it drops TCP 7443 (sudo ufw allow 7443/tcp)
skip  edge certificate  the edge API port does not answer

Checks that need what failed are skipped, not failed again.

For scripts

--json prints every check with its status, detail and fix. The exit code is 0 when every check is ok or skipped, 2 when the worst is a warning, and 1 when a check failed:

oz0 doctor --json | jq -r '.checks[] | select(.status != "ok") | "\(.name): \(.fix)"'

In a support bundle

oz0 support bundle and oz0-node support bundle include doctor's report as doctor.json, beside the versions, the nodes, the recent findings and the logs, all redacted and made on this machine; nothing is sent anywhere. The bundle's doctor leaves out the model keys' checks, which call the providers. See Ask for help with a support bundle.

The long names orchestrator-zero and orchestrator-zero-node take the same commands.