--- title: "Doctor" description: "Check a running setup from where you are: an operator's machine, a server or a node. Each check that fails says how to fix it." canonical_url: "https://docs.oz0.ai/fleet/doctor" --- # Doctor > Check a running setup from where you are: an operator's machine, a server or a node. Each check that fails says how to fix it. For: cluster operators. Run the check from the machine whose access you need to test. Follow the reported fix, then run it again to confirm the problem is resolved. `oz0 doctor` checks the setup your CLI context points at, from the machine you run it on. `oz0-node doctor` checks a node from the machine the node runs on. Each check says what it found, and each one that is not ok says how to fix it. ```bash [Terminal] oz0 doctor ``` ```text ok CLI context cluster dev, namespace default ok operator certificate valid until 2027-01-05 (89 days) ok edge Temporal port 127.0.0.1:7233 takes connections (1ms) ok edge API port 127.0.0.1:7443 takes connections (0s) ok edge certificate 127.0.0.1:7443's certificate is from the cluster's CA, valid until 2026-11-06 (29 days) ok clock within a second or two of the server's ok management https://127.0.0.1:8443 answers ok database SQLite answered a ping in 0s ok migrations schema version 74, as this server expects ok servers 1 up: dev ok cluster CA valid until 2036-10-04 ok nodes 1 of 1 online ok node certificates every node's is renewed in time ok node sandboxes 1 of 1 nodes reported, every one at tier 1 with no known advisory ok temporal namespace default answered in 1ms warn model keys tenant default has no model key, so its agents' calls to cloud models are refused (MissingKey) fix: store one: printf %s "$ANTHROPIC_API_KEY" | oz0 secret set ANTHROPIC_API_KEY --tenant default; or serve a local model on a node (models.yaml, see Local models) ok disk /home/you/.config/orchestrator-zero: 366.4 GB free of 1006.9 GB (64% used) 17 checks: 16 ok, 1 warnings, 0 failed, 0 skipped ``` ## What it checks From this machine: - The CLI context and the operator certificate: that there is a context, and that its certificate is valid and not about to end. - The edge's ports: that the Temporal port (7233) and the API port (7443) take connections from here. Nodes need both; operators need the API port. - The edge's certificate: that the API port answers TLS with a certificate of the cluster's CA, for the address you use, that is not about to end. - The clock: this machine's clock against the edge's. Certificates and Temporal's timeouts depend on it; more than 30 seconds apart fails. - Disk: the free space where the context lives, or a server's `--data-dir`. Management's own checks, through the admin API. Only cluster admins may run them; for anyone else doctor skips them, which does not count as a failure: - The database and its migrations: that PostgreSQL (or SQLite in dev mode) answers, how fast, and that its schema is the one this server expects. - The servers: which ones report in, and which have not for a minute. - The cluster CA, and the nodes: how many are online, which are not healthy, and whether each node's certificate was renewed in time. A running node renews its own at two thirds of its life. - The keys and backups a recovery needs (a cluster; a dev server skips them): `master key backup` warns until someone ran `oz0 server master-key confirm-backup` (without the master key everything sealed under it is lost, encrypted histories included); `secrets key backup`, with the [roles apart](https://docs.oz0.ai/fleet/separate-roles), until `oz0 server secrets-key confirm-backup` recorded the key the cluster seals to; `database backup` while the newest recorded backup is older than the newest tenant payload key, whose histories a database loss would leave unreadable.
`payload encryption` warns when an edge runs with `--encrypt-payloads=false`, and `temporal retention` when a namespace does not keep what its tenants' retention wants, with the command that fixes it (on Temporal Cloud, `tcld namespace retention set`). `keys sealed by the edges`, with the [roles apart](https://docs.oz0.ai/fleet/separate-roles), warns until an edge has recorded that every data subject's key and payload key is sealed by an edge; until then the edges that were there before still re-seal older keys, for a day from their start at most, and their logs count the keys they pass over because they open with no key. `master key source` and `secrets key source` say which running servers read each key from a plain file and which through a [key command](https://docs.oz0.ai/admin/key-commands), and warn when a server's last unwrap failed or its command cannot reach the wrapping key, or while the plain copy that wrapping keeps (`.plain`) is still beside a server's wrapped key file. `erasure ledger` reads the whole [erasure ledger](https://docs.oz0.ai/govern/personal-data/retention#backups) as a restore would and fails when a line does not verify, when the file cannot be read, or when it no longer holds the newest erasure the database says it recorded; erasures are refused until it is fixed. A tenant's members see the same lines for their tenant, as yes or no without key IDs, holders or server names, under Settings, Encryption or with `oz0 tenant encryption ` ([encryption status](https://docs.oz0.ai/fleet/temporal-backends/encryption#encryption-status-and-key-rotation)). - Shared pools (while the cluster shares pools): `shared pools` warns about each accepted node that cannot poll them, because its supervisor predates them or because every runtime it starts or runs does, with the fix (`oz0 node upgrade --binary` or `--runtime`, or `--queue-pools=plugin` until then).
`fair dispatch` names the backends that are an external Temporal cluster or Temporal Cloud, where a shared pool's plugins take turns only if fair dispatch is on there (`matching.enableFairness`, or the namespace's fairness): management cannot read that setting, so the line is a skipped check that says what to look at, and leaves the exit code alone ([Upgrades](https://docs.oz0.ai/fleet/upgrades/shared-pools#shared-pools-by-default)). - Harness sessions: how many harness sessions nodes lost in the last day and how many finished within a drain; it warns for each node that lost three or more, with what became of them (restarted, waiting for a person, failed) and what they spent, and suggests draining the node before planned work and looking at why it stops. - Secrets past their expiry (`secret_overdue`): each tenant's secrets whose [expiry](https://docs.oz0.ai/admin/secrets/rotation#expiry-and-reminders) has passed, with the fix: rotate them, or move the expiry. It warns; the secrets keep working. - The nodes' sandboxes: which accepted nodes run [sandbox tier 0](https://docs.oz0.ai/concepts/security/sandbox-tiers#sandboxes), and which run a bubblewrap with a known advisory, from the nodes' [findings](https://docs.oz0.ai/fleet/nodes#findings).
It warns only about what you can fix on a node: a Linux or macOS node at tier 0 with a fix. A node run with `--sandbox off`, a Windows node, a node in a container (the quickstart's, or a Helm chart's pod) and a bubblewrap advisory no distribution has fixed yet are named in the detail of an `ok` check, so doctor stays green, and exits 0, on a setup that is as good as it can be. When the nodes to fix share a fix, the check shows it; otherwise `oz0 node show ` has each one's. The edge's checks for the tenant (`--tenant`), through the runtime API. Only the tenant's admins may run them; anyone else sees a skip: - Temporal: that the tenant's namespace answers, on whichever backend it is (embedded, an external cluster or Temporal Cloud). - Secrets past their expiry (`secret_overdue`): the tenant's own, as for cluster admins above. - Model keys: each key the tenant has, checked with the provider's list of models; for a [pooled](https://docs.oz0.ai/spend/models-and-keys/deployment-pools#pools-of-deployments) `anthropic` or `openai` record, each member's key, at the member's endpoint. That spends no tokens. `--skip-model-keys` leaves it out. A record that signs in rather than sending its key in a header (OAuth2 or SigV4, see [Models and keys](https://docs.oz0.ai/spend/models-and-keys)) is not checked: its secret goes only to its token URL, or into a signature. The tenant's tool policy (`--tenant`), through the admin API. Anyone who may read the tool policy may run it (the tenant's readers and admins); anyone else sees a skip: - The tool policy (`tool policy`): the [policy lint](https://docs.oz0.ai/govern/tool-policy/review-changes#the-policy-lint), one warning per finding, named `tool policy rule `, each with the command that fixes it: a rule an earlier one hides, a pattern that matches no function, a condition on an argument the function does not take, a condition on `tool.class` without its host form, a shadow rule never turned on, and a rule unused for 90 days.
With no finding it is one `ok` line that says how many rules it checked against how many functions. A warning here makes doctor exit 2, as any warning does; the policy works as it is. The tenant's sandbox floor (`--tenant`), through the admin API, for anyone who may read the tenant's catalog and nodes: - The sandbox floor (`sandbox floor`): whether the tenant's agents that can run a shell or write files have a node to run on. Their sessions run only on a node with sandbox tier 1, or one a tenant admin [opted out](https://docs.oz0.ai/fleet/nodes#the-sandbox-opt-out); it warns when the tenant has such agents and neither, naming them and both fixes. It reads only what servers before that rule answer too, so run the new version's `oz0 doctor` against the cluster before you [upgrade it](https://docs.oz0.ai/fleet/upgrades/sandbox-changes#shell-and-write-sessions-run-on-sandbox-tier-1-whoever-started-the-job). On a dev server it is `ok`, and says that the dev server runs them on any node. - The nodes' sandboxes line above also names every node that holds the opt-out, and warns while one does. Run it on a server too, with `--data-dir`, to check that server's disk. Point it at another edge address with `--edge-api` and `--edge-temporal`. ## On a node ```bash [Terminal] oz0-node doctor ``` ```text ok identity node node-79186dd87b6d of tenant default in cluster dev ok node certificate valid until 2026-10-08T05:33:04Z, renewed while the node runs ok edge Temporal port edge.example.com:7233 takes connections (2ms) ok edge API port edge.example.com:7443 takes connections (2ms) ok edge certificate edge.example.com:7443's certificate is from the cluster's CA, valid until 2026-11-06 (29 days) ok clock within a second or two of the server's ok disk /home/you/.local/share/orchestrator-zero-node: 366.2 GB free of 1006.9 GB (64% used) ok sandbox tier 1 (bubblewrap 0.9.0) skip local models no /home/you/.local/share/orchestrator-zero-node/models.yaml: this node serves no local model 9 checks: 8 ok, 0 warnings, 0 failed, 1 skipped ``` It reads the node's data directory (`--data-dir`), so it works whether or not the node runs: that the machine has joined, that the edge's ports answer from here, the clock, the disk (a node over 95% full takes no work), the [sandbox](https://docs.oz0.ai/concepts/security/sandbox-tiers#sandboxes) tier it can run, and each [local model](https://docs.oz0.ai/fleet/local-models) server in `models.yaml`, on loopback only: its kind, version, calls at once, slots and formats when it is ready, `hung` for one that takes the connection and does not answer, and `refused` with the reason for one the node will not use. For an mlx server it hashes the pinned model's files first, waiting up to 3 minutes, as the running node does once. A sandbox at tier 0 warns with the fix the node works out (on Ubuntu 24.04, `sudo oz0-node sandbox setup`: see [Sandbox tier 1 on Linux](https://docs.oz0.ai/getting-started/install/linux-sandbox#sandbox-tier-1-on-linux)), and a bubblewrap with a known advisory shows as `bubblewrap advisory`, as the [findings](https://docs.oz0.ai/fleet/nodes#findings) do. As in `oz0 doctor`, only what can be fixed on the node warns: the advisory, a container and Windows are `ok` with the detail. ## A check that fails A port that does not answer, for example: ```text fail edge API port 127.0.0.1:7443 does not take connections: connection refused fix: check that the server runs (oz0 server start, or oz0 dev), that it serves the edge's API on port 7443 (--edge-api-port), and that no firewall between here and it drops TCP 7443 (sudo ufw allow 7443/tcp) skip edge certificate the edge API port does not answer ``` Checks that need what failed are skipped, not failed again. ## For scripts `--json` prints every check with its status, detail and fix. The exit code is 0 when every check is ok or skipped, 2 when the worst is a warning, and 1 when a check failed: ```bash [Terminal] oz0 doctor --json | jq -r '.checks[] | select(.status != "ok") | "\(.name): \(.fix)"' ``` ## In a support bundle `oz0 support bundle` and `oz0-node support bundle` include doctor's report as `doctor.json`, beside the versions, the nodes, the recent findings and the logs, all redacted and made on this machine; nothing is sent anywhere. The bundle's doctor leaves out the model keys' checks, which call the providers. See [Ask for help with a support bundle](https://docs.oz0.ai/fleet/troubleshooting#ask-for-help-with-a-support-bundle). The long names `orchestrator-zero` and `orchestrator-zero-node` take the same commands. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.