Troubleshooting

For: cluster operators.

Find the symptom below after running doctor. A support bundle gathers diagnostic information when the checks do not explain the failure.

Start with oz0 doctor where you are, and oz0-node doctor on a node: they check ports, certificates, clocks, the database, Temporal, model keys and disk, and each check that fails says how to fix it (Doctor). Then --log-level debug on the server or the node, and --json on CLI commands for the full picture.

Nodes

join fails with a certificate or hash error

The node checks the server against the CA hash in the token or in --ca-hash before it sends anything. Check that:

• the hash matches oz0 cluster info;
• --server is an address on the edge's certificate: the server's --advertise address, or a name you gave it with --san;
• the token has not expired or been revoked (oz0 token list).

A node waits for approval forever

It joined without a token. Accept it: oz0 node list --pending, then oz0 node accept <node-id>.

A node stopped working and says it is blocked

An operator blocked it. Check with oz0 node show <node-id> and let it back in with node unblock; the node notices within ten seconds.

A node cannot reach the edge

Nodes need outbound access to the edge on 7233 and 7443, and nothing else. Check firewalls and proxies between them. A node with several edges in its list tries the others on its own. oz0-node doctor on the node says which port does not answer from there, and checks the edge's certificate and the clock.

Agents and jobs

A job stays RUNNING and nothing happens

Ask first: oz0 job why <job-id> says what the job waits for, in plain words, and the job's page in the console says the same (why is a job waiting?). The usual answer is that no node is ready to run the agent. oz0 catalog shows the nodes ready for each agent. Check that the plugin is installed for the node (plugin list, and --select if you used it), and that the node has every label in the agent's requires.labels. The job starts as soon as a node can take it.

A job waits after a node went away

Its tools run on its home node, the node that picked it up. If that node is gone, the job's next tool call waits for it to come back. Bring the node back, or cancel the job and start it again.

A job fails with a model error

When a model call fails, the job's error starts with what kind of failure it was, followed by the provider's own reason:

For a missing key:

printf %s "$ANTHROPIC_API_KEY" | oz0 secret set ANTHROPIC_API_KEY

Only ModelUnavailable is retried, by Temporal alone: five attempts, each one call to each of the agent's models, waiting as long as the provider's retry-after header asks. The others fail the job after one call.

The job failed with InvalidInput

The input does not match the agent's input_schema. The error says where. With an input schema, pass JSON: oz0 run reviewer '{"repo": "acme/api", "pr": 42}'.

The job ended at a limit

It reached max_steps, max_tokens or timeout. Raise the limit in the agent definition if the work really needs more, or make the instructions more focused.

The job failed with OutcomeNotMet

The agent's outcome check failed every round, or the budget ran out between rounds. The job's trace and its findings have each verdict with the judge's reasons; fix the instructions, the rubric or the number of rounds.

The job failed with Killed

A kill switch stopped the agent, one of its tools or the tenant. oz0 kill list shows the switches that hold, and who set them.

Plugins

plugin list shows an error on a node

The node could not build or start the plugin, and kept the previous version running. The error says why. Common causes: a dependency that does not build on that platform (check requires.platforms), a uv.lock that does not match pyproject.toml (run uv lock and commit it), or an MCP command that does not start (try it by hand with uv run).

plugin install says the repository cannot be reached

The edge fetches over HTTPS; a private repository needs a token in a tenant secret, named with --secret (a private repository). SSH URLs are not supported. Check the URL, the token's access, and that the edge can reach the Git host.

oz0 dev cannot find the node or the runtime bundle

It needs orchestrator-zero-node next to the server binary, on PATH or in the dist/ folder next to an installed server's binary (or --node-binary), and a runtime bundle: --runtime-bundle, $OZ0_RUNTIME_BUNDLE, the newest in ./dist/runtime, or, for an installed server, the runtime in its dist/ folder, which the dev node downloads from the dev edge. Build one with make runtime-bundle.

A tool call waits for approval, or is refused, and the plugin did not ask for that

The tenant's tool policy decided by the function's MCP annotations. Since v1.0 a function that declares none, or is not read-only and says nothing about destructiveHint, waits for a person. The refusal or the approval says which rule decided and how to fix it, such as To fix: add readOnlyHint: true to jev.decide if it only reads; oz0 dev lists the functions of your plugin that do not run freely when it installs it. Add the annotations to the tool, or let the function run with a rule: oz0 policy allow jev.decide --reason "...".

A first oz0 dev answers but later runs fail with MissingKey

The first run answers from recorded answers and needs no key; every later model call needs one. Store it as oz0 dev says, printf %s "$ANTHROPIC_API_KEY" | oz0 secret set ANTHROPIC_API_KEY, or set ANTHROPIC_API_KEY in the environment before oz0 dev, which stores it for the dev tenant.

Servers

The server refuses a port

Temporal's ports must be 32767 or lower, because its PostgreSQL schema stores them in a small integer column. The defaults are fine.

The server refuses to start on the database

The database's schema is newer than this binary knows: another server was upgraded first. Upgrade this one too.

Temporal reports too many connections

Give PostgreSQL a max_connections of at least 100 per server.

Ask for help with a support bundle

When you ask someone for help, a support bundle gives them what they need in one file, without a call to your cluster. It is made on your machine, sent nowhere, and you read it before you send it by hand.

oz0 support bundle                                    # oz0-support-<time>.tar.gz here
oz0 support bundle --job <job-id> --since 6h          # with a job, as job export writes it
journalctl -u orchestrator-zero --since -1d | oz0 support bundle --log -
systemctl cat orchestrator-zero | oz0 support bundle --config -   # with the server's flags and environment
oz0-node support bundle                               # on a node: oz0-node-support-<time>.tar.gz

What is in it (oz0 support bundle, with what the CLI context may read):

• the versions of the CLI and of every server;
• oz0 doctor, without the model keys' checks (which call the providers): ports, certificates, the clock, and for a cluster admin management's checks, with the database migrations' state;
• the CLI context's addresses and file names (never the key);
• the cluster, the nodes with their versions, health and findings;
• the jobs with findings and the error groups of the last --since (a day by default, at most 30);
• the logs you give with --log (a file, or - for standard input; the server has no log file of its own: systemd keeps it in the journal), their last 4 MiB from --since on, each line redacted whole before a line longer than 16 KiB is cut;
• the server's configuration you give with --config (an env file, a compose file, or - for standard input such as systemctl cat orchestrator-zero): its flags and environment, which no API returns;
• with --data-dir, a server's data folder's listing: names and sizes, never what the files hold;
• the jobs you name with --job, as job export writes them: the trace, findings and tags. What the steps read and wrote is in it only with --content and --job, which needs the content permission and is in the audit log; the manifest's content says whether any export returned it.

oz0-node support bundle holds the node's version and runtimes, oz0-node doctor, the identity's summary (its key is never opened), its health, status and models.yaml, the data directory's listing, and the end of logs/node.log and of each --log. With --job, the listing of that job's session folders on the node; what the job read and wrote (the text files of its work folder) only with --content as well.

A job's process may still be writing in its folder while the bundle is made, so the bundle reads each folder through a root that nothing inside it can lead out of: a link is not followed out of it, even one that takes a folder's place while the bundle reads it, so the bundle never holds a file from outside the job's folder.

It visits at most 10,000 entries and keeps at most 200 files and 32 MiB of a job.

What the bundle redacts

Every file goes through a redactor before it is written. The following values become [redacted:<kind>]:

• PEM blocks, including keys and certificates.
• The user and password in URLs, and URL query parameters named like a secret, such as token, key, sig and password.
• Authorization headers and bearer tokens.
• Tokens with known shapes: oz0_ API keys, K10 join tokens, provider and forge keys, Telegram bot tokens, JWTs and sealed values.
• The secret part of Slack, Discord, Telegram and Teams webhook URLs.
• Values of keys named like secrets, including password=, "api_key": ... and names ending in _KEY, such as OZ0_MODEL_KEY. A quoted value is redacted whole; a YAML value is redacted to the end of its line.
• An environment file's value to the end of its line, such as OZ0_POSTGRES_PASSWORD=two words or export KEY=..., and systemd's Environment="KEY=a b" up to its closing quote.
• Values after flags named like secrets, such as --token VALUE and --password=VALUE.
• Long dense base64, such as sealed blobs, and email addresses.

A job export also replaces the tenant's own secrets of 6 characters or more, by name.

On a node, the bundle also replaces every value of the tenant's secrets that the node holds fingerprints of: the set the edge sends the node for its egress proxy and its scan of tool calls, salted digests of each value raw, in base64 and base64url, in hex and percent-encoded.

Each match becomes [redacted:fingerprint], in logs, state files, a job's files and the manifest's own errors alike. The node never holds the values to compare, so neither does the bundle.

Values shorter than 16 bytes or built the way people build passwords are not in a node's set (upgrades), and only shapes and names find those. A node whose fingerprint file exists but cannot be read writes no bundle and says why; run it as the node's user.

What it cannot know. The redactor recognises shapes and names, not meaning: a secret in a free-text field, a secret written as bare hex (it looks like a digest), a customer's name or an internal host name stays as it is. So:

1. Unpack it and read it: tar -xzf oz0-support-<time>.tar.gz. README.txt says what each file is, and manifest.json lists every file with its source and checksum, the parts left out and why (a call your role may not make, a log that could not be read), and how many of each kind were redacted.
2. Remove anything you would not send, and pack it again if you changed it.
3. Send it by hand, to the people you choose. Nothing in Orchestrator Zero uploads it.

A bundle never replaces an existing file, and is written readable only by you (mode 0600).

In short: a bundle is made on the machine and sent nowhere; it never holds the node's or the CLI's key; it holds what jobs read and wrote only with --content; and it replaces every secret its redactor recognises by shape or name and, on a node, every value of the tenant's secrets the node holds, matched by fingerprint. It cannot recognise everything, so read it before you send it.