---
title: "Troubleshooting"
description: "Common problems, what causes them and how to fix them."
canonical_url: "https://docs.oz0.ai/fleet/troubleshooting"
---
# Troubleshooting

> Common problems, what causes them and how to fix them.

For: cluster operators.

Find the symptom below after running doctor. A [support bundle](https://docs.oz0.ai/fleet/troubleshooting#ask-for-help-with-a-support-bundle) gathers diagnostic information when the checks do not explain the failure.

Start with `oz0 doctor` where you are, and `oz0-node doctor` on a node: they check ports, certificates, clocks, the database, Temporal, model keys and disk, and each check that fails says how to fix it ([Doctor](https://docs.oz0.ai/fleet/doctor)). Then `--log-level debug` on the server or the node, and `--json` on CLI commands for the full picture.

## Nodes

### `join` fails with a certificate or hash error

The node checks the server against the CA hash in the token or in `--ca-hash` before it sends anything. Check that:

- the hash matches `oz0 cluster info`;
- `--server` is an address on the edge's certificate: the server's `--advertise` address, or a name you gave it with `--san`;
- the token has not expired or been revoked (`oz0 token list`).

### A node waits for approval forever

It joined without a token. Accept it: `oz0 node list --pending`, then `oz0 node accept <node-id>`.

### A node stopped working and says it is blocked

An operator blocked it. Check with `oz0 node show <node-id>` and let it back in with `node unblock`; the node notices within ten seconds.

### A node cannot reach the edge

Nodes need outbound access to the edge on 7233 and 7443, and nothing else. Check firewalls and proxies between them. A node with several edges in its list tries the others on its own. `oz0-node doctor` on the node says which port does not answer from there, and checks the edge's certificate and the clock.

## Agents and jobs

### A job stays `RUNNING` and nothing happens

Ask first: `oz0 job why <job-id>` says what the job waits for, in plain words, and the job's page in the console says the same ([why is a job waiting?](https://docs.oz0.ai/fleet/queues#why-is-a-job-waiting)). The usual answer is that no node is ready to run the agent. `oz0 catalog` shows the nodes ready for each agent. Check that the plugin is installed for the node (`plugin list`, and `--select` if you used it), and that the node has every label in the agent's `requires.labels`. The job starts as soon as a node can take it.

### A job waits after a node went away

Its tools run on its home node, the node that picked it up. If that node is gone, the job's next tool call waits for it to come back. Bring the node back, or cancel the job and start it again.

### A job fails with a model error

When a model call fails, the job's error starts with what kind of failure it was, followed by the provider's own reason:

<table>
<thead>
  <tr>
    <th>
      Error
    </th>
    
    <th>
      What happened
    </th>
    
    <th>
      What to do
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        MissingKey
      </code>
    </td>
    
    <td>
      The tenant has no key for any of the agent's models, so the gateway refused every call with <code>
        412
      </code>
    </td>
    
    <td>
      Set the key, as the error says, and run the job again
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ModelUnavailable
      </code>
    </td>
    
    <td>
      The provider was overloaded, rate limited or down, or the connection broke, on all five attempts
    </td>
    
    <td>
      Run the job again later, or give the agent a fallback model on another provider
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ModelRejected
      </code>
    </td>
    
    <td>
      The provider refused the request itself: a key it does not accept (<code>
        401
      </code>
      
      ), no access (<code>
        403
      </code>
      
      ), a model name it does not know (<code>
        404
      </code>
      
      ), or a request that is invalid (<code>
        400
      </code>
      
      ) or too large (<code>
        413
      </code>
      
      ). Anthropic also answers <code>
        400
      </code>
      
       when usage reaches a spend limit you set ("You have reached your specified API usage limits")
    </td>
    
    <td>
      Fix what the provider's reason names, or raise your own spend limit; another attempt would fail the same way
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        SpendLimitReached
      </code>
    </td>
    
    <td>
      The provider account's spend cap or quota is used up: a <code>
        429
      </code>
      
       with Anthropic's <code>
        enforced_spend_limit_reached
      </code>
      
      , or OpenAI's <code>
        insufficient_quota
      </code>
      
      , <code>
        credit_balance_exhausted
      </code>
      
      , <code>
        organization_spend_limit_exceeded
      </code>
      
      , <code>
        project_spend_limit_exceeded
      </code>
      
       or <code>
        organization_usage_limit_exceeded
      </code>
    </td>
    
    <td>
      Add credits or raise the limit with the provider, or wait until it resets
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        BudgetExceeded
      </code>
    </td>
    
    <td>
      The job tree spent its budget
    </td>
    
    <td>
      See <a href="https://docs.oz0.ai/build/parts/contracts-and-limits">
        Contracts and limits
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        QuotaExceeded
      </code>
    </td>
    
    <td>
      A quota of the person or team the job is charged to was spent, and the budget request was denied or not decided in time
    </td>
    
    <td>
      See <a href="https://docs.oz0.ai/spend/budgets-and-quotas/charging#when-a-quota-is-reached">
        When a quota is reached
      </a>
    </td>
  </tr>
</tbody>
</table>

For a missing key:

```bash [Terminal]
printf %s "$ANTHROPIC_API_KEY" | oz0 secret set ANTHROPIC_API_KEY
```

Only `ModelUnavailable` is retried, by Temporal alone: five attempts, each one call to each of the agent's models, waiting as long as the provider's `retry-after` header asks. The others fail the job after one call.

### The job failed with `InvalidInput`

The input does not match the agent's `input_schema`. The error says where. With an input schema, pass JSON: `oz0 run reviewer '{"repo": "acme/api", "pr": 42}'`.

### The job ended at a limit

It reached `max_steps`, `max_tokens` or `timeout`. Raise the limit in the agent definition if the work really needs more, or make the instructions more focused.

### The job failed with `OutcomeNotMet`

The agent's [outcome check](https://docs.oz0.ai/improve/outcome-checks) failed every round, or the budget ran out between rounds. The job's trace and its findings have each verdict with the judge's reasons; fix the instructions, the rubric or the number of rounds.

### The job failed with `Killed`

A [kill switch](https://docs.oz0.ai/govern/kill-switches) stopped the agent, one of its tools or the tenant. `oz0 kill list` shows the switches that hold, and who set them.

## Plugins

### `plugin list` shows an error on a node

The node could not build or start the plugin, and kept the previous version running. The error says why. Common causes: a dependency that does not build on that platform (check `requires.platforms`), a `uv.lock` that does not match `pyproject.toml` (run `uv lock` and commit it), or an MCP command that does not start (try it by hand with `uv run`).

### `plugin install` says the repository cannot be reached

The edge fetches over HTTPS; a private repository needs a token in a tenant secret, named with `--secret` ([a private repository](https://docs.oz0.ai/build/plugins/install/sources#a-private-repository)). SSH URLs are not supported. Check the URL, the token's access, and that the edge can reach the Git host.

### `oz0 dev` cannot find the node or the runtime bundle

It needs `orchestrator-zero-node` next to the server binary, on `PATH` or in the `dist/` folder next to an installed server's binary (or `--node-binary`), and a runtime bundle: `--runtime-bundle`, `$OZ0_RUNTIME_BUNDLE`, the newest in `./dist/runtime`, or, for an installed server, the runtime in its `dist/` folder, which the dev node downloads from the dev edge. Build one with `make runtime-bundle`.

### A tool call waits for approval, or is refused, and the plugin did not ask for that

The tenant's [tool policy](https://docs.oz0.ai/govern/tool-policy) decided by the function's MCP annotations. Since v1.0 a function that declares none, or is not read-only and says nothing about `destructiveHint`, waits for a person. The refusal or the approval says which rule decided and how to fix it, such as `To fix: add readOnlyHint: true to jev.decide if it only reads`; `oz0 dev` lists the functions of your plugin that do not run freely when it installs it. Add the annotations to the tool, or let the function run with a rule: `oz0 policy allow jev.decide --reason "..."`.

### A first `oz0 dev` answers but later runs fail with `MissingKey`

The first run answers from recorded answers and needs no key; every later model call needs one. Store it as `oz0 dev` says, `printf %s "$ANTHROPIC_API_KEY" | oz0 secret set ANTHROPIC_API_KEY`, or set `ANTHROPIC_API_KEY` in the environment before `oz0 dev`, which stores it for the dev tenant.

## Servers

### The server refuses a port

Temporal's ports must be 32767 or lower, because its PostgreSQL schema stores them in a small integer column. The defaults are fine.

### The server refuses to start on the database

The database's schema is newer than this binary knows: another server was upgraded first. Upgrade this one too.

### Temporal reports too many connections

Give PostgreSQL a `max_connections` of at least 100 per server.

## Ask for help with a support bundle

When you ask someone for help, a support bundle gives them what they need in one file, without a call to your cluster. It is made on your machine, sent nowhere, and you read it before you send it by hand.

```bash [Terminal]
oz0 support bundle                                    # oz0-support-<time>.tar.gz here
oz0 support bundle --job <job-id> --since 6h          # with a job, as job export writes it
journalctl -u orchestrator-zero --since -1d | oz0 support bundle --log -
systemctl cat orchestrator-zero | oz0 support bundle --config -   # with the server's flags and environment
oz0-node support bundle                               # on a node: oz0-node-support-<time>.tar.gz
```

What is in it (`oz0 support bundle`, with what the CLI context may read):

- the versions of the CLI and of every server;
- `oz0 doctor`, without the model keys' checks (which call the providers): ports, certificates, the clock, and for a cluster admin management's checks, with the database migrations' state;
- the CLI context's addresses and file names (never the key);
- the cluster, the nodes with their versions, health and findings;
- the jobs with findings and the error groups of the last `--since` (a day by default, at most 30);
- the logs you give with `--log` (a file, or `-` for standard input; the server has no log file of its own: systemd keeps it in the journal), their last 4 MiB from `--since` on, each line redacted whole before a line longer than 16 KiB is cut;
- the server's configuration you give with `--config` (an env file, a compose file, or `-` for standard input such as `systemctl cat orchestrator-zero`): its flags and environment, which no API returns;
- with `--data-dir`, a server's data folder's listing: names and sizes, never what the files hold;
- the jobs you name with `--job`, as [`job export`](https://docs.oz0.ai/improve/errors-and-insights#export-a-job) writes them: the trace, findings and tags. What the steps read and wrote is in it only with `--content` and `--job`, which needs the [content permission](https://docs.oz0.ai/admin/tenants) and is in the audit log; the manifest's `content` says whether any export returned it.

`oz0-node support bundle` holds the node's version and runtimes, `oz0-node doctor`, the identity's summary (its key is never opened), its health, status and `models.yaml`, the data directory's listing, and the end of `logs/node.log` and of each `--log`. With `--job`, the listing of that job's session folders on the node; what the job read and wrote (the text files of its work folder) only with `--content` as well.

A job's process may still be writing in its folder while the bundle is made, so the bundle reads each folder through a root that nothing inside it can lead out of: a link is not followed out of it, even one that takes a folder's place while the bundle reads it, so the bundle never holds a file from outside the job's folder.

It visits at most 10,000 entries and keeps at most 200 files and 32 MiB of a job.

### What the bundle redacts

Every file goes through a redactor before it is written. The following values become `[redacted:<kind>]`:

- PEM blocks, including keys and certificates.
- The user and password in URLs, and URL query parameters named like a secret, such as `token`, `key`, `sig` and `password`.
- `Authorization` headers and bearer tokens.
- Tokens with known shapes: `oz0_` API keys, `K10` join tokens, provider and forge keys, Telegram bot tokens, JWTs and sealed values.
- The secret part of Slack, Discord, Telegram and Teams webhook URLs.
- Values of keys named like secrets, including `password=`, `"api_key": ...` and names ending in `_KEY`, such as `OZ0_MODEL_KEY`. A quoted value is redacted whole; a YAML value is redacted to the end of its line.
- An environment file's value to the end of its line, such as `OZ0_POSTGRES_PASSWORD=two words` or `export KEY=...`, and systemd's `Environment="KEY=a b"` up to its closing quote.
- Values after flags named like secrets, such as `--token VALUE` and `--password=VALUE`.
- Long dense base64, such as sealed blobs, and email addresses.

A job export also replaces the tenant's own secrets of 6 characters or more, by name.

On a node, the bundle also replaces every value of the tenant's secrets that the node holds fingerprints of: the set the edge sends the node for its egress proxy and its scan of tool calls, salted digests of each value raw, in base64 and base64url, in hex and percent-encoded.

Each match becomes `[redacted:fingerprint]`, in logs, state files, a job's files and the manifest's own errors alike. The node never holds the values to compare, so neither does the bundle.

Values shorter than 16 bytes or built the way people build passwords are not in a node's set ([upgrades](https://docs.oz0.ai/fleet/upgrades/data-migrations#nodes-look-only-for-secrets-a-guess-would-not-find)), and only shapes and names find those. A node whose fingerprint file exists but cannot be read writes no bundle and says why; run it as the node's user.

What it cannot know. The redactor recognises shapes and names, not meaning: a secret in a free-text field, a secret written as bare hex (it looks like a digest), a customer's name or an internal host name stays as it is. So:

1. Unpack it and read it: `tar -xzf oz0-support-<time>.tar.gz`. `README.txt` says what each file is, and `manifest.json` lists every file with its source and checksum, the parts left out and why (a call your role may not make, a log that could not be read), and how many of each kind were redacted.
2. Remove anything you would not send, and pack it again if you changed it.
3. Send it by hand, to the people you choose. Nothing in Orchestrator Zero uploads it.

A bundle never replaces an existing file, and is written readable only by you (mode 0600).

In short: a bundle is made on the machine and sent nowhere; it never holds the node's or the CLI's key; it holds what jobs read and wrote only with `--content`; and it replaces every secret its redactor recognises by shape or name and, on a node, every value of the tenant's secrets the node holds, matched by fingerprint. It cannot recognise everything, so read it before you send it.


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.