--- title: "Tracing and metrics" description: "Metrics for Prometheus, and an OpenTelemetry trace for every job." canonical_url: "https://docs.oz0.ai/fleet/observability" --- # Tracing and metrics > Metrics for Prometheus, and an OpenTelemetry trace for every job. For: cluster operators. Enable a metrics listener for Prometheus or an OTLP destination for traces. Use the measurement definitions below when building queries and alerts. ## Metrics Each server serves `/metrics` for Prometheus when you give it an address: ```bash [Terminal] oz0 server start --metrics-listen 127.0.0.1:9464 ... curl -s http://127.0.0.1:9464/metrics | grep '^oz0_' ``` The endpoint is plain HTTP and not authenticated, so keep it on a private interface or behind your scraper's network. Each server counts what it saw itself; let Prometheus add them up across a cluster.
Metric Labels What it counts
oz0_llm_calls_total tenant , provider , model , status Model calls through the gateway. status is 2xx , 4xx , 5xx , or the exact code for 429, 500, 502, 503 and 529
oz0_llm_tokens_total tenant , provider , model , kind Tokens: input , output , cache_read , cache_write
oz0_llm_cost_usd_total tenant , provider , model What the calls cost, in dollars
oz0_llm_pool_events_total tenant , provider , member , event Details
oz0_gateway_records_total outcome The gateway's signed records of model calls: written , late (committed after the call's response ended), failed (could not be signed, such as while no gateway key is in force) and lost (could not be written or queued in time). A lost record shows later as a node_only call in the two-way check
oz0_metering_writes_retried_total outcome Usage rows and quota charges whose write failed and was kept to be tried again: spooled (kept on the server's disk, in <data-dir>/metering ), spool_written (written from there, after a restart too), spool_dropped (given up there: older than a month, or unreadable), and, where no disk takes them, queued (kept in memory), retried (written from memory) and lost (the memory queue was full, or the edge stopped first)
oz0_metering_writes_waiting Usage rows and quota charges waiting to be tried again, on disk and in memory
oz0_reservation_settle_failures_total Budgeted calls whose settlement failed, left to the retirement pass, which charges each at least its reservation
oz0_llm_first_token_seconds tenant , provider , model , node A histogram of the time to first token of streamed calls, from the upstream send. Buckets from 50 ms to 64 s
oz0_llm_decode_tokens_per_second tenant , provider , model , node A histogram of each call's decode speed in output tokens per second: the model server's own figure where it reports one, else measured from the stream. Buckets from 1 to 640
oz0_llm_queue_seconds tenant , provider , model , node A histogram of the time a call waited before it was sent: the edge's checks and reservation, a pool's moves, a node's admission and its wait for a free slot on its model server. Buckets from 1 ms to 120 s
oz0_reported_cost_usd_total tenant , plugin , service What plugins reported for their own services ( Tools )
oz0_jobs_started_total tenant , agent , by Jobs started: by is api (the CLI, the console, apps, MCP and A2A clients), webhook , routine , event or gate
oz0_nodes_connected tenant Nodes with a control stream open to this server
oz0_node_health_score tenant , node Each connected node's health score
oz0_node_health_state tenant , node , state 1 for the node's state: healthy , degraded or unhealthy
oz0_gate_verdicts_total tenant , plugin , verdict Quality gates : passed or stopped
oz0_rollouts_total tenant , kind , verdict Rollouts that ended, of a runtime or a plugin ( kind ): done or rolled-back
oz0_harness_sessions_total tenant , node , agent , outcome Harness sessions a node's stop or loss touched: drained (finished within the node's drain or stop), restarted , waiting (for a person's restart approval) or failed . A session that waited and then restarted counts in both. Each counts once across the cluster
oz0_harness_session_seconds_total tenant , node , agent , outcome How long those sessions ran before they finished or were lost
oz0_harness_session_cost_usd_total tenant , node , agent , outcome What they spent, in dollars; a lost session's spend stays spent
oz0_edge_state state 1 for this server's edge's state : fresh , behind , fenced or fail-static
oz0_edge_state_changes_total from , to Changes of the edge's state
oz0_edge_registry_version The registry version this edge applied last
oz0_edge_registry_age_seconds How long ago this edge last read the registry
oz0_registry_reloads_refused_total reason Registry reloads the edge refused as suspicious: empty or mass-block
oz0_artifact_gc_passes_total dry_run , failed Passes of the plugin artifact collector
oz0_artifact_gc_collected_total , oz0_artifact_gc_collected_bytes_total dry_run Unreferenced artifacts it deleted, or would have, and their size
oz0_artifact_gc_kept , oz0_artifact_gc_young Artifacts the last pass kept: referenced, or within the grace period
The three timing histograms name a `node` only for calls to a node's own model servers (provider `local`: its local calls and the fleet's calls it served), so you can size one machine against another; a cloud model's calls have `node=""`, since the node that asked says nothing about the cloud model's speed. They count models and machines, never people: no metric has a person's or a team's label. A call that did not measure a figure is not counted in its histogram: an unstreamed call has no time to first token, and a call to a server that reports no timings of its own and was not streamed has no decode speed. Go's and the process's own metrics (`go_*`, `process_*`) are there too. A useful alert to start from: ```yaml [alerts.yaml] - alert: NodeUnhealthy expr: oz0_node_health_state{state="unhealthy"} == 1 for: 2m - alert: EdgeFenced # this server lost the database while others did not expr: oz0_edge_state{state="fenced"} == 1 - alert: EdgeFailStatic # no edge reaches the database: blocks wait until it is back expr: oz0_edge_state{state="fail-static"} == 1 for: 1m - alert: RegistryReloadRefused # an edge kept its last good registry; confirm the change if it is meant expr: increase(oz0_registry_reloads_refused_total[10m]) > 0 - alert: LocalModelSlow # a node's own model server's first token got slow: busy, or too small for its model expr: histogram_quantile(0.9, sum by (node, model, le) (rate(oz0_llm_first_token_seconds_bucket{provider="local"}[15m]))) > 5 for: 15m - alert: NodeLosesSessions # harness sessions lost with a node: drain it before maintenance, look at why it stops expr: sum by (node) (increase(oz0_harness_sessions_total{outcome!="drained"}[1h])) >= 3 - alert: MeteringLost # spend or a gateway record the edge could not keep: look at the database first expr: increase(oz0_metering_writes_retried_total{outcome=~"lost|spool_dropped"}[10m]) > 0 or increase(oz0_gateway_records_total{outcome=~"lost|failed"}[10m]) > 0 ``` ## Traces Every job is one OpenTelemetry trace. It starts on the edge, which starts the job's workflow, and continues on the nodes that run it: - the job's workflow and every child job, on whichever nodes they run; - each activity: model requests, tool calls, hooks, the job's start; - each model call, with its model, provider and tokens, in OpenTelemetry's GenAI conventions (`chat claude-sonnet-5-5`, `gen_ai.usage.input_tokens`); - each agent's run and each of its tool calls (`invoke_agent`, `execute_tool`). Send them to any backend that takes OTLP over HTTP, such as an OpenTelemetry Collector, Jaeger, Grafana Tempo, Honeycomb or Langfuse: ```bash [Terminal] oz0 server start --otlp-endpoint http://collector:4318 oz0 server start --otlp-endpoint https://otlp.example.com --otlp-header-file Authorization=/etc/oz0/otlp-authorization OZ0_OTLP_HEADERS="X-Scope-OrgID=team-a" oz0 server start --otlp-endpoint https://otlp.example.com ``` A header's value comes from a file (`--otlp-header-file NAME=FILE`, the file holding `Bearer ...`) or from `OZ0_OTLP_HEADERS` (`name=value,name=value`, as OpenTelemetry's `OTEL_EXPORTER_OTLP_HEADERS` writes it, values percent-encoded where they hold a comma), never from an argument, which every user of the machine can read. The Helm chart mounts `otlp.secretHeaders` from their Secrets as files. Nodes send their spans to the edge, never to the backend: the edge tells them at connect whether it relays spans, and sets `oz0.tenant` and `oz0.node` on every span from the node's certificate, so a node cannot pass its spans off as another's. The edge's own spans carry `service.name` `orchestrator-zero`, the nodes' `oz0-runtime`. Prompts and answers stay out of the spans, because they may hold your customers' data: a span shows the shape of each message, not its text. Without `--otlp-endpoint`, nothing is traced and nodes export nothing. The edge does not trace the live stream's polls, so watching a job adds no spans of its own. Harness sessions are one activity span today; the Claude Agent SDK's own spans inside a session do not join the job's trace yet. ## Live view The live stream works from the CLI, the console and the runtime API's `WatchJob`, on Temporal's Workflow Streams. Each viewer follows the stream on its own today; the edge will subscribe once per job and fan it out, so many viewers do not load Temporal. ### Pool event metrics What happened to the members of [pooled provider records](https://docs.oz0.ai/spend/models-and-keys/deployment-pools#pools-of-deployments): `429`, `5xx`, `4xx` (a `401`, `403` or `404`: the member refused the key, or has no such deployment), `unreachable` (nothing left the edge), `cooldown` (a cooldown started), `moved` (a call went on to another member before any output) and `unsigned` (the edge could not sign a call for it because of its own key, such as a key secret that is not set; counted at most once a minute per member). A call moved off a member is counted once in `oz0_llm_calls_total`, under the member that served it ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.