Tracing and metrics

For: cluster operators.

Enable a metrics listener for Prometheus or an OTLP destination for traces. Use the measurement definitions below when building queries and alerts.

Metrics

Each server serves /metrics for Prometheus when you give it an address:

oz0 server start --metrics-listen 127.0.0.1:9464 ...
curl -s http://127.0.0.1:9464/metrics | grep '^oz0_'

The endpoint is plain HTTP and not authenticated, so keep it on a private interface or behind your scraper's network. Each server counts what it saw itself; let Prometheus add them up across a cluster.

The three timing histograms name a node only for calls to a node's own model servers (provider local: its local calls and the fleet's calls it served), so you can size one machine against another; a cloud model's calls have node="", since the node that asked says nothing about the cloud model's speed.

They count models and machines, never people: no metric has a person's or a team's label. A call that did not measure a figure is not counted in its histogram: an unstreamed call has no time to first token, and a call to a server that reports no timings of its own and was not streamed has no decode speed.

Go's and the process's own metrics (go_*, process_*) are there too. A useful alert to start from:

- alert: NodeUnhealthy
  expr: oz0_node_health_state{state="unhealthy"} == 1
  for: 2m
- alert: EdgeFenced # this server lost the database while others did not
  expr: oz0_edge_state{state="fenced"} == 1
- alert: EdgeFailStatic # no edge reaches the database: blocks wait until it is back
  expr: oz0_edge_state{state="fail-static"} == 1
  for: 1m
- alert: RegistryReloadRefused # an edge kept its last good registry; confirm the change if it is meant
  expr: increase(oz0_registry_reloads_refused_total[10m]) > 0
- alert: LocalModelSlow # a node's own model server's first token got slow: busy, or too small for its model
  expr: histogram_quantile(0.9, sum by (node, model, le) (rate(oz0_llm_first_token_seconds_bucket{provider="local"}[15m]))) > 5
  for: 15m
- alert: NodeLosesSessions # harness sessions lost with a node: drain it before maintenance, look at why it stops
  expr: sum by (node) (increase(oz0_harness_sessions_total{outcome!="drained"}[1h])) >= 3
- alert: MeteringLost # spend or a gateway record the edge could not keep: look at the database first
  expr: increase(oz0_metering_writes_retried_total{outcome=~"lost|spool_dropped"}[10m]) > 0 or increase(oz0_gateway_records_total{outcome=~"lost|failed"}[10m]) > 0

Traces

Every job is one OpenTelemetry trace. It starts on the edge, which starts the job's workflow, and continues on the nodes that run it:

• the job's workflow and every child job, on whichever nodes they run;
• each activity: model requests, tool calls, hooks, the job's start;
• each model call, with its model, provider and tokens, in OpenTelemetry's GenAI conventions (chat claude-sonnet-5-5, gen_ai.usage.input_tokens);
• each agent's run and each of its tool calls (invoke_agent, execute_tool).

Send them to any backend that takes OTLP over HTTP, such as an OpenTelemetry Collector, Jaeger, Grafana Tempo, Honeycomb or Langfuse:

oz0 server start --otlp-endpoint http://collector:4318
oz0 server start --otlp-endpoint https://otlp.example.com --otlp-header-file Authorization=/etc/oz0/otlp-authorization
OZ0_OTLP_HEADERS="X-Scope-OrgID=team-a" oz0 server start --otlp-endpoint https://otlp.example.com

A header's value comes from a file (--otlp-header-file NAME=FILE, the file holding Bearer ...) or from OZ0_OTLP_HEADERS (name=value,name=value, as OpenTelemetry's OTEL_EXPORTER_OTLP_HEADERS writes it, values percent-encoded where they hold a comma), never from an argument, which every user of the machine can read. The Helm chart mounts otlp.secretHeaders from their Secrets as files.

Nodes send their spans to the edge, never to the backend: the edge tells them at connect whether it relays spans, and sets oz0.tenant and oz0.node on every span from the node's certificate, so a node cannot pass its spans off as another's. The edge's own spans carry service.name orchestrator-zero, the nodes' oz0-runtime.

Prompts and answers stay out of the spans, because they may hold your customers' data: a span shows the shape of each message, not its text. Without --otlp-endpoint, nothing is traced and nodes export nothing.

Live view

The live stream works from the CLI, the console and the runtime API's WatchJob, on Temporal's Workflow Streams. Each viewer follows the stream on its own today; the edge will subscribe once per job and fan it out, so many viewers do not load Temporal.

Pool event metrics

What happened to the members of pooled provider records: 429, 5xx, 4xx (a 401, 403 or 404: the member refused the key, or has no such deployment), unreachable (nothing left the edge), cooldown (a cooldown started), moved (a call went on to another member before any output) and unsigned (the edge could not sign a call for it because of its own key, such as a key secret that is not set; counted at most once a minute per member). A call moved off a member is counted once in oz0_llm_calls_total, under the member that served it