--- title: "Queues" description: "Why a job is waiting, and what each task queue holds, its backlog and the nodes that poll it." canonical_url: "https://docs.oz0.ai/fleet/queues" --- # Queues > Why a job is waiting, and what each task queue holds, its backlog and the nodes that poll it. For: cluster operators. Use the queue list to see which workers can take a job. Read queue names from the system because shared pools and rollout queues have different naming rules. Every job runs on Temporal task queues. This page is about what waits where and why: a job's pending steps and what each waits for, and every queue with its backlog and the nodes that poll it. Queues and "why is it waiting?" come from Temporal, through the edge. Errors grouped by what they say, KPIs, deep dives into agents, tools and nodes, search, tags and job export are on [Errors and insights](https://docs.oz0.ai/improve/errors-and-insights). ## Why is a job waiting? ```bash [Terminal] oz0 job why ``` ```text [Output] It waits for its child job job-41c0/helper-0a1b (helper): it waits for a person to approve team.hello, asked 2m10s ago; it is denied if nobody answers by 2026-10-04T09:12:00Z. All it waits for: approval It waits for a person to approve team.hello, asked 2m10s ago (in child job job-41c0/helper-0a1b) child It waits for its child job job-41c0/helper-0a1b (helper): ... Open steps: KIND NAME QUEUE STATE ATTEMPT NODE child helper - started 0 - ``` The answer looks at the job's open steps, the queues they wait on, the nodes that could take them, and its child jobs, three levels down: the first 10 open child jobs of each job, and 24 child jobs in all, children before grandchildren. The others are listed as open steps, with a line saying how many were not looked into; `job why` on one of them says what it waits for. What it can say, most important first:

Code	It means
`approval`	A person must approve or deny a step; the deadline is when it is denied
`budget`	A quota refused a model call and the job waits for a budget decision, on the budget request it names; or the job's own budget is spent, so the gateway refuses its model calls and the next one fails the job with `BudgetExceeded` . This is a reason it waits, not the `budget` finding , which says the job has spent 80% of its own budget
`not_on_node`	Nodes without the job's agent took its next decision and gave it back, and it still waits (how often, and the nodes Temporal's history names; refusals before the job's last completed decision do not count), for example while they install the plugin version that has it; it also says whether any node has the agent, or that no installed plugin defines it any more
`no_node`	No node may take the work: none has the agent with the labels it needs, or the flow's plugin
`unhealthy`	The nodes that could take it are offline, unhealthy or blocked
`no_poller`	Nodes could take it, but none polls the queue now: their runtime may be starting
`rate_limit`	A provider refused for now; the step tries again at the time shown
`retry`	A step failed and runs again at the time shown
`backlog`	The work is in line behind other tasks on its queue
`workflow`	The job's next decision waits for its home node, or its workflow code fails on the node
`rollout` , `gate`	It runs on a rollout's or a quality gate's own queue, which only some nodes poll
`timer`	The job sleeps until a time
`child`	A child job runs; its own reasons are listed too

A job of an agent at its [`limits.concurrent`](https://docs.oz0.ai/reference/agent-definition) shows `retry` on its start: the start asks again every 5 to 7 seconds and goes ahead at its first ask after one of the agent's jobs completes, fails or is cancelled (within about 30 seconds after one is stopped or times out), so a long wait there means the agent's jobs are long, not that the queue is stuck. In the console, a running job's page says the same above its steps, and links to the queue. It asks again every 15 seconds while the tab is in view. ## Queues ```bash [Terminal] oz0 queue list oz0 queue list --node-queues # also each node's own queue ``` ```text [Output] QUEUE KIND STATE BACKLOG OLDEST POLLERS PLUGINS AGENTS REASON pool.team.3eb7c03b pool no_pollers 3 4m12s - team trainer Work waits on pool.team.3eb7c03b, and no node may take it: no node runs the plugin team and has the labels gpu=a100 pool.team pool ok 0 - node-1 team lead,helper,triage 1 node polls it pool.ops pool idle 0 - node-2 ops deploy nothing waits; 1 node polls it ``` A plugin's agents and flows share one queue, a pool, for each set of node labels they need ([routing](https://docs.oz0.ai/concepts/routing)); PLUGINS names the plugin and AGENTS whose jobs it carries. With [shared pools](https://docs.oz0.ai/concepts/routing#shared-pools) a pool (`pool.~`) carries the agents and flows of every plugin with one node selector and label set: PLUGINS names them all, the console's dialog links each, and when nobody polls it "why is it waiting?" names the plugin each node lacks, since a node polls a shared pool only while it runs every plugin of it. Its backlog counts every plugin's tasks together, so read one plugin's waits from its jobs. Read a queue's name from here (`--json` for scripts), never build it from a plugin's name. AGENTS names whose jobs it carries, and the dialog in the console lists them all with the labels the pool needs. A queue of kind `agent` or `flow` is a legacy queue from before pools that jobs still run on. Each queue shows Temporal's backlog of workflow tasks (a job's next decision) and activity tasks (its steps), how long the oldest has waited, how fast tasks arrive and leave, the nodes that poll it now, and the nodes that may, with their health. Queues with a problem come first. The Queues page in the console has the same list, refreshed every 5 seconds, and a dialog per queue. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.