Routing
For: cluster operators and plugin authors.
Temporal routes jobs through task queues. Each node polls only the queues it qualifies for, and Temporal gives a task to the first free poller.
The queues
| Queue | Polled by | Carries |
|---|---|---|
pool.<plugin> | Every node of the tenant that runs the plugin | The jobs of the plugin's flows and of its agents that require no labels |
pool.<plugin>.<labels> | Every node that runs the plugin and has every label of one set | The jobs of the plugin's agents that require that set of labels |
pool.~<pair> | With shared pools: every node that matches one node selector, has one label set and runs every plugin of the pool | The jobs of every plugin's agents and flows with that selector and label set (shared pools) |
node.<id> | One node | Tool calls for jobs whose home is that node, and other work that must run there |
oz0.default | Every node | Work for any node, such as tool calls for tools with affinity: any |
| Extra queues | Nodes whose queues label lists them | Work an operator wants on a chosen set of nodes |
A plugin's agents share a queue, a pool, for each set of labels they require; <labels> is a short hash of the set. The job names its agent, and the node that takes it runs that agent: every node that polls a pool runs every agent in it. So a tenant with 1,000 agents in one plugin has one queue, and the queues a node polls stay few however many agents there are. oz0 catalog shows each agent's pool, and oz0 queue list the agents each pool carries.
An agent's or flow's name is unique within a tenant, across agents and flows, so a job always means one definition.
During a plugin update a node goes on serving the plugin's pool with the version it has until the new one is ready, so it can be offered a job of an agent that only the new version has. It hands such a job back instead of failing it, and the job runs on a node that has the new version (limits and scaling).
| Prefix | Means |
|---|---|
gate.<commit>.<pool> | A new version under its quality gate, on its test node (evals): one per pool its agents and flows fall in, shared or its own |
roll.<commit>.<pool> | A new version in a rollout, on the nodes it has reached (install): one per pool, and only that plugin's jobs are routed there |
cand.<id>.<pool> | A developer's private candidate, on the nodes they chose, for the jobs that name it (develop and test) |
sandbox.<queue> | The sandboxed work of a job queue, which only nodes with sandbox tier 1 poll |
Read a queue's name from oz0 queue list --json, the console's Queues page or the job, never build it from a plugin's name: with shared pools a job's queue no longer names its plugin.
Before pools each agent had a queue, agent.<name>, and each flow flow.<name>. Jobs that started there before an upgrade finish there: the nodes that run the agent poll such a queue while the tenant has jobs on any of them, together with the queues of the agents those jobs may delegate to, and stop ten minutes after the last one ended. The set changes rarely on purpose, since a node restarts its runtime when its queues change.
Shared pools
Servers start with --queue-pools=shared unless told otherwise: a tenant gets one pool per node selector and label set across its plugins, instead of one per plugin and label set (--queue-pools=plugin, the fallback for a fleet whose nodes cannot poll shared pools yet).
Plugins installed without a selector, with agents that require no labels, share one pool; each label set in use adds one, and so does each selector. A team of two dozen plugins then uses a handful of queues instead of a queue per plugin, which keeps the tenant well within the queue limit, Temporal Cloud's included, and each node polls a handful of queues.
The pool's name is pool.~ and the first 8 hex digits of the SHA-256 of its pair: the selector's key=value lines in key order, a line --, then the labels' key=value lines in key order (each line ending with a newline). No plugin name holds ~, so a shared pool's name is never a plugin's. oz0 queue list names each pool's plugins (PLUGINS) and agents.
What keeps routing safe:
- A node polls a shared pool only while it runs every plugin of it. The edge tells each node the members of every shared pool it may poll.
The node starts polling once it runs the version of every member the edge assigns; it goes on while a member updates, or after an update of one failed (the old version hands back a job of an agent only the new version has, as on a plugin's own pool); and it stops while a member is one it has never run, such as a plugin newly installed into the pool, until that plugin is ready.
"Why is it waiting?" names the plugin a node lacks. - A job is its agent's plugin's. The edge finds the plugin of a job on a shared pool from the agent or flow the job runs (for an event delivery or a chat post, from the event source or channel the edge started it for), never from the queue: a call or a child job that only another plugin of the pool was granted is refused, and the node's runtime fails a job whose agent's plugin is not a member of the pool it runs on.
- A plugin no node may start yet keeps its own pool. A plugin whose required settings or secrets the tenant has not set is left out of the shared pool, so it does not keep every node from polling it; its jobs wait on
pool.<plugin>with what is missing named, and once it is set the plugin joins the shared pool. - One plugin cannot take the pool. Every job on a shared pool carries its plugin as Temporal's fairness key, so the plugins' jobs take turns on the pool instead of waiting behind one plugin's backlog; a child job carries its own plugin's key.
With the embedded Temporal,--queue-pools=sharedturns fair dispatch on for every queue from the server's start, and it stays on after a switch back; an external cluster turns onmatching.enableFairness, and Temporal Cloud its namespace's fairness.
Without fair dispatch the keys do nothing, and one plugin's backlog delays the others' jobs; turn it on before the servers switch, or keep--queue-pools=plugin(upgrades). An agent can also cap its own jobs withlimits.concurrent: past it, its jobs wait to start until one of them ends.
A new version's quality gate, rollout and private candidate follow the shared pool, each on its own prefix of it (gate.<commit>.pool.~<pair>, roll.<commit>.pool.~<pair>, cand.<id>.pool.~<pair>), which carries that one plugin's jobs: the edge routes only the plugin's own jobs there, so a node polls such a queue once it runs the candidate, whatever else the pool carries.
A gate's test node, and a rollout's nodes until its last step, take none of the plain shared pools the plugin is a member of meanwhile, so no job of the fleet reaches the candidate; the pool's other plugins run on the tenant's other nodes until then.
A gate therefore prefers a node whose leaving keeps another online node on each of those pools; when it can test only on the last node of a pool, as on a one-node dev server, the pool's other plugins wait until the gate ends, and the gate's reason names them.
A rollout needs two nodes or more, and its canaries leave the pool while the others carry it. A rollout's share of new jobs is counted per pool over the nodes with its labels, as before.
Switching the setting either way drains: jobs that started on the other mode's names finish there. The edge looks for them every 30 seconds, the nodes that run their plugins poll those names besides their pools, and stop ten minutes after the last one ended. The same holds for a plugin that an update moves to another selector or label set: its running jobs finish on its old shared pool, on the nodes that run it. Routines' schedules move to the new names at the edge's next pass. See upgrades.
Labels decide where agents run
An agent can require labels:
requires:
labels: {gpu: "a100"}
Only nodes with gpu=a100 poll the pool of the plugin's agents that need gpu=a100, so only they take trainer's jobs. Labels come from the join token or from the operator who accepts the node. Plugins can be limited to nodes with a label as well, with plugin install --select gpu=a100.
Tools follow the job
The node that picks up a job is its home node. Tools with the default affinity, node, run on the home node's queue, so an agent can work with files, devices and local models on that machine. The home is chosen when the job starts and recorded in its history, so it stays the same if the job's steps move between nodes.
Tools with affinity: any run on any node that has the plugin. Use it for stateless tools such as a web fetcher.