Migrate to shared worker pools

For: cluster operators.

Prepare compatible nodes and fair dispatch on external backends before switching pools.

Shared pools

From this release a server may share a tenant's task queues across its plugins: oz0 server start --queue-pools=shared gives a tenant one pool per node selector and label set, instead of one per plugin and label set (shared pools). It is the default from the release after it (shared pools by default); --queue-pools=plugin names every queue as before. What you may notice when you switch:

• Management puts it in force at its start, and every edge and node follows it within seconds. Give every management server the same value; the audit log records each change as cluster.queue_pools. In split mode the Temporal role takes the flag too, since shared pools turn Temporal's fair dispatch on from its start.
• Upgrade the nodes first. A node whose supervisor predates shared pools gets none of them (the edge logs a warning for it) and takes only the queues of its plugins that keep pools of their own.
• Running jobs finish where they started. Jobs on pool.<plugin> names keep running there: the edge finds them every 30 seconds and the nodes that run their plugins poll those names besides the shared pools, until ten minutes after the last one ended.
  Switching back to plugin drains the shared names the same way, on the nodes that run every plugin of each, and Temporal's fair dispatch stays on from then on, since the jobs still running there carry their plugins' keys.
  An update that moves a plugin to another selector or label set leaves its running jobs on its old shared pool; they drain there on the nodes that run the plugin.
• Runtime versions from before shared pools poll none of them. A job pinned to an older runtime version runs on that version as before, on the queue it started on; its new child jobs go where the edge's routes say, and one on a shared pool runs on the current runtime version, which polls it.
• Routines move at once. Each routine's schedule is rewritten to its target's new pool at the edge's next pass, and so is it when an update changes a plugin's selector or its agents' labels.
• Fewer queues and workers per node. A tenant uses a pool per selector and label set in use; oz0 queue list shows each pool's plugins. A shared pool's worker takes 100 activities at once for each of its plugins, up to 50 per logical CPU; oz0-node run --activity-slots sets another number.
• Scripts read queue names. A script that waits on a queue reads its name from oz0 queue list --json or from the job, never builds pool.<plugin> itself.

Shared pools by default

oz0 server start now shares a tenant's pools across its plugins unless you pass --queue-pools=plugin. Quality gates, rollouts and private candidates follow the shared pools too, each on its own prefix of the pool (gate.<commit>.pool.~<pair>, roll.<commit>.pool.~<pair>, cand.<id>.pool.~<pair>). The order:

1. On an external Temporal cluster or Temporal Cloud, turn fair dispatch on first. On a shared pool every job carries its plugin as Temporal's fairness key, so one plugin's backlog does not hold up the others; that works only where the backend dispatches fairly. The embedded Temporal turns it on by itself. An external cluster needs matching.enableFairness in its dynamic config, and Temporal Cloud the namespace's fairness setting.
   Without it the pool still works, but its plugins no longer take turns: in our saturation test the quiet plugins' first job waited 18 to 25 seconds behind one plugin's 200, against 1.6 to 7.1 seconds with the keys honoured (Limits and scaling). If you cannot turn it on yet, keep --queue-pools=plugin. oz0 doctor reminds you with a fair dispatch line on such a backend, since management cannot read the setting there.

Shared pools have been tested only on the embedded Temporal and on a stand-in of Temporal Cloud (backend validation). 2. Upgrade the nodes, supervisor and runtime, before or after the servers. A node's supervisor and runtime that know shared pools work with an older server too, so upgrading them first leaves no gap. A node whose supervisor predates shared pools gets none of them, and a supervisor gives none to a runtime that predates them: such a node takes only tool calls, the jobs pinned to its runtime and the work of its own queues, and the tenant's new agent and flow jobs wait for the upgraded nodes. 3. Upgrade the management servers and the edges. At its first start management switches the cluster to shared pools and records cluster.queue_pools in the audit log. A cluster with no plugin installed yet takes the mode without a switch, since nothing can be waiting on the old names.

What drains, and for how long:

• Jobs on pool.<plugin> names finish there. The edge looks for them every 30 seconds; the nodes that run their plugins poll those names besides the shared pools, and stop ten minutes after the last one ended. Their child jobs started after the switch go to the shared pools.
• A rollout planned before the switch keeps its queue names (roll.<commit>.pool.<plugin>) until it ends, and its nodes take the plugin's jobs on the new pools in its last step; its roll queues then drain as any rollout's do. A rollout that was still waiting for its turn starts on the new names.
• A quality gate testing at the restart runs again from the start on the new names, as after any server restart, and a private candidate answers on its new queues at once: restart oz0 dev for a job you started on the old ones.

What changes for gates and rollouts: the node that tests a candidate, and a rollout's nodes before its last step, leave the shared pools of the plugin they run a candidate of, since those pools carry the plugin's fleet jobs. The pools' other plugins keep running on the tenant's other nodes. A gate prefers a node whose leaving keeps another node on each pool; on a tenant with a single node, such as a dev server, the other plugins of the pool wait until the gate ends, and the gate's reason (oz0 plugin show) names them (Quality gates).

To keep per-plugin pools, start every management server with --queue-pools=plugin (in split mode the Temporal role too); switching back later drains the same way.

oz0 doctor has a shared pools line that names every node that cannot poll them, by its supervisor or by the runtime it runs, and oz0 node show and the console's Nodes page show the same warning on the node, with the fix: upgrade the node's binary (oz0 node upgrade --binary <node>), or move it to a runtime that knows shared pools (oz0 node upgrade <node> --runtime <version>, or let the runtime rollout under way finish), or run the servers with --queue-pools=plugin until you can.