Capacity measurements

For: cluster operators and plugin authors.

What we measured

make load-agents installs one plugin with LOAD_AGENTS agents on a dev server with one node, runs a job on the first, the middle and the last agent, then upgrades the node to a second runtime version and runs the three again. LOAD_POOLS spreads the agents over that many label sets. On a development machine that ran eleven other builds at the same time:

The times count from the moment the install or the upgrade began, and include the runtime's own start. The node had sandbox tier 1, so it also polled a sandbox queue next to each pool. The number of agents costs little; the number of queues is what costs time.

Before pools, every agent was a queue, and a node with sandbox tier 1 polled a second one per agent: on the same machine, 150 agents made the first job wait 353 seconds after the install, and after a runtime upgrade the new version was not current five minutes later. An earlier run measured 539 seconds for the last of 300 agents.

A tenant of many small plugins

Plugins are meant to be small: a connector for each outside system, a capability for each job, and agents of one file each. With shared pools, the default, that costs no queue of its own: a tenant's plugins on the same nodes share one pool per label set their agents require, so a tenant of many small plugins uses a handful of queues.

Per plugin (--queue-pools=plugin), each plugin with an agent or a flow takes at least one queue, however small it is: a connector with tools only takes none, a connector that carries an agent one, a capability with an agent and a flow one (two when its agent requires a node label, since its flow needs none), and an agent of one file one.

make load-tenant installs tenants of that shape on a dev server, one plugin at a time with a job after each install, then runs a job on every agent and flow at once and upgrades the runtime. On a 16-CPU development machine that ran other builds at the same time, with one node, one in five of the capabilities' and one-file agents requiring one of two node labels:

The first row's queue count comes from the plugins' real manifests; its times come from stand-ins with the same names, kinds, agents and flows and one generated tool each, since the real plugins wait for their outside systems and required settings before a node starts them.

The other rows are generated plugins of the shape they name. "Every job after the last install" runs a job on every agent and flow at once, timed from the last install's start, so it includes the runtime's restart.

The per-plugin figures in brackets were measured before shared pools.

Two per-plugin runs of the team on the same build as the shared ones, interleaved with them on a machine whose other work moved every run by up to 20 seconds, took a median of 21.0 and 24.8 s to the first job after an install (shared: 29.6 and 23.3 s), and 22.6 and 44.4 s to the runtime upgrade (shared: 21.1 and 27.2 s): with 3 queues or 21, most of the wait after an install is the runtime's restart, which starts the plugins' tool servers again, and the fewer queues show after an upgrade.

Per plugin, the large tenant's last five one-file agents were refused at install, since they would have taken it to 51 queues and more; with shared pools all 60 plugins installed on 3 queues. Temporal refused no queue in any run.

Most of the wait after an install is the runtime's restart: a node restarts its runtime whenever its plugins change, and the runtime starts its plugins' tool servers again. Its start took about 7 seconds with one plugin and a median of 24 to 36 seconds with 20 to 30 tool servers, which is why the first job after an install took longer the more the tenant had installed.

Near the limit

• oz0 plugin preview <source> says what the install would take of the tenant's limits, before anything is installed:
  Output

  The tenant's plugins would have 26 of 5000 agents and flows, on 3 of 33 task queues.
  A quality gate or a rollout of it adds 2 task queue name(s) to the runtime version.
  

• oz0 queue list lists the tenant's pools with the agents each carries, and oz0 catalog each agent's queue.
• An install that would pass the limit is refused with the message under The limits; the tenant's other plugins keep running, and updates that add no queue still go through.

To stay further from the limit, keep the labels an agent requires to the ones it needs and the node selectors of plugins to the ones the fleet needs (each pair is a queue), and on Temporal Cloud give a large organisation more than one tenant. Per plugin, also keep the agents of one job in one plugin rather than one file each when they need the same nodes.

A fleet under load

make load-fleet runs a cluster of two servers on PostgreSQL with the embedded Temporal, a fake model API, two nodes with the real Python runtime and 60 more nodes, and runs thousands of jobs through it.

The 60 are real node supervisors, unchanged, that run a small fake runtime instead of the Python one: everything between a node and the model API is the product's (the join, the control stream, the node's local Temporal and LLM proxies, the edge's Temporal proxy and its checks, the LLM gateway and its metering, the registry, the Worker Deployment and Temporal), and only the agent loop is fake, so that one machine holds the fleet.

The jobs are agents that call the model twice and a tool once, leads that hand work to four agents at once, and flows, 200 at once in the fleet and 20 at once on the Python nodes.

These numbers are what one development machine gave: 16 CPUs and 30 GiB under WSL2, shared with eight other builds and their tests (a load average of 26 to 107 during the runs), and a PostgreSQL 16 container they shared too. A run's numbers move with the machine by a factor of two; the report of each run states how long a bare commit took meanwhile. They are not the product's ceilings, which a dedicated database moves.

make load-jobs runs the same cluster with 8 fleet nodes and 5,000 jobs, 300 at once. In a quiet minute (a bare commit at 2.1 ms) it did 1,678 jobs a minute and 96 model calls a second through the gateway, with both servers at 3.5 cores; the Python nodes did 157 jobs a minute beside it. A job then waited p50 6.9 s for one of the 8 nodes to take it: at that rate the nodes, not the servers, were full.

What limits what

• The database. The servers were never busy; they waited for PostgreSQL. In the database, the statements running most often were Temporal's commits and its history inserts. A faster disk or a database of its own raises every number above; more history shards spread the writes (the test used 16; server init makes 1,024).
• Minutes-long tails when the database is slow. In most runs a job in a few hundred waited minutes: Temporal recorded one of its tasks as started, but no node received it, because the database was slower than Temporal's matching allows for (a second), and the task waited for its timeout: 10 s for a workflow task, the 5 minutes of a tool call for a tool call. The job then finishes. On a database that keeps up it is rare.
• Long polls per frontend. Each node polls each of its task queues (the shared queue, its own, and its plugins' pools) with five pollers of each kind. Temporal's frontend takes 1,200 long polls of each kind per namespace, which held about 80 nodes of a tenant per server: 100 nodes on one server got "namespace concurrent poller limit exceeded" and some jobs failed. The embedded Temporal now takes 20,000 per server. With an external Temporal, raise frontend.namespaceCount in its dynamic configuration; Temporal Cloud allows 20,000 pollers of each type per namespace.
• Nodes per edge. An edge holds about 30 long polls and one control stream per node, and a server about 190 goroutines per node: 100 fleet nodes on one server ran 1,000 jobs without a failure once the frontend took their polls, with the server at 1.2 GiB.
• Joins from one address. The edge takes one join a second from an address after a burst of ten, and a node waits for its turn: 60 nodes joining at once from one address through two edges took 20 s, 100 through one edge 86 s.
• Model calls of one owner. Each model call is charged to its job's owner and team, on one row each per day; one edge writes the charges that come together as one. Before that, 64 calls at once for one principal took 7.6 ms each to charge on this machine, about 130 a second; now 0.3 ms.

Other Temporal backends

The same test runs against a Temporal the servers do not run (LOAD_BACKEND=external, scripts/faketemporal: one process with SQLite) and against the same behind a stand-in for Temporal Cloud (LOAD_BACKEND=cloud, scripts/fakecloud: TLS, an API key per namespace and Cloud's namespace names, with every payload encrypted), with 20 fleet nodes and 300 jobs at 50 at once:

The namespace rewrite, the API key, TLS and the stand-in's own proxy cost 0.2 to 0.4 ms a call. What makes the stand-in slow is the stand-in: one process on SQLite, behind one more proxy. It records a task's start more slowly than Temporal's matching allows for (a second), so some tasks count as started and reach no worker, and wait for their timeout (10 s for a workflow task, the activity's own for an activity).

The same happens on PostgreSQL, rarely; it is where the minutes-long tails above come from. These are not Temporal Cloud's numbers: a run against Cloud itself needs an account.

Run it yourself

make pg-up
make load-fleet                              # 2 servers, 60 nodes, 2,000 jobs, 3 blocks
make load-jobs                               # 8 nodes, 5,000 jobs: throughput
LOAD_BACKEND=cloud LOAD_NODES=20 make load-fleet

LOAD_SERVERS, LOAD_NODES, LOAD_REAL_NODES, LOAD_JOBS, LOAD_CONCURRENCY, LOAD_REAL_JOBS, LOAD_MIX and LOAD_BLOCKS change the run; LOAD_BIN_DIR runs another build's binaries for a before and after, best with LOAD_PROFILE=0, since only a build with --pprof is profiled.

Each run leaves report.md, summary.json, a CPU and heap profile of the first server, the median and the slowest job's history, and every log, in ~/.cache/oz0-load/<time>-<backend> (or LOAD_OUT).

A fleet node takes about 75 MiB in all, so 60 need about 4.5 GiB. make load-fleet took 9 minutes on this machine, make load-jobs 4 to 5.

A run exits 1, after it has written its report, when a job failed or did not start, or when a block did not reach every edge within a minute (the report says "never" for that edge); LOAD_REAL_NODES must be 1 or more, since the first real node also fetches uv and Python into the nodes' shared cache before the others start.

server start --pprof serves Go's profiles at /debug/pprof/ on the --metrics-listen address, for profiling a server of your own under load. It is not authenticated, and profiles and goroutine dumps reveal the server's internals, so keep the address private. It does not serve /debug/pprof/cmdline, to avoid exposing process arguments. Database credentials come from OZ0_DB or --db-file, never a --db argument.