Tenant and queue limits

For: cluster operators and plugin authors.

The limits

Shared management work

Limit

Why

Each holds memory or disk the tenants share, so one tenant never holds every place and each is held for a bounded time: a busy one is refused with resource_exhausted and says whether the server or the tenant is full, a client that stops reading an export or reads it more slowly than 1 MiB/s on average loses its place (deadline_exceeded, export again from a faster connection), and so does one that sends an upload too slowly

Management proxy request bodies

Limit

For calls from people and operators, a server holds at most 320 MB, with at most three quarters used by one caller. It counts the memory occupied by bytes as they arrive, including each buffer through which a body grows, so a body can count as about twice its size.

A separate 32 MB allowance holds calls anyone can make, including anonymous calls and app API keys, only so they can be sent to another edge.

A body that sends nothing for 30 s, or averages less than 64 KiB/s afterwards, is cut off with 408. Its memory is released as soon as the answer begins.

Why

Management reads a person's or an operator's body whole to sign it for the edge, so many large bodies at once could exhaust its memory: such a call past the bound gets 503 with Retry-After.

Only bytes that came count, so calls that say they are large and then trickle or stop hold nothing for others, and a long answer, a stream, or a caller that reads slowly holds no body.

Calls nobody signs never take that room and never get that 503: past their own bound, before or while their bytes arrive, they are streamed to one edge, unheld, and are not sent again if that edge is fenced. A body of a known length streamed so has the time reading it would have had, and is cut off past it

YAML input bounds

Limit

1 MiB, a manifest or a routine 256 KiB; 1,024 keys in one mapping, each key once, and 16,777,216 pairs of keys to compare in all, each alias of a mapping counting its pairs again and long keys counting for more. Aliases may add four times the document's bytes and 64 KiB more, a node counting for 16 bytes past its text. An OpenAPI document or a lockfile 16 MiB, 16,384 keys in one mapping and 268,435,456 pairs. A routine has at most 64 triggers and 64 outputs

Why

A YAML decoder compares every key of a mapping with every other, and an alias multiplies what it names, for the decoder and for whatever reads the result, so a few kilobytes could cost a server every tenant shares tens of seconds and gigabytes. The check takes one pass and refuses with one line, before the decoder runs.

JSON Schema bounds

Limit

256 KiB, 128 levels deep, 4,096 objects, lists and booleans, keys of 1 KiB, 1 MiB of paths to them in all, 256 $id, 256 places named by references outside definitions and, from draft 2019-09 on, $defs, and at most 16,384 schemas applied to one value with every reference followed.

A regular expression has 1 KiB of text and about 4,096 instructions, counted before it compiles, and a schema's come to about 131,072 instructions in all.

A number has 128 characters and stays within 1e400. A schema that reaches itself without moving into the value, and $dynamicRef and $recursiveRef, are refused

Why

The compiler's work grows with the square of what a schema holds, and references let a small schema double at every step. A schema past the bound is refused where it is installed or saved, and refuses every value where it is met: it is never skipped.

Schema evaluation bounds

Limit

131,072 steps and 2 more for each of its bytes, at most 2,097,152; 4,096 levels of schema and value together; numbers as above. A step is one schema applied to one value, more for a long string, a wide object, a regular expression, a comparison, a long number, and for each comparison uniqueItems makes between items that hash alike; a list of integers takes 4 steps an item

Why

Each visit reads the value, so that small schemas can ask for minutes on a value of a megabyte. The steps are charges measured on a set of named shapes, not a proof: on those, the ceiling is about half a second of the validator's work. Past it the value is refused, never let through.

plugin install refuses a plugin that would take its tenant past either limit, and says which:

invalid request: tenant acme's agents and flows would use 51 task queues, and a tenant may use at most 50: a plugin's agents share one queue for each set of node labels they require, and each queue is registered in the tenant's Worker Deployment

A tenant that is past a limit already, such as one that had more than 50 single-agent plugins before pools, keeps its plugins and may still update them: an install is refused only when it would raise the tenant's count. A patch or a security fix that adds no queue goes through; a new plugin, or an update that adds a label set, does not.

Agents and flows are found by name, so within a tenant an agent and a flow of different plugins may not share a name either. A version that waits for its quality gate or rolls out holds its names as an installed one does, since it becomes the installed one without another check. A clash that an installed plugin already had is kept on its updates.

Tool IDs are unique per tenant too: agents, hooks, approvals, tool snapshots and the tool policy name a function <tool>.<function> without its plugin, so plugin install refuses a tool ID another plugin of the tenant provides.

Two plugins installed before that rule may share one; their updates keep it, plugin list and the Tools page show it, and a server logs it and records it in the audit log (plugin.tool_clash) when it starts.

No node runs either plugin's server of such an ID, and jobs whose agents or flows use it fail when they start, until an admin removes one of the plugins or updates one to a version whose tool has another ID.

The queue limit is 50, or a third of what a Worker Deployment version holds, less the shared queue, whichever is smaller: an update registers its pools again on its quality gate's queues and on its rollout's queues, so a version must hold three times the usual queues.

With shared pools a gate or a rollout adds one name for each pool its plugin's agents and flows fall in, usually one or two, and plugin preview says how many; those names stay in the runtime version until the next one, so a tenant that updates its plugins many times within one runtime version uses that room up, one name per update and pool.

The embedded Temporal holds 1,100 per version; Temporal Cloud holds 100 by default, which gives 33. The 33 applies once the server runs with the cloud backend, which tells the limit how many queues its versions hold; until then the limit is 50 whatever the backend.

10,000 agents in one tenant

make load-scale installs 10,000 agents in ten generated plugins in one tenant, on one node, and then ten tenants of 1,000 agents with a node each, and times what a tenant's size reaches: the install, the catalog, the inventory and its export, MCP's tools/list, the chat picker, Ask, the OpenAPI document and a typed client, the routes each node gets, and the runtime's start. It stops itself when the machine's free memory falls under 4 GB. On a 16-CPU, 30 GB development machine under WSL2 that ran other builds at the same time (a load average of 7 to 22):

Ten tenants of 1,000 agents ran side by side: their ten runtimes took 2.7 GB together, 275 MB the largest, and each tenant's lists answered in milliseconds.

So a tenant's lists come in pages (at most 100 agents and flows since the console's lists moved to pages of 25 to 100; the measurements above were taken with pages of 1,000), with a search on the server (the catalog, the inventory, MCP), Ask searches the agents you may start instead of reading them all, and the cap is 5,000: the run held at 10,000, and half of it leaves room for a busier machine and for real agents' larger prompts and schemas.

A node that runs 5,000 agents needs about 600 MB for its runtime and half a minute to start it.

How many queues a plugin uses

With shared pools, the default, plugins with the same node selector share the queues of the label sets their agents require, counting a local model (local/<model>) as the label model.<model>=ready; flows and agents without labels share the first. So the limit counts the distinct pools of all of the tenant's plugins together:

• 50 plugins of one agent each, with no selector and no labels, use one queue;
• ready plugins with the same selector and required labels share a queue; plugins with no agents or flows use no agent queue;
• each node selector and each label set in use adds one: a plugin installed with --select region=eu gets the pool of that selector, and agents that need gpu=a100 the pool of that label.

A plugin no node may start yet, because a setting or secret it requires is missing, takes a queue of its own until it is set.

Per plugin (oz0 server start --queue-pools=plugin), each plugin uses one queue per distinct set of labels its agents require: a plugin of 1,000 agents with no labels uses one, a plugin whose agents need gpu=a100 or nothing uses two, and 50 plugins of one agent each use 50, the default embedded limit. Plugins with no agents or flows use no agent queue.

oz0 catalog shows each agent's queue, and oz0 queue list the plugins and agents each queue carries.

What a pool means for capacity

A node runs one worker per queue it polls, and each worker takes up to 100 activities at once (half while the node is degraded). The agents of a pool share their worker, so a node takes up to 100 of their model and tool calls at once, whichever agents they are for. A plugin that needs more capacity per node can split its agents by labels, at the cost of a queue each.

A shared pool carries every member plugin's work on the node, so its worker takes 100 activities at once for each member plugin, up to 50 per logical CPU of the machine (never fewer than 100), half while degraded. oz0-node run --activity-slots N (or OZ0_ACTIVITY_SLOTS in the node's environment) sets the activity slots of every queue the node polls instead, from 1 to 10,000.

Within a shared pool the plugins take turns: each job carries its plugin as Temporal's fairness key, so a plugin with a long backlog does not keep the others' jobs waiting behind it.

An agent can cap its own jobs with limits.concurrent (1 to 1,000): at most that many of its jobs run at once in the tenant, counting child jobs (a job under one of the agent's own jobs, such as a best-of candidate, runs in that job's place), and a job past it waits at its start until one of them ends, asking again about every 5 seconds; it is never refused for it.

A job's place frees as soon as the edge that passed its last step to Temporal (the step that completes, fails or cancels it) has checked that the job no longer runs, so the next waiting job starts at its next ask, within about 7 seconds, and an agent at its limit keeps about N jobs running under a steady backlog.

A job that ends another way (stopped by a person, timed out, or ended through an edge that restarted since) frees its place within about 30 seconds, when the server next looks for ended jobs or a waiting start finds the agent full.

make load-saturate measures it: one plugin starts 200 jobs at once in the tenant's default shared pool, and four other plugins of the pool start 5 jobs each right behind them, on one node with 10 activity slots and a model that answers after a second. The quiet plugins' jobs, median:

Fairness keys let a quiet plugin's job start within seconds behind another plugin's backlog. limits.concurrent prevents a busy agent from taking the whole node, at a cost to its throughput. Its 200 jobs took 178 to 206 s, against 126 s without the limit on the same machine that day. When a place freed only every 30 seconds, they took 489 to 699 s.

A pool per plugin separates queues but not the node, whose runtime serves all the plugins. The quiet jobs therefore took as long in the end. Two runs of a variant differed by up to 15 seconds on a machine shared with other builds.

When a plugin update adds an agent

A node goes on serving a plugin's pool with the version it runs while it installs a new one, and for as long as it runs that version when the install fails.

So it can be offered work for an agent or flow that only the new version has: a new job's first step, or the next step of a job that is already running on another node, for example after that node's runtime restarted.

It does not fail that job: the runtime hands the step back to Temporal, which offers it again, to a node that has the new version or to this one once it has loaded it.

The job waits meanwhile, at first for 10 seconds between offers and longer the longer it waits (at most about 10 minutes); the node logs agent 'x' is not on this node (or flow 'x') each time.

A node whose install of the new version fails keeps handing such steps back, so the jobs run on the nodes that have it; if no node has it, they wait until one does.

A job of an agent or flow that an update removed from every node waits too, as it did before pools, until you cancel it or its time limit ends it.

More than the limits

• More agents: split them across tenants; each tenant has its own Worker Deployment and its own limits. A load test may raise the cap with the server's hidden --max-agents-per-tenant (in split mode, on the management role, where installs are checked); past 5,000 is what we have not measured with margin.
• More queues: the embedded Temporal's per-version limit is dynamic configuration (matching.maxTaskQueuesInDeploymentVersion), but registration time grows faster than the queues do, so the limit stays where it was measured.