Pool model deployments
For: tenant administrators and cost owners.
Use a pool when deployments have separate capacity. Members remain subject to the tenant's data rules.
Pools of deployments
A provider record may hold a pool: several deployments of the same models behind the same wire format, each with its own endpoint, key, region and data-keeping. For each call the gateway chooses one member, and when a member answers 429 (or a 5xx that is its whole answer) before anything reached the agent, the same request goes to another member while the first cools down. The agent sees one answer, and the call is recorded once.
What adds capacity, and what does not. A pool helps only where the capacity really is separate:
- deployments in several regions, or in several subscriptions, where the provider counts its quota per region, per subscription or per deployment (Azure OpenAI counts it per subscription, region and model; Bedrock per account, region and model);
- self-hosted replicas, such as several vLLM servers.
Several keys of one OpenAI project, or of one Anthropic workspace, share one limit: the provider counts limits per organization and project or workspace, not per key. A pool of such keys adds nothing; it only moves the 429 a moment later. Pool what your contracts with the provider allow.
Prefer the provider's own routing where it has one. An Azure Global deployment, or a Bedrock cross-region inference profile (a model ID such as eu.anthropic.claude-...), routes for you behind one endpoint: point a plain record at it. Pool across subscriptions, regions or replicas where there is no such routing.
printf %s "$SWEDEN_KEY" | oz0 secret set AZURE_SWEDEN_KEY --host acme-sweden.openai.azure.com
printf %s "$FRANCE_KEY" | oz0 secret set AZURE_FRANCE_KEY --host acme-france.openai.azure.com
oz0 provider pool add azure-eu sweden --preset azure-openai --base-url https://acme-sweden.openai.azure.com/openai \
--key-secret AZURE_SWEDEN_KEY --region eu-north
oz0 provider pool add azure-eu france --base-url https://acme-france.openai.azure.com/openai \
--key-secret AZURE_FRANCE_KEY --region eu-west --weight 2 --price gpt-5.5=2.75,11,0.275
oz0 provider show azure-eu
# Bedrock in two regions of one account: each member signs for its own region.
oz0 provider pool add bedrock-eu stockholm --preset bedrock --base-url https://bedrock-runtime.eu-north-1.amazonaws.com \
--region eu-north --auth-kind sigv4 --aws-service bedrock --aws-region eu-north-1 --access-key-secret AWS_ACCESS_KEY_ID
oz0 provider pool add bedrock-eu ireland --base-url https://bedrock-runtime.eu-west-1.amazonaws.com \
--region eu-west --auth-kind sigv4 --aws-service bedrock --aws-region eu-west-1 --access-key-secret AWS_ACCESS_KEY_ID
A record with a pool has no base_url of its own: its members are its endpoints. pool add on a record that calls its own base_url keeps calling it, as a member (--record-member, primary by default). pool remove of the last member does the reverse: the record calls that member's endpoint itself, with what the member set of its own (its key secret, region, data-keeping, auth and prices).
A member takes from the record what it does not set itself: the auth (whole, when the member names no kind and no header), the key secret, the region and retains_data. Each member's auth is checked as the record's is: SigV4 needs its access key secret and signing region, OAuth2 its token URL and client.
| Member field | What it is |
|---|---|
id | Its name, as calls and the cost log name it |
base_url | Its own endpoint: https, or http to the edge's own machine or on a record cluster admins own (shared, or --internal) |
auth, key_secret | Its own key or signing, when it differs from the record's |
region, retains_data | Where it processes data, and whether it keeps it |
limits | Its requests, input tokens and output tokens per minute (--rpm, --input-tpm, --output-tpm), where its provider's headers do not say them |
prices | Details |
weight | Its share among members with the same headroom; 1 when not given |
oz0 provider set warns when two members share a key secret or a host: they are often keys of one account.
How a member is chosen.
- Data rules first. A member the tenant's data rules refuse (its region, or whether it keeps data) is never chosen for that tenant, so a pool never moves data where a single record could not. When the rules refuse every member, the call gets
403 oz0_data_rule. - The same member for a job's next call. A job, or a conversation's next turn, stays on the member that served its last call while that member has headroom: its prompt cache is there. Edges do not share what they remember, so an edge that did not serve the last call takes the member the job's last recorded call names, else the member the job's or conversation's ID hashes to among the members with headroom (spread by weight): every edge picks the same one.
- Otherwise the most headroom, by weight, for a call of no job or conversation, or when no member has more than 5% of its limits left. Each edge learns a member's headroom from the rate-limit headers its answers carry (Anthropic's
anthropic-ratelimit-*, OpenAI's and Azure'sx-ratelimit-*; a record names others inrate_limitswhere its provider differs) and from its own count since the last reset, against the declared limits where there are no headers. Bedrock sends no such headers, so declare its limits.
Rate limits and server errors
A member that answers 429 is skipped until the time its answer names: retry-after-ms, retry-after, or the reset of the limit its headers say is spent (ten seconds when it names none; at least five minutes for an account whose money or quota is spent, which no retry fixes soon).
A 5xx cools it for five seconds, unless it says longer.
A member the call never reached whole (it could not be connected to, or the edge's proxy may not be sent to it) is tried no more for five seconds either, and the call goes to another member; a call its node gave up on before it went out goes nowhere and cools no member.
Authentication and missing deployments
A member that refuses the call's key (401, 403) or has no such deployment (404) is skipped while another member may take the call, and cools for five minutes once another member served it; a 403 or 404 that every member answers is the call's own (a model or a beta no key may use), so it cools none.
Calls with an unknown outcome
A 502 or 504 is not moved, since the model may have run and billed: the call fails as it would without a pool, and its outcome is unknown.
Once any output has reached the agent, a failure fails the call as it does without a pool, and the agent's own retries and fallbacks apply.
When every available member is cooling
When every member the call may go to is cooling, the call gets 429 oz0_pool_cooling with a Retry-After of the first member's return, which the runtime waits for.
While any member the call may go to is cooling after a rate limit or a server error, another member's missing key or spent account is not the call's answer, since either ends the job: the call waits for the cooling member.
A member whose account is spent, or whose key was refused, is not waited for: when no other member may take the call, the answer is the spent account's own 429, or, for members earlier calls found spent, 429 oz0_pool_spent with the code insufficient_quota, and the runtime ends the job as spent rather than retrying for minutes.
Keys the edge cannot use
A member the edge cannot sign a call for because of its own key (its key secret is not set or not bound to the member's address, its key does not sign, or its token endpoint refuses its credential) is passed over and tried only after the others for a minute; when no member is cooling and none other may take the call, the call fails as a missing key does, with 412.
A failure a moment fixes (a token endpoint or a cloud that does not answer, a key the edge cannot read now) passes the member over for that call only.
A call cut while a member's key is read (its node gave up, or a kill switch revoked the job) ends there with 503 oz0_call_ended, and goes to no other member.
Cache effects
A provider keeps its prompt cache per account or workspace, so a call that moves to another member reads none of what the first member cached; the cache report names the miss member_changed, with what it cost (the prompt cache's causes). A pool that moves often may cost more in cache misses than its 429s did.
Budgets and accounting
The gateway takes a call's reservation once, before it chooses a member, at the highest price any member the call may go to could charge, and settles it at the price of the member that served the call. Budgets, quotas and chargeback count the call once, and the cost log and the usage export name the member (member). A member that answered 429 first is not charged.
Inspect pool health
oz0 provider show and oz0 provider pool list list each member with each edge's last view of it: its headroom, a cooldown and why, and its last status (the tenant's provider admins see the edges' views). Edges learn on their own, so two may disagree for a few seconds. When every member a tenant's calls may reach is cooling at once, oz0 doctor shows a pool_cooling finding for cluster admins. The metrics count each member's events (oz0_llm_pool_events_total, by provider and member: Observability).
Member prices
Its own price per model (--price <model>=<input>,<output>[,<cache read>[,<cache write>]], dollars per million tokens), for a region priced differently. Each price it gives replaces only that price of the record's for the model, so a member dearer for input and output keeps the record's cache prices. When the record has no price of its own for the model (genai-prices prices it), a member's price must be whole: its cache-read price, and on the Anthropic Messages format its cache-write price too, or the record is refused, since cache tokens left unpriced would cost nothing