Configure caches and server tools
For: tenant administrators and cost owners.
Account for provider cache controls, server tools and charges beyond text tokens.
Cost
The gateway reads usage from every response and stream, per format, and prices it: Anthropic's message_start and message_delta events, the last chunk of a chat completion streamed with stream_options.include_usage (which the gateway adds to a streamed chat completion that does not ask for it), and the Responses API's response.completed.
A message_delta that reports more input than message_start, as a call with server tools such as web search does, counts its input.
An answer whose usage the gateway cannot read, such as a successful answer that carries none, a JSON answer or a stream's line over 8 MB, is an unknown outcome, as a stream that breaks off is: charged at least its reservation.
Tokens read from the cache are counted apart from new input tokens, so they are priced as cache reads.
A record's own price wins; otherwise the provider's reported cost when its price source says so, then genai-prices. Each call is recorded with its job, agent, node and model. See Cost and usage.
The prompt cache
Providers keep the start of a prompt they saw recently and charge less for reading it again: Anthropic's Messages API (and the platforms that speak it) caches only what a request marks, for five minutes or, at a higher write price, for an hour; OpenAI's APIs cache every long enough prefix by themselves. What each provider does is part of its record (cache), with defaults for each wire format, so a record for another platform says its own facts.
The facts are picked by the wire format of each call. The Messages format defaults to marks and the Responses format to the automatic cache. The Chat format, which many providers speak and few cache, defaults to no cache: the shared openai record and the Azure OpenAI presets say their Chat calls cache, and a record for another provider that caches Chat calls says so itself (--cache '{"style":"automatic"}'). Claude on Bedrock, Vertex and AI Foundry gets plain marks, with no longer lifetime and no re-warm, until those platforms' handling of them has been checked live.
With the tenant's setting auto, the default, the gateway manages it:
- It marks the stable start of each Messages request that carries no marks of its own and is long enough to cache: the system prompt (which covers the tool list) and the history up to the platform's status block, which stays the request's last part. A request with its own marks, such as a harness's, goes as it was sent.
- It asks for the longer lifetime when the request offers a function that waits for a person's approval (its plugin's
requires_approval, or a tool policy rule that asks for every call of it) and a wait longer than the short lifetime is likely enough to pay the higher write price.
A job's first call writes everything, so it rarely pays there; later calls write only what is new, so it usually does for agents that wait for people.
How likely a long wait is comes from the gaps the edge measured after this agent's earlier calls of that kind, and from the median time people took to decide the agent's approvals of those functions over the last 30 days (a median under the short lifetime asks for no longer one); neither is shown, and the median counts only when at least as many people decided as works-council mode's threshold, so it never stands for one person.
Approvals that a rule asks for only with some arguments, or that a job's tainted data asks for, are not known before the call, so their waits get the re-warm only.
- It warms the cache again after a wait. When a person decides an approval and the job's prefix has lapsed, the edge re-sends the job's last request once as the smallest request the provider allows (
max_tokens: 0on the Messages API), metered on the job, under its budget, while the approved tool runs, so the next model call reads the cache.
It counts against the job's budget but not against its tree's model calls or tokens, so it never takes a call the agent would otherwise have had. This moves the cache write off the job's next step; it does not make the write free.
The edge holds each job's last request in memory only for this, never written down: at most a day, dropped sooner when the job ends or its tenant is deleted, and at most 64 MiB per edge. An edge that did not serve the job's last call sends nothing.
With off, calls go to the provider as agents wrote them:
oz0 tenant cache default off
oz0 tenant cache default auto
Turning it off loosens no control, so it takes no reason; admins of the tenant may change it, and each change is audited (tenant.prompt_cache). Either way, every call's cache figures are recorded: see Cost and usage.
A record's cache facts, as JSON:
# a platform that takes no cache marks
oz0 provider set myproxy --cache '{"style":"none"}'
# a Chat-format provider that caches by itself
oz0 provider set myllm --cache '{"style":"automatic"}'
# facts for the record's Responses calls rather than its first format's
oz0 provider set azure-openai --cache '{"style":"automatic","formats":["openai-chat","openai-responses"]}'
# another minimum, and no re-warms
oz0 provider set anthropic --shared --cache '{"min_tokens":2048,"rewarm_field":"none"}'
# the longer lifetime's own price, per million tokens
oz0 provider model anthropic claude-sonnet-5-5 --cache-write-long 6
| Field | What it says |
|---|---|
style | breakpoints (caches what a request marks), automatic (caches by itself) or none |
formats | the record's wire formats these facts are for; empty is its first format, and its other formats take their defaults |
short_ttl, long_ttl | the lifetimes as the provider names them, such as 5m and 1h; a short mark names none, and long_ttl none asks for no longer lifetime |
short_seconds, long_seconds | how long they are |
short_write, long_write, read | the cache's prices as multiples of the input price, where the price list has none |
min_tokens | the smallest prefix the provider caches; shorter requests are not marked |
long_param | for an automatic provider, the fields that ask for its longer retention, such as {"prompt_cache_retention": "24h"} (model-specific, so not set by default) |
rewarm_field, rewarm_tokens | the smallest answer a re-warm asks for; none turns re-warms off |
The longer lifetime's billing and the re-warm have been measured against a stand-in provider, not yet against a live one; set rewarm_field to none for a platform whose API refuses a request that asks for no output.
The money the cache saved and the cost of each miss are priced at the call's own size: a model whose list price steps up above a prompt length is priced at the step the call is in.
Server tools
The gateway passes a provider's calls through as they are, so the provider's own server tools work for the agents and harnesses that use them: code execution at once, web search and web fetch once the tenant turned them on. Server tools work only with that provider's models, and the data they fetch leaves your environment; tools from plugins work with every model.
What a tool costs is counted from what the answer reports, apart from the model's tokens: usage.server_tool_use on the Messages API, tool_usage and the tool calls in output on the Responses API, the grounding queries on Google's.
The reservation of a call counts them first, at the tool's price and the limit the call sets, with what each use adds to the input the model reads next.
Under reserve: strict a server tool needs a limit its format reads, max_uses on the Messages format or max_tool_calls on the Responses format, and a tool nothing prices is refused.
Web fetch has no fee, so what it costs is what it adds to the input: under strict it needs max_content_tokens too, and holds at least a full model window for each use, because that field limits text and a fetched PDF comes in past it.
A provider's reported cost, on a record priced that way, is its whole word on a call: no tool's fee goes on top of it.
What a call may cost besides tokens
The prices are data, shipped for each wire format and each preset, and a record replaces them one by one:
| What | Where it comes from | Shipped for |
|---|---|---|
| A tool's fee | tools.<name>.usd for each use; result_tokens is what one use may add to the input, result_limit the tool's own field that replaces that figure, and binary_tokens what one use may add whatever that field says, which a strict budget holds; input_mtok and output_mtok for a tool that reports tokens of its own; limit is the field its format reads as its limit, one of the tool's or max_tool_calls; reports_as is the name answers report it by, where that is another tool's | Web search and code execution on the Messages format; web search, file search, containers and image generation on the Responses format; search and Maps grounding on Google's; a router's web plugin and a live-search source on their presets |
| A price tier | tiers.<field>.<word> multiplies a call's tokens by the word in service_tier, speed or inference_geo. auto is what a call that names none can land on | Each vendor's own words: OpenAI's on records that follow its prices, Anthropic's fast mode and US inference on records that follow its, Google's tiers, Bedrock's and Groq's. A record that speaks one of their formats and follows another's prices has only auto and the format's ordinary tier |
| Audio and images | models.<model> with audio_input_mtok, audio_output_mtok and image_output_mtok, over a record's price for the model; the price list's otherwise | Whatever the price list knows |
| Cache writes | Each lifetime's rate: the model's cache_write_mtok and cache_write_long_mtok, or short_write and long_write of the record's cache facts | Five minutes and an hour on the Messages format, 30 minutes and 12 hours on the Responses and Chat formats |
| Inline content | media_tokens.<type> is what one item of a media type, or of the word before its slash, adds to the input past its bytes, with 0 for what its bytes bound. A type the table does not name has no bound, so a strict budget refuses it | Text, JSON, XML and audio by their bytes; an image at 10,000 tokens, past what providers read from one |
A call is reserved at the dearest tier it can land on: one that names default at the standard price, one that names no tier on OpenAI's records at the fast price, since a project can make that its default, and one on Anthropic's at the US region's, as a workspace's default can be.
A word that lowers a price, such as flex or batch, lowers no reservation: a provider may ignore it, and the call then runs as one that names none.
It is charged at the tier the answer names. An answer that names no tier may be held to a dearer tier the call asked for by name, never to a cheaper one, so a discount counts only when the answer states it.
One that names a word its record has no price for pays what the call held for that field, and the edge logs the word, so that the record can price it.
A model that has a rate of its own for images or audio is held at that rate unless the call restricts its answer to text, as such a model may answer in images with nothing asked. Predicted output holds its bytes as output, since what a prediction gets wrong bills as output. A list of other models a router may fall back on, in models or fallbacks, holds the dearest of them, at the price assumed for a name nothing prices; under reserve: strict, route and more than 16 models are refused.
A record's own facts, as JSON, have the last word, over web_search_usd too. A record that corrects one fact of a tool keeps the rest of that tool's, its prices among them, a price stated as 0 makes the tool free, and a change that states no facts keeps those it has:
oz0 provider set azure-openai --charges '{"tools":{"web_search":{"usd":0.02}},"tiers":{"service_tier":{"priority":1.8}}}'