Prompt caching

Caching works on prefixes: the stable part of a prompt, such as system instructions, tool definitions, reference documents or the earlier turns of a conversation, must come first and stay byte-identical between requests. Anything that changes, such as a timestamp near the top, breaks the cache for everything after it.

Cached input is billed at a small fraction of the normal input rate, and that fraction varies widely between providers. For chat and coding agents, which resend the whole conversation every turn, cached input can be most of the input, so compare providers on their cached-input rate, not only the headline price.

How it works

When a request arrives, the provider processes the input tokens into an internal state before generating the answer. With caching, it keeps that state for the opening section of the prompt for a short time. A later request with an identical opening section skips that work, which is faster and is billed at the cached-input rate.

Some providers cache automatically; others need you to mark the cacheable section. Either way the cache is keyed on the exact prefix, so the order of the prompt matters: stable material first, variable material last.

Example

A coding agent resends its system prompt, tool definitions and the conversation so far on every turn. In a 40-turn session, most of each turn's input is the cached prefix. Providers serving the same model can charge very different cached-input rates, so that one rate decides most of the session's bill.

In practice

You do not usually control caching yourself; your developer or the AI product does. The question to ask is simple: do the usage reports show cached input tokens, and what share of input is cached on a normal day? For a chatbot or agent that resends long instructions, a low share suggests the prompt is built in an order that breaks the cache. The usual culprit is something that changes on every request, such as the current time or a user's name, placed near the start.

Ask also whether the provider caches automatically or needs the developer to mark the cacheable part, because the setup differs between providers and a feature moved from one to another can lose its caching unnoticed. Check whether writing to the cache carries any extra charge, which some providers apply, and whether longer cache lifetimes are worth paying for given your traffic pattern. A tool used in bursts may benefit from a longer lifetime; one used constantly keeps the cache warm anyway. Revisit the numbers after any prompt change, since a small edit at the top can undo the savings.

Why it matters for your business

If you run a support chatbot, an internal assistant or an AI agent, prompt caching can decide whether the feature stays affordable as usage grows. These tools resend the same long instructions and conversation history on every turn, and caching bills that repeated part at a fraction of the normal rate. The work to benefit is small: put stable material first, keep it identical, and check the cached-token counts in the usage reports. It does little for one-off requests that never repeat. When a developer quotes running costs for an AI feature, it is reasonable to ask whether caching is set up and what the estimate assumes about it.

Common mistakes

• Putting a timestamp, user ID or other changing value at the top of the prompt, which breaks the cache every time.
• Comparing providers on headline input price when most of the traffic is cached.
• Not checking the cached-token counts the API reports, so a broken cache goes unnoticed.

Questions people ask

Does prompt caching change the answers I get?

No. Caching only skips reprocessing the identical opening section of a prompt; the model then generates its answer exactly as it would have without the cache. Answers may still vary from one request to the next for the usual reasons, such as temperature settings, but caching itself does not affect what the model says.

How long does a prompt cache last?

Usually only a few minutes after the last request that used it, though the exact duration varies by provider and some offer longer retention, sometimes at an extra charge. Busy applications keep the cache warm naturally because requests arrive constantly. Tools used occasionally may find the cache has expired between uses.

Is prompt caching a privacy risk?

Major providers scope caches to your own account or organisation, so other customers cannot read or reuse your cached prompts. Cached data is held only for the cache lifetime, which is short unless you choose extended retention. As with any AI service, check the provider's data-retention terms, especially if prompts contain personal data covered by GDPR.