Prompt Caching Across Turns, and Its Sharp Edges

Every turn in an agent conversation resends the same system prompt and the same tool definitions. For a well-specified agent that is a few thousand tokens of identical prefix, paid for on every message, forever.

Prompt caching removes that. The mechanics are simple and the distributed-systems edges are where the actual work is.

The key is the stable part

cacheKeyFor({ model, system, tools })

A SHA-256 over the model, the system prompt, and the tool definitions, with each tool reduced to its name, description, and parameter schema.

What is not in the key is the conversation. That is the entire idea: the key covers the part that does not change between turns, so every turn of every conversation using the same agent configuration hits the same cache entry.

Reducing tools to those three fields is deliberate. A tool object at runtime carries a handler function and whatever closure it was built with. Hashing the whole thing would produce a different key on every process start, which is a cache that never hits and is worse than no cache because it also pays to write.

Extending, not recreating

Entries live for 600 seconds. The refresh rule:

TTL_SECONDS     = 600
EXTEND_BELOW_MS = 300_000

If more than half the lifetime remains, use the entry. If less, extend it before use.

Extending at the halfway mark rather than at expiry is doing real work. An entry that expires between the check and the request produces a failure on a live turn, and the window for that grows as you approach the deadline. Refreshing with five minutes of headroom means the race effectively cannot happen.

If extension fails, the stored entry is cleared rather than retried. A cache that cannot be extended is a cache that no longer exists, and treating it as present produces a confusing error on the next call instead of an obvious miss.

One writer

if (!(await store.claimCache(key))) return null;

Creating a cache entry costs a provider call. With several service instances, a cold start means every instance discovers the miss simultaneously and every one of them tries to create the same entry.

So creation is behind a claim. One instance wins and creates. The others get null and proceed uncached for that turn.

Proceeding uncached is the right losing behaviour. The alternative is waiting for the winner, which converts a cost optimisation into a latency dependency, on a turn a customer is waiting through. A cache should never be able to make a request slower than not having it.

Failure has a cooldown

RETRY_AFTER_MS = 10 * 60 * 1000

When caching fails for a configuration, that configuration is marked unusable for ten minutes and every request in that window skips the cache path entirely.

Without this, a prompt that cannot be cached, because it is under the provider's minimum token count, or contains something unsupported, generates a failed create attempt on every single turn. That is a per-turn latency penalty and a per-turn cost, in exchange for nothing, repeated indefinitely.

Ten minutes of not trying is much cheaper than being optimistic on a schedule.

Implicit beats explicit when it applies

if (mode === 'explicit' && cache.implicit && !hasFiles) {
  return { mode: 'implicit', minTokens: 0 };
}

Some providers cache prefixes automatically with no lifecycle to manage. When implicit caching is available and there are no files involved, the policy uses it and skips the explicit path.

The file exclusion is the interesting condition. Files change what gets sent in ways prefix caching does not handle well, so an explicit entry is worth its complexity there. Everywhere else, a cache with no state to manage has no claim lock, no extension race, no cooldown, and no stale entry. Free is better than correct-and-managed.

What it is worth

For a support agent with a substantial system prompt and a dozen tools, the cached prefix is often the majority of the input on a short turn. Cached input is billed at a fraction of the normal rate and, more usefully here, is processed faster.

That second part is the reason to do this on voice. Prompt caching is sold as a cost feature and it shows up as a latency feature, in time to first token, which is the number that decides whether a phone conversation feels alive.

The rule this all follows

Every edge above resolves the same way: the cache is allowed to save money and is never allowed to cost time.

Lost the claim, proceed uncached. Extension failed, proceed uncached. Caching broken for this config, stop trying and proceed uncached. There is no path where a turn waits on the cache.

Get that rule wrong and you build something that is cheaper on average and occasionally makes a customer wait two seconds for an optimisation they did not ask for.