AI engineering / Beginner / 01.24
Prompt Caching
How repeat calls get cheaper and faster when the start of a prompt stays the same, why order matters, and how to cache whole answers safely.
13 min readFree
Paying less for the parts that repeat
The last lesson, Long-Context Processing, ended with a promise: next, prompt caching, paying less for the parts that repeat. Here it is.
Think of a student who keeps a long handout open on the desk and only reads each new question, instead of re-reading all twenty pages every time. Many apps send the same long start of a prompt again and again: the same rules, the same tool list, the same document. Prompt caching lets the provider reuse its work on that repeated start, so repeat calls can be cheaper and faster.
By the end you will know what a prefix is, how providers match it, why the order of your prompt matters, how long a cache lasts, and how that differs from a cache you build yourself to store whole answers. You will also run two small programs.
Key words
- Prompt caching
- The provider keeps its work on the start of a prompt, so a later request with the same start can skip that work.
- Prefix
- The first part of a prompt, from the very first character up to some point. Two prompts share a prefix when they begin the same way.
- Cache
- A place that stores something so the next request for it is faster.
- Cache hit
- A request that finds what it needs in the cache.
- Cache miss
- A request that finds nothing in the cache, so the work is done from scratch.
- TTL
- Short for time to live: how long a cached entry stays before it expires.
- Response caching
- Storing whole answers in your own app, so the same question gets the stored answer without calling a model.
- Hash
- A fixed-length fingerprint made from some text. The same text always gives the same fingerprint.
What a prefix cache does
Start with the plain idea of a cache. Wikipedia describes it as "a hardware or software component that stores data so that future requests for that data can be served faster". It also names the two outcomes: a cache hit is when the requested data is found, and a cache miss is when it cannot be found.
Prompt caching is a cache for the model's work. The Anthropic docs page on prompt caching says it works by "letting the API resume from specific prefixes in your prompts". The OpenAI guide puts it like this: it preserves state "for a reusable prefix: the unchanged tokens at the beginning of a prompt". It adds that the model "still needs to process any new input to generate a new response".
One more fact from the Anthropic page: prompt caching "has no effect on output token generation". The answer you get is the same as without caching. Caching changes cost and speed, not what the model says.
How it is switched on and how it matches
Providers differ in how you turn it on. For the Anthropic Messages API, its docs page says: "The easiest way is to add `"cache_control": {"type": "ephemeral"}` at the top level of your request body". The docs also describe adding a breakpoint on individual content blocks instead. Here is that line as the docs give it.
# From the Anthropic docs, Prompt caching page, shown without its output (not run here)
"cache_control": {"type": "ephemeral"}The most important idea is that matching is strict. The Anthropic page says: "Cache hits require 100 percent identical prompt segments, including all text and images up to and including the block marked with cache control." The OpenAI guide says: "Cache reuse requires the entire rendered prefix to match."
Think of two people reading a book aloud from page one. They are in sync only until the first word that differs. After that, nothing lines up, even if the rest of the text is the same.
- 1
Your app sends a request
It has a long stable start and a changing end.
- 2
The provider looks for a saved start
It compares the beginning of your prompt with entries it has kept.
- 3
Same start found: a hit
It reuses its saved work for that part and reads only the new text.
- 4
No match: a miss
It reads everything from scratch, and can save the start for next time.
Tip
Even a tiny change early in the prompt breaks the match from that point on. One extra space, a changed date or a reordered list in the first lines can turn every request into a miss.
The design rule: stable first, changing last
This gives one simple rule for building prompts. Put the parts that stay the same at the front. Put the parts that change at the back.
The Anthropic page lists what is worth caching: "stable, reusable content such as system instructions, background information, large contexts, or frequent tool definitions". It also says to "place cached content at the prompt's beginning for best performance". And for a prompt with a static start and a varying end, "timestamps, per-request context, the incoming message", the breakpoint goes at the end of the stable part, not at the varying block.
Stable: put first
The system prompt, the tool list, long documents, examples. These are the same on every request.
Changes sometimes: put in the middle
Things like a document that updates daily. They break the match from their position onward, so keep them after the truly stable parts.
Changes every time: put last
The user's new message, the current time, anything personal to this one request.
The Anthropic page also shows how changes spread. Its cache follows an order: tools, then system, then messages. A change at one level "invalidate[s] that level and all subsequent levels". For example, it says that modifying tool definitions invalidates the entire cache. So the earlier a thing sits in the prompt, the more it costs you when it changes.
Let us see it in code. The next program builds two pairs of prompts for an imaginary cafe helper. In the first pair the question comes last. In the second pair it comes first. It then counts how many characters at the start two prompts share. Characters are a simple stand-in for tokens here, so the idea stays easy to see.
def shared_prefix(a, b):
"""Count how many characters at the start of a and b are identical."""
n = 0
for x, y in zip(a, b):
if x != y:
break
n += 1
return n
rules = "You are a helpful cafe assistant. Be kind. Never promise a refund. " * 3
menu = "Menu: latte 1, mocha 2, tea 3, cake 4, soup 5. " * 4
# Good order: stable parts first, the changing question last.
good1 = rules + menu + "Question: is the cake fresh?"
good2 = rules + menu + "Question: do you have oat milk?"
# Bad order: the changing question comes first.
bad1 = "Question: is the cake fresh? " + rules + menu
bad2 = "Question: do you have oat milk? " + rules + menu
print("prompt length:", len(good1), "characters")
print("good order shares:", shared_prefix(good1, good2))
print("bad order shares: ", shared_prefix(bad1, bad2))Output
prompt length: 417 characters good order shares: 399 bad order shares: 10
The prompt has 417 characters. With the question last, two different questions still share the first 399 characters, which is almost all of it. A provider could reuse its work on those 399. With the question first, the two prompts split after only 10 characters, just the word "Question:" and a space. Nearly all the work would have to be done again.
Tip
A common slip is putting the current date or a user name at the top of the system prompt. It looks harmless, but it changes the first line, so nothing after it can match. Move it to the end.
Caches expire
A cache does not keep things forever. The Anthropic docs say: "The cache's default minimum lifetime (TTL) is 5 minutes. This lifetime is refreshed each time the cached content is used." So if your app sends a request every minute, the entry keeps staying fresh. If nothing arrives for a while, it goes away and the next request is a miss.
The OpenAI guide describes retention too, and says the details depend on the model. For earlier models it lists an `in_memory` setting where entries "typically remain active for around 5 to 10 minutes of inactivity". Check the current page for your model before you rely on any number.
What does this mean for you? Prompt caching helps most when the same start is used often and close together. A busy chat product gets many hits. A tool used once a day may get few, because the entry is gone before the next call. Also, the provider decides what is saved and for how long, so you cannot count on a hit. Treat it as a bonus on cost and speed, never as something your app depends on to work.
Quick check
A chat app sends the same long rules, then the user's message. Where should the user's message go for caching to help?
Your own cache: storing whole answers
Prompt caching is done by the provider. There is a second kind of cache you can build yourself, in your own app. It is response caching: when the same question comes in again, you hand back the stored answer and do not call the model at all.
Prompt caching
Done by the provider. It reuses work on the start of a prompt. The model is still called, and still writes a fresh answer.
Response caching
Done by you. It stores a full answer. On a hit, the model is not called at all.
Why both can exist
They help in different places. One makes a call cheaper. The other can avoid the call completely.
Response caching is plain software, the same idea as Wikipedia's cache: store a result, and serve it when the same request comes again. You need two things. A key that stands for the question, and a time to live so old answers expire.
For the key, we use a hash. The Python docs for hashlib explain that you use "sha256() to create a SHA-256 hash object", feed it bytes with the update method, and ask for the result with hexdigest. The same text always gives the same fingerprint, so a fingerprint makes a neat key. We also clean the question first (trim spaces, lowercase) so tiny differences do not cause a miss.
Run it: a response cache with a fake clock
The program below builds a small response cache. Because waiting for real seconds would be slow, it uses a fake clock: a number we set by hand to pretend time has passed.
The function fake_model is a stand-in so the code runs free, with no keys and no network. It returns fixed text. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The costs printed at the end use a made-up price of 2 cents per call. It is not a real price.
import hashlib
class Clock:
"""A fake clock so the example runs instantly and the same way every time."""
def __init__(self):
self.now = 0
def fake_model(question):
# A stand-in. A real model is slower, costs money for every token,
# can give a different answer each run, and can fail or refuse.
return "Answer to: " + question
def make_key(question):
cleaned = question.strip().lower()
return hashlib.sha256(cleaned.encode("utf-8")).hexdigest()
class ResponseCache:
def __init__(self, clock, ttl):
self.clock = clock
self.ttl = ttl
self.store = {} # key -> (answer, time saved)
def get(self, question):
key = make_key(question)
entry = self.store.get(key)
if entry is None:
return None
answer, saved_at = entry
if self.clock.now - saved_at >= self.ttl:
del self.store[key] # too old, throw it away
return None
return answer
def put(self, question, answer):
self.store[make_key(question)] = (answer, self.clock.now)
clock = Clock()
cache = ResponseCache(clock, ttl=60) # entries live 60 seconds
model_calls = 0
requests = [
(0, "What time do you open?"),
(10, "what time do you open? "),
(20, "Do you have oat milk?"),
(30, "What time do you open?"),
(70, "What time do you open?"),
(75, "Do you have oat milk?"),
]
for when, question in requests:
clock.now = when
answer = cache.get(question)
if answer is None:
answer = fake_model(question)
cache.put(question, answer)
model_calls += 1
print(f"t={when:>2}s MISS {question.strip()}")
else:
print(f"t={when:>2}s HIT {question.strip()}")
print("model calls:", model_calls, "of", len(requests))
made_up_cost = 2 # made-up: 2 cents per model call
print("made-up cost without cache:", len(requests) * made_up_cost, "cents")
print("made-up cost with cache: ", model_calls * made_up_cost, "cents")Output
t= 0s MISS What time do you open? t=10s HIT what time do you open? t=20s MISS Do you have oat milk? t=30s HIT What time do you open? t=70s MISS What time do you open? t=75s HIT Do you have oat milk? model calls: 3 of 6 made-up cost without cache: 12 cents made-up cost with cache: 6 cents
The output walks through it. At 0 seconds the first question is a miss, and at 10 seconds a differently spaced copy is a hit. At 70 seconds the first question comes back, but it was saved at 0 seconds. That is 70 seconds old, and the limit is 60, so the entry has expired. It is thrown away and the stand-in model runs again: a miss. Then at 75 seconds the oat milk question is a hit, because it was saved at 20 seconds and is only 55 seconds old.
In total there were 3 model calls for 6 requests. With the made-up price, that is 12 cents without the cache and 6 cents with it. Half the cost, in this tiny pretend case. The real saving depends on how often questions repeat.
When your own cache is safe, and when it is not
Response caching is powerful, and it can also give wrong answers. Ask one question before using it: would everyone who sends this exact question be happy with the same answer, right now?
Usually safe
General facts that rarely change: opening hours, what a feature does, a product description. The answer is the same for everyone.
Safe only with a short TTL
Things that change slowly, like a price list or a stock level. Pick a TTL short enough that a stale answer does little harm.
Not safe
Personal or time-sensitive answers: "what is my balance?", "what is the weather now?", anything with a user's private data. Never share these across users.
The danger with personal answers is real. If the key is only the question text, then "what is my order status?" from one person would return the stored answer for another person. If you must cache a personal answer, put the user's id inside the key, so each person has their own entry. For time-sensitive answers, keep the TTL very short or do not cache them.
Quick check
Which of these is the SAFEST answer to store in your own response cache?
What to remember
Order your prompt
Stable parts first: system prompt, tools, long documents. Changing parts last: the user's message, the time, personal details.
Expect misses
Matching is exact, entries expire after a while, and the provider decides what it keeps. Count on it as a bonus only.
Cache answers yourself, carefully
Use a clean key, a TTL, and the user's id in the key for personal answers. Skip it for anything time-sensitive.
Next: context isolation, keeping each task's context to itself.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.