AI engineering / Beginner / 01.05
Context Window
Everything a model can see in one request, what fills it up, what to do when it is full, and why more is not always better.
13 min readFree
Everything the model can see at once
The last lesson, Tokens, ended with a promise: next, the context window, everything the model can see at once. Here it is.
Picture a desk. You can only work with what is on the desk right now. A big desk fits many papers. A small desk fits a few. If you want to put a new paper down and the desk is full, something has to come off.
A model works the same way. Each time you send a request, it reads one pile of text and answers from that pile. The size of the pile it can handle is called the context window.
By the end you will know what goes into that pile, what the model does and does not know, what to do when the pile is too big, why long piles can still go wrong, and why size costs money. You will also run two small programs.
Key words
- Context
- All the text the model is given for one request: instructions, the chat so far, any documents, and your question.
- Context window
- The most text, counted in tokens, that a model can handle in one request.
- Token
- One small piece of text, like a word or part of a word. The Tokens lesson covers this, and the Tokenization lesson in Machine Learning Foundations goes deeper.
- System prompt
- Instructions you write once for the whole chat, such as "You are a friendly cafe helper." The user does not usually see it.
- Turn
- One message in a chat, from either the user or the model.
- Token budget
- The number of tokens you allow yourself to send. You set it so the request fits the window.
- Truncate
- To cut text off so it fits, for example by dropping the oldest messages.
- Summarise
- To replace a long piece of text with a short version that keeps the main points.
What a context window is
Here is how the Anthropic docs page on context windows defines it. The window refers to "all the text a language model can reference when generating a response, including the response itself". The page also calls it a "working memory" for the model.
Two parts of that sentence matter. First, the word "all": the window holds everything the model reads, not only your latest message. Second, "including the response itself": the answer the model writes also takes up room in the same window. So input and output share one limit.
Wikipedia's page on large language models points to the window from another side. To find which tokens matter to each other, it says, the attention mechanism works "within the scope of the context window". In plain words, the model can only connect pieces of text that are both inside the window.
What goes into the context
When you build a chat product, it feels like the model "remembers" the conversation. Look closer and you will see it does not. Your code sends the whole conversation again with every new message. The model reads it all from the top, then answers.
So the context is a stack of parts, and your job as an AI engineer is to decide what goes in it.
The system prompt
Your standing instructions: who the model is, what tone to use, what it must never do. It goes in every single request, so its tokens are paid for every time.
The chat so far
Every earlier message from the user and the model. This part grows with each turn and is usually what fills the window first.
Documents and the question
Any file, page or database result you paste in, plus the new question at the end. This is the part that changes most from request to request.
This stack is also what people mean when they say "context" on its own, as in "give the model more context". Context is whatever you place in front of the model. Choosing it with care is a skill, and a later lesson, Context Engineering, goes much deeper.
The model only knows what is in front of it
This is the most useful idea in the whole lesson, so read it twice. The model knows two things: what it learned in training, and what is in the context. Nothing else.
It does not know your name unless your name is in the context. It does not know your company's price list unless you paste it in. It does not know what happened yesterday, because yesterday's chat is not in today's request unless you put it there.
Let's test that. The program below has a tiny stand-in for a model. It looks through the messages it is given and, if someone said "Hi, my name is ...", it answers with that name. Otherwise it says it does not know.
def fake_model(messages):
# Stand-in: it only "knows" what is inside the messages it is given.
for role, text in messages:
if role == "user" and text.startswith("Hi, my name is "):
return "Your name is " + text[len("Hi, my name is "):].rstrip(".") + "."
return "Sorry, I do not know."
chat = [("user", "Hi, my name is Asha."), ("assistant", "Hello Asha!"), ("user", "What is my name again?")]
print("Full chat: ", fake_model(chat))
print("Newest only: ", fake_model(chat[-1:]))Output
Full chat: Your name is Asha. Newest only: Sorry, I do not know.
Important: `fake_model` is a stand-in so the code runs free, with no key and no internet. It is not a real model. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The stand-in always answers the same way, which is only handy for learning.
Now the lesson in the output. With the full chat, it found the name. With only the newest question, it had nothing to go on and said "Sorry, I do not know." A real model behaves the same way in spirit: take a message out of the context and the model cannot use it.
What happens when the window is full
Sooner or later a chat gets long enough to fill the window. What then? Per the Anthropic page, if the input alone already exceeds the model's context window, the API returns a 400 error, "invalid_request_error", with the message "prompt is too long".
That is one vendor's behaviour, and other services may act differently. The safe lesson is general: you should plan for a full window before your users hit it. The same page suggests counting tokens before you send, to stay within the limit.
Remember the window is shared by input and output. The same page says that on newer models, if generation reaches the window's limit, it stops with a special stop reason, "model_context_window_exceeded". The answer can be cut off in the middle because the output ran out of room.
You have three main moves when things do not fit.
- 1
Drop old turns
Throw away the oldest messages first and keep the newest ones. This is simple and free, but the model forgets anything that was only said early on.
- 2
Summarise
Replace a long stretch of old chat with a short summary. The Anthropic page describes compaction this way: it "automatically summarizes earlier parts of the conversation on the server, so the conversation can continue past the context window limit". You can also do this yourself in your own code.
- 3
Keep only what is relevant
Do not paste a whole 50-page document when the question is about one paragraph. Pick the parts that matter and send only those.
The page also notes that chat apps such as claude.ai "can also manage the context window on a rolling 'first in, first out' basis". First in, first out means the oldest message leaves first, like a queue at a shop. That is move one, done automatically.
Tip
No move is free. Dropping loses facts. Summaries can lose details, and the model that writes the summary also costs tokens. Pick the move that loses the least for your product.
A chat-history trimmer you can run
Let's build move one, dropping old turns, in plain Python. The rules are simple.
- 1
Always keep the system prompt
Your standing instructions must never be trimmed away.
- 2
Always keep the newest message
That is the question the model has to answer right now.
- 3
Add older turns, newest first
Walk backwards through the chat. Add each turn while there is room. Stop at the first one that does not fit.
One honest shortcut: to keep this simple, we estimate tokens as words. One word counts as one token. Real tokens are not the same as words (the Tokens lesson explained why), so a real trimmer uses the provider's own token counter. The shape of the code stays the same.
In words, the rule is the following. The system prompt, plus the newest message, plus the older turns we keep, must not go over the budget.
def count(text):
return len(text.split())
system = "You are a friendly cafe helper."
turns = [
("user", "Hi, my name is Asha."),
("assistant", "Hello Asha, how can I help?"),
("user", "Do you have oat milk?"),
("assistant", "Yes, oat milk costs the same as dairy."),
("user", "I am allergic to nuts."),
("assistant", "Noted. Our oat milk is nut free."),
("user", "What is my name again?"),
]
def trim(system, turns, budget):
used = count(system)
newest = turns[-1]
used += count(newest[1])
kept = [newest]
for turn in reversed(turns[:-1]):
cost = count(turn[1])
if used + cost > budget:
break
kept.insert(0, turn)
used += cost
dropped = turns[: len(turns) - len(kept)]
return kept, dropped, used
for budget in (30, 16):
kept, dropped, used = trim(system, turns, budget)
print("Budget", budget, "-> used", used)
print(" system:", system)
for role, text in kept:
print(" keep", role + ":", text)
for role, text in dropped:
print(" drop", role + ":", text)Output
Budget 30 -> used 23 system: You are a friendly cafe helper. keep user: I am allergic to nuts. keep assistant: Noted. Our oat milk is nut free. keep user: What is my name again? drop user: Hi, my name is Asha. drop assistant: Hello Asha, how can I help? drop user: Do you have oat milk? drop assistant: Yes, oat milk costs the same as dairy. Budget 16 -> used 11 system: You are a friendly cafe helper. keep user: What is my name again? drop user: Hi, my name is Asha. drop assistant: Hello Asha, how can I help? drop user: Do you have oat milk? drop assistant: Yes, oat milk costs the same as dairy. drop user: I am allergic to nuts. drop assistant: Noted. Our oat milk is nut free.
The sums: the system prompt is 6 words and the question is 5. With a budget of 30, the trimmer adds the 7-word and 5-word turns before it, reaching 23. The next turn is 8 words, and 31 is over 30, so it stops. With a budget of 16, a 7-word turn would make 18, so only the question stays.
Look at what both runs lost. In each one, the turn that said "Hi, my name is Asha." was dropped, and the final question asks "What is my name again?" A real model sent this trimmed chat could not answer, exactly like the stand-in in the last section. Trimming is not free. This is why real products often keep a short summary of the early chat, or pin key facts such as the user's name in the system prompt.
A big window is not a perfect memory
You might think: if the window is huge, just put everything in and be done. It is not that simple. Fitting in the window and being used well are different things.
A research paper called "Lost in the Middle: How Language Models Use Long Contexts" tested this on two tasks that require finding the right information in the input: multi-document question answering and key-value retrieval.
The abstract reports that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models".
The Anthropic page makes a related point. It says a larger window lets a model handle more lengthy prompts, but "more context isn't automatically better". It adds: "As token count grows, accuracy and recall degrade", and names the effect "context rot".
Tip
Put the facts that matter most near the start or the end of your request, not buried in the middle. And send only what the model needs. Short, focused context is a good default.
Quick check
Where did the Lost in the Middle paper find models used information best?
Bigger windows cost more
Every token you send is a token you may pay for. The docs page says that thinking tokens, which are the extra tokens a model writes while it works something out, are "billed as output tokens". Input and output are both counted, and each has its own price at most providers.
A chat makes this add up fast. Because the whole history goes in again with each message, a long chat sends the same old messages over and over. Ten turns means the first message has been sent ten times.
Size can change the price in other ways too. The same Anthropic page notes that for one model, its Haiku 5.5, prompts over 100,000 tokens cost more, while for the other 1M-token models long requests are billed at standard pricing. So a window being big does not by itself mean the price jumps. It depends on the provider and the model, and the rules change, so read the current price page before you build.
Here is a made-up sum, with a made-up price, only to show the shape. Say a provider charged 1 coin per 1,000 input tokens. A chat that sends 2,000 tokens on turn one and 4,000 on turn two sends 6,000 in total, which would cost 6 coins. That is not a real price, and your numbers will differ.
How big are windows?
When we fetched the Anthropic page, it listed many of that company's models with a 1M-token window and others, including one marked deprecated, with a 200k-token window. It also said a single request can generate up to 128k output tokens on the 1M models. Treat those as a snapshot, not a rule.
The habit to build is to look up the limit for the exact model and version you use, in that provider's own docs, and to keep it in your settings instead of hard-coding a guess.
Quick check
A user's chat fills the window. Which move lets the conversation continue and keeps a short record of the early part?
Recap and what to try
Quick check
A model is asked about a fact that was said early in a long chat, but your code trimmed that turn. What happens?
Next: sampling parameters, the dials that make answers safe or surprising.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.