AI engineering / Beginner / 01.15
Context Engineering
Choosing everything a model sees on each call, not just the words of one prompt, and the small layer of code that does the choosing.
14 min readFree
Choosing what the model gets to see
The last lesson, Streaming Responses, ended with a promise: next, context engineering, choosing what the model gets to see. Here it is.
Think of a cook who has never seen your kitchen. You hand over a tray with everything they need for tonight: the recipe, a note about who is allergic to what, the ingredients, and the order from table four. The dinner will only be as good as that tray. Pack too little and the cook guesses. Pack the whole pantry and the cook wastes time and misses the one thing that mattered.
Building with a model is the same. On every call, you pack a tray. Context engineering is the job of packing it well.
By the end you will know what context engineering is, how it differs from prompt engineering, what a context layer is, why it matters, and you will run a small context layer of your own.
Key words
- Context
- Everything the model reads for one call: your instructions, the chat so far, any documents, tool results and your question. The Context Window lesson covers the size limit.
- Prompt engineering
- Writing and organising the instructions you give the model, so the answer comes out well.
- Context engineering
- Choosing and managing everything that goes into the context on each call, not just the instructions.
- Context layer
- The part of your app's code that gathers, filters, orders and trims the pieces before each call. This lesson's name for it.
- Retrieved document
- A note, page or file your code looks up and pastes into the context because it is about the question.
- Tool result
- What came back when the model, or your code, ran a tool such as a search or a database lookup.
- Memory
- Facts about the user or the task that your app saves between calls and brings back when they are needed.
- Priority
- A number you give each piece to say how much it matters. When space runs out, the lowest numbers go first.
- Budget
- The most text you allow yourself to send on one call. In this lesson we count it in words.
What context engineering is
The Anthropic engineering blog describes a change in what builders pay attention to. After years of focus on prompts, it says, "a new term has come to prominence: context engineering". The shift, in its words, is "less about finding the right words and phrases for your prompts, and more about answering the broader question of 'what configuration of context is most likely to generate our model's desired behavior?'"
That post defines context itself as "the set of tokens included when sampling from a large-language model (LLM)". A token is one small piece of text, and the Tokenization lesson in Machine Learning Foundations goes deeper. In plain words: context is everything the model reads before it answers.
The LangChain blog gives a short definition. "Context engineering is the art and science of filling the context window with just the right information at each step of an agent's trajectory." An agent here is a program that lets a model take several steps, such as calling a tool and then reading what came back. Each step needs its own tray.
Wikipedia's page on prompt engineering puts the two ideas next to each other. Context engineering, it says, "is the related area of software engineering that focuses on the management of non-prompt and prompt contexts supplied to the GenAI model, such as system instructions, metadata, API tools and tokens".
Notice the word "each" in both definitions. This is not something you do once at the start. It happens on every call.
Context vs prompt engineering
Prompt engineering is how you word the ask. Context engineering is what goes into the window and how it is put together, on every turn. The first is a sentence. The second is a system.
Anthropic's post draws the line this way. Prompt engineering "refers to methods for writing and organizing LLM instructions for optimal outcomes". Context engineering "refers to the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts". The post calls it "the natural progression of prompt engineering".
Why did the line move? The post says the early use cases were mostly one-shot jobs, like classifying a message or writing a short text, so the prompt was the biggest part. Then people began building agents that work over many turns. For those, the post says, you need strategies for managing "the entire context state (system instructions, tools, Model Context Protocol (MCP), external data, message history, etc)".
The same post makes a point about timing. In contrast to the one-off task of writing a prompt, it says, context engineering "is iterative and the curation phase happens each time we decide what to pass to the model".
Prompt engineering
You write good instructions. The question is: how should I say this? You do it when you build and when you change the wording.
Context engineering
You decide what the model sees. The question is: what should be on the tray right now? Your code does it on every call.
Together
Good wording still matters. It is one part of the tray, and the tray has other parts that wording cannot fix.
The Prompt Engineering lesson covers the wording side. This lesson is about the tray.
What goes into the context
A call to a model is not just a question. The tray usually holds several kinds of pieces.
- 1
Instructions
The standing rules: who the model is, what tone to use, what it must never do. The System Prompting lesson covers these.
- 2
Chat history
The turns so far. The model does not remember them by itself, so your code sends the ones that matter again.
- 3
Retrieved documents
Notes or pages your code looked up because they relate to the question.
- 4
Tool results
What came back from a search, a database or any other tool the model used.
- 5
Memory
Saved facts about the user, such as a name or an allergy, brought back when useful.
The LangChain post groups the same ideas into three kinds of context: instructions (it lists "prompts, memories, few-shot examples, tool descriptions"), knowledge ("facts, memories") and tools ("feedback from tool calls"). The names differ a little from our list. The point is the same: the tray has many parts, and your code decides each one.
Tip
Ask of every piece: if I removed this, would the answer get worse? If not, it is probably not worth its space.
Why it matters
A model knows only two things: what it learned in training, and what is on the tray. If your user's name, price list or yesterday's chat is not on the tray, the model cannot use it, and it may guess. That is the cost of too little.
Too much has costs too. The LangChain post says long-running agents use many tokens, which "can exceed the size of the context window, balloon cost / latency, or degrade agent performance". Latency means the wait before the answer comes back.
There is a quieter problem. The Anthropic post says that as the number of tokens in the window goes up, "the model's ability to accurately recall information from that context decreases", an effect it calls context rot. It concludes that context "must be treated as a finite resource with diminishing marginal returns".
A research paper points the same way. "Lost in the Middle" tested multi-document question answering and key-value retrieval. Its abstract reports that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models".
So where does that leave you? The Anthropic post gives the goal in one line: good context engineering means "finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome". Small, and useful.
Quick check
A user asks about their allergy, but your code never put it in the call. What will a model most likely do?
What is a context layer?
If the tray changes on every call, someone has to pack it. In a real app, that someone is code. We will call it the context layer: the part of your app that sits between your data and the model, and gathers, filters, orders and trims the pieces before each call.
A small honesty note. "Context layer" is the name this lesson uses for that job. It is not a fixed product or a standard library. The sources describe the work, such as writing, selecting, compressing and isolating context, which is how the LangChain post groups its strategies. How you split your code is up to you.
- 1
Gather
Collect every piece that might help: rules, profile, recent turns, retrieved notes, tool results.
- 2
Filter
Throw out what is not about this question. A note about the cafe's history will not help someone ordering a cake.
- 3
Order
Decide where each piece goes. The paper above suggests the start and the end of a long input are where models use information best.
- 4
Trim
If the total is over your budget, drop or shorten pieces, least important first.
- 5
Send
Join what is left into one prompt and make the call.
Trimming also has a bigger cousin. The Anthropic post describes compaction as "taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary". We will keep to dropping pieces here, which is simpler, and the Context Window lesson showed how a trimmer can lose facts.
Build a tiny context layer
Let's build one in plain Python. It has eight pieces: system rules, a user profile, the last three chat turns, two retrieved notes and one tool result. Each piece gets a priority. A higher number means it stays longer.
To assemble, the code adds up the words. While the total is over the budget, it removes the piece with the lowest priority. In words, the rule is that the words of everything kept must not go over the budget.
One honest shortcut: we count words, where a real layer counts tokens with the provider's own counter, because tokens are not the same as words. The shape of the code stays the same.
def words(text):
return len(text.split())
def gather():
# Each piece: (name, priority, text). A higher priority number is kept longer.
pieces = [
("rules", 10, "You are a friendly cafe helper. Never promise a refund."),
("profile", 6, "User: Asha. Allergic to nuts. Likes oat milk."),
("turn 1", 4, "user: Do you have oat milk?"),
("turn 2", 5, "assistant: Yes, oat milk costs the same as dairy."),
("turn 3", 9, "user: Can I order a cake for Friday?"),
("note A", 8, "Note: Cakes need two days notice and are made nut free on request."),
("note B", 2, "Note: The cafe was opened in 2009 by two friends who loved baking."),
("tool", 7, "Tool result: Friday has 3 cake slots left."),
]
return pieces
def assemble(pieces, budget):
kept = list(pieces)
dropped = []
while sum(words(p[2]) for p in kept) > budget:
lowest = min(kept, key=lambda p: p[1])
kept.remove(lowest)
dropped.append(lowest)
return kept, dropped
def fake_model(prompt):
# Stand-in: fixed text, so the code runs free.
return "(stand-in answer, " + str(words(prompt)) + " words read)"
for budget in (60, 40):
kept, dropped = assemble(gather(), budget)
prompt = "\n".join(p[2] for p in kept)
print("Budget", budget, "-> used", words(prompt))
for name, priority, text in kept:
print(" keep", name, "(priority " + str(priority) + ",", words(text), "words)")
for name, priority, text in dropped:
print(" drop", name, "(priority " + str(priority) + ",", words(text), "words)")
print(" ", fake_model(prompt))Output
Budget 60 -> used 56 keep rules (priority 10, 10 words) keep profile (priority 6, 8 words) keep turn 2 (priority 5, 9 words) keep turn 3 (priority 9, 8 words) keep note A (priority 8, 13 words) keep tool (priority 7, 8 words) drop note B (priority 2, 13 words) drop turn 1 (priority 4, 6 words) (stand-in answer, 56 words read) Budget 40 -> used 39 keep rules (priority 10, 10 words) keep turn 3 (priority 9, 8 words) keep note A (priority 8, 13 words) keep tool (priority 7, 8 words) drop note B (priority 2, 13 words) drop turn 1 (priority 4, 6 words) drop turn 2 (priority 5, 9 words) drop profile (priority 6, 8 words) (stand-in answer, 39 words read)
Important: `fake_model` is a stand-in so the code runs free, with no key and no internet. It is not a real model, and it only reports how many words it was given. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse.
Now read the output. The eight pieces add up to 75 words. With a budget of 60, the code removed "note B" (13 words, priority 2) and then "turn 1" (6 words, priority 4), leaving 56. With a budget of 40, it also removed "turn 2" and "profile", leaving 39.
Look at who survived: the rules, the newest user turn, the cake note and the tool result about cake slots. Those are exactly what the question "Can I order a cake for Friday?" needs. The cafe's history was dropped first, which is right.
Now look at what was lost. At 40 words, the profile with Asha's nut allergy is gone, and a cake answer might need it. The priorities were our guess. A real app would pin something like an allergy higher, or move it into the rules, which have priority 10. Trimming always costs something, so choose what to lose on purpose.
Tip
Print what your layer kept and dropped, as this program does. When an answer looks wrong, the first question is what was on the tray. Often you will see the answer in that list.
Habits of a good context layer
A few habits come from the sources and from the program above.
Keep it small and useful
Anthropic's advice is to keep context "informative, yet tight". Add a piece only when it earns its place.
Decide on purpose
Give every piece a priority, and know what drops first. A hidden rule will surprise you when the budget changes.
Look at what was sent
Log the final prompt, or the list of kept and dropped pieces. You cannot fix a tray you have not seen.
The Anthropic post also describes two ways to fill the tray. One is to look things up before the call, which is what our notes were. The other, which it calls "just in time", keeps light references such as file paths and loads data when needed using tools. It notes the trade-off: "runtime exploration is slower than retrieving pre-computed data". Many real systems mix the two.
For teams building agents, the same post gives modest advice: "do the simplest thing that works".
Quick check
What is a context layer?
Recap
Context engineering is choosing what the model sees on every call. Prompt engineering is the wording of the ask, and it is one piece of the tray. The context layer is the code that gathers, filters, orders and trims, so the tray is small, useful and checked.
Quick check
Which sentence best separates the two ideas?
Next: input format, laying out what you send so the model can find it.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.