esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.22

Context Compaction

How to shrink a long chat or agent run so it fits the window and costs less, without losing the facts that matter.

13 min readFree

A chat that keeps growing

The last lesson, Memory Systems, ended with a promise: next, context compaction, shrinking a long history without losing the plot. Here it is.

Think of a notebook with a fixed number of pages. A long chat fills it up. Every new message needs room, and every old message you keep takes room away. At some point you must decide what to do with the pages you already wrote.

A product built on an AI model has the same problem. The model can only look at a limited amount of text at once. A long conversation, or an agent that has been working for an hour, will not fit. Even when it fits, a long history costs more and gives the model more to wade through.

By the end you will know four ways to shrink a history, when to trigger the shrinking, what can go wrong, and how to check that nothing important was lost. You will also run a small program that does it.

Key words

Compaction
Making a long history shorter so it fits the window and costs less, while keeping what matters.
Context window
The amount of text a model can look at in one go. The Context Window lesson in this course covers it.
Turn
One message in the history, from the user or from the model.
Summary
A short note that says the main points of a longer text.
Rolling summary
A summary you update again and again as the chat grows, so it always stands in for everything older.
Pinned fact
A detail you mark as must-keep, like an allergy or a deadline. Compaction may never lose it.
Tool result
The answer a tool sends back to the model, such as a file's contents or search results.
Threshold
A size limit. When the history grows past it, compaction starts.

Why shrink at all

There are three reasons, and the first is the obvious one.

  • It must fit

    If the history is longer than the window, something has to go. Better that you choose what goes than that it gets cut at random.

  • It costs less

    Model providers usually charge for the tokens you send. A shorter history means fewer tokens on every later message. A token is a small piece of text; the Tokenization lesson in Machine Learning Foundations goes deeper.

  • It stays focused

    A history full of old, finished side-topics can distract the model. A clean history keeps its attention on the job.

The MemGPT paper on arXiv starts from the same worry: language models are "constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis".

The third reason is not just a hunch. Anthropic's engineering post on context engineering says waiting for bigger windows is not a full answer. In its words, it is "likely that for the foreseeable future, context windows of all sizes will be subject to context pollution and information relevance concerns". Pollution here means clutter that makes the useful parts harder to use.

What compaction is

The Anthropic post gives a short definition. Compaction "is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary".

Read that slowly. Three things happen. The history gets close to the limit. You boil the old part down to a summary. You start a fresh window that holds the summary plus the latest messages.

The same post warns about the hard part. It says "the art of compaction lies in the selection of what to keep versus what to discard", because being too aggressive can lose "subtle but critical context whose importance only becomes apparent later".

Four ways to shrink a history

Most products mix these. Start with the simplest and add the next one only when you need it.

1. Drop old turns

The easiest way: keep only the newest turns and delete the rest. It is fast, free and needs no model. This is sometimes called a sliding window, because the window of kept messages slides forward as new ones arrive.

The price is plain. Whatever was said early is gone for good. If the user told you about an allergy in message two, and you dropped message two, the model has no idea. Use this alone only when old turns truly do not matter, such as a quick Q and A where each question stands by itself.

2. Summarise older turns

Instead of deleting the old turns, replace them with a short note. The note keeps the main points in far fewer words. The Wikipedia page on automatic summarization describes the idea this way: it is "the process of shortening a set of data computationally, to create a subset (a summary) that represents the most important or relevant information within the original content".

When the chat grows again, you summarise the old summary together with the turns that came after it. The summary "rolls" forward. That is a rolling summary.

In a real product, a model writes the summary. You send it the old turns and ask for a short note. That costs one extra model call each time you compact, so you do it only now and then, not on every message.

3. Keep key facts word for word

Summaries lose detail. "Booking closes on 20 March" can turn into "booking closes soon". That is no use to anyone.

So split what you keep into two piles. The first pile is pinned facts: names, dates, numbers, decisions, promises, safety details. Copy these into the history exactly as they were said. The second pile is everything else, which a summary can squash.

The Anthropic post describes the same priority. For Claude Code, it says the model "preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages". Decisions and open problems stay. Repeats go.

4. Clear old tool results

Agents call tools, and tools often send back a lot of text: a whole file, a long page of search results. The agent reads it once, acts on it, and moves on. Yet the raw text keeps sitting in the history.

The Anthropic post calls this "low-hanging superfluous content" and asks: "once a tool has been called deep in the message history, why would the agent need to see the raw result again?" It calls tool result clearing "one of the safest lightest touch forms of compaction".

The Claude docs page on context editing describes one way to do it. It says the clearing strategy "clears tool results when conversation context grows beyond your configured threshold", and that the API "automatically clears the oldest tool results in chronological order" and replaces each one "with placeholder text indicating to Claude that it was removed". The page lists settings you can change: when it starts (a trigger), how many recent tool uses to keep, and tools whose results should never be cleared.

Here is the snippet from that page, copied as it appears. It is shown without its output and is not run here, because it needs a real account and key.

Text
# From the Claude docs page "Context editing", shown without its output (not run here)
response = client.beta.messages.create(
    model="claude-opus-5-5",
    max_tokens=4096,
    messages=[{"role": "user", "content": "Search for recent developments in AI"}],
    tools=[{"type": "web_search_20250305", "name": "web_search"}],
    betas=["context-management-2025-06-27"],
    context_management={"edits": [{"type": "clear_tool_uses_20250919"}]},
)

When to start

Compaction costs something: a summary call, and a risk of losing detail. So do not run it on every message. Run it when the history crosses a size limit, the threshold.

  1. 1

    Measure

    After each turn, count the size of the history. Real products count tokens. A word count is a rough stand-in that is fine for learning.

  2. 2

    Compare

    Is the size above your threshold? A common choice is a share of the window, like 70 or 80 percent, so there is still room left for the model's answer.

  3. 3

    Compact

    If yes, keep the newest few turns as they are, and turn the older ones into a summary plus pinned facts.

  4. 4

    Replace

    Put the summary at the start of the new history, followed by the kept turns. Store the old turns somewhere safe (see the checks below).

The Claude docs page uses the same shape: its clearing strategy has a trigger, with a default of 100,000 input tokens, and a setting for how many recent tool uses to keep. Your numbers will differ. Pick them by testing on real chats from your product.

Try it: a tiny compactor

Now the program. It holds a chat of 12 turns about planning a trip. It counts words as a rough size. When the total passes a limit of 150 words (a made-up number for this demo), it replaces the oldest 8 turns with a summary and keeps the newest 4.

Important: the summariser here is a stand-in, so the code runs free with no account and no internet. It does not understand anything. Its rule is fixed: it keeps every sentence that contains a pinned word, here "allergic" or "deadline", and throws away the rest. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. It would also write fresh sentences, not just copy old ones.

Python
LIMIT = 150  # words; a made-up budget for this demo
PINNED = ["allergic", "deadline"]  # facts that must survive

history = [
    "User: Hi, I'm planning a trip to Lisbon for my sister's birthday.",
    "Bot: Lovely! When do you want to go, and who is coming?",
    "User: Four of us, in May. My sister is allergic to peanuts, so food matters.",
    "Bot: Noted. Do you prefer hotels or rented flats for the four of you?",
    "User: A flat with a kitchen, close to the old town and the tram stops.",
    "Bot: Good idea. Flats near the old town often sit on hills, so pack walking shoes.",
    "User: Booking deadline is 20 March because the group discount ends then.",
    "Bot: Understood. I will aim to have a short list of flats before that day.",
    "User: Also, what is the weather like in May around Lisbon this time of year?",
    "Bot: Usually mild and sunny, with the odd shower, so bring a light jacket.",
    "User: Great. Can you suggest a simple three day plan with a few relaxed meals?",
    "Bot: Day one old town and viewpoints, day two a river walk, day three a day trip.",
]

def words(text):
    return len(text.split())

def size(turns):
    return sum(words(t) for t in turns)

def stand_in_summariser(turns):
    # Stand-in: keeps only sentences that hold a pinned keyword. A real model would write fresh text.
    kept = []
    for turn in turns:
        for sentence in turn.split(". "):
            if any(k in sentence.lower() for k in PINNED):
                kept.append(sentence.strip().rstrip("."))
    return "Summary of earlier chat: " + ". ".join(kept) + "."

def compact(turns, limit, keep_new=4):
    if size(turns) <= limit:
        return turns
    old, new = turns[:-keep_new], turns[-keep_new:]
    return [stand_in_summariser(old)] + new

after = compact(history, LIMIT)
print("before:", len(history), "turns,", size(history), "words")
print("after: ", len(after), "turns,", size(after), "words")
print(after[0])
text = " ".join(after).lower()
for fact in PINNED:
    print(fact, "survived:", fact in text)
print("originals stored:", len(history) == 12)

Output

before: 12 turns, 172 words
after:  5 turns, 86 words
Summary of earlier chat: My sister is allergic to peanuts, so food matters. User: Booking deadline is 20 March because the group discount ends then.
allergic survived: True
deadline survived: True
originals stored: True

Read the output. The chat shrank from 12 turns and 172 words to 5 turns and 86 words. That is one summary turn plus the 4 newest turns, and roughly half the size.

Then comes the important part: the check. Both pinned facts, "allergic" and "deadline", are still in the new history. The program tests this on purpose, because a shorter history is only a win if the key details came along.

Look at how the stand-in did its job, and where it is crude. The deadline sentence still starts with "User:", because the rule copies a whole piece of text between full stops. A real summariser would write something cleaner, like "The user must book by 20 March." But it could also write "The user must book soon", and then your check would catch it.

Tip

compact() slices the list and never edits it, so the original stays whole. Copy that habit: compact a copy, never the source.

What can go wrong

  • Details vanish

    A summary drops the date, the name, the odd number. The model then answers without them and sounds sure anyway.

  • Mistakes get baked in

    If a summary says something wrong, every later turn trusts it. The original message is gone, so nobody can see the slip.

  • Compaction runs too early or too late

    Too early and you lose useful detail for no reason. Too late and the chat does not fit, or the answer has no room.

There is a fourth, quieter problem. Each time you compact, you summarise a summary. Like photocopying a photocopy, each round loses a little. This is why pinned facts matter: they are copied word for word and never get blurrier.

The Anthropic post gives practical advice for tuning. It recommends you "start by maximizing recall to ensure your compaction prompt captures every relevant piece of information from the trace, then iterate to improve precision by eliminating superfluous content". In plain words: first make sure nothing important is missed, even if the summary is long. Only then trim the extras.

Quick check

Quick check

A chat is near the limit. It holds a user's allergy and a booking date from early on. What is the safest way to compact it?

How to check it works

A compactor you have not tested is a bug waiting to happen. Three habits catch most problems.

  1. 1

    Keep the originals

    Store the full history somewhere, such as a database table, even after you compact. If a summary turns out wrong, you can read what was really said and rebuild it. Showing a user their real history also needs the originals.

  2. 2

    Test that key facts survive

    Write a small list of must-keep facts and check each one is still in the compacted history. The program above does this with a simple word match. With a real model, also ask a question that needs the fact and see if the answer is right.

  3. 3

    Compare before and after

    Run the same set of questions on the full history and on the compacted one. If the answers differ in ways that matter, your summary is losing something.

One more check

Quick check

Why store the original, uncompacted history even after compacting?

What to remember

  • Four methods

    Drop old turns, summarise them, pin key facts word for word, and clear old tool results. Start simple and add more only as needed.

  • Trigger on size

    Compact when the history passes a threshold. Keep the newest turns whole.

  • Always check

    Keep the originals and test that your pinned facts survive every time.

Next: long-context processing, working with documents bigger than the window.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.