esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.23

Long-Context Processing

What to do with a long document: how to place it when it fits, and how to split, combine or search it when it does not.

13 min readFree

A document that is bigger than you hoped

The last lesson, Context Compaction, ended with a promise: next, long-context processing, working with documents bigger than the window. Here it is.

Think of a student with a 300-page textbook and one question. She does not read every page out loud to a friend. She opens the book at the right place, reads a little, and writes a short note. Models face the same problem. Sometimes the whole book fits on the desk. Sometimes it does not.

This lesson has two halves. First: the document fits, but it is long. How do you place it so the model does well? Second: it does not fit. How do you cut it up and still get one good answer? You will also run a small program that does the cutting.

Key words

Long context
A request that holds a lot of text, such as a whole report or many files, close to the size of the context window.
Context window
The most text a model can handle in one request. The Context Window lesson covers it.
Chunk
One piece of a document you cut up, for example ten sentences.
Overlap
A few sentences that appear at the end of one chunk and again at the start of the next, so an idea is not cut in half.
Map
Do the same small job on every chunk, one chunk at a time.
Reduce
Combine all the small answers into one.
Retrieval
Searching a large pile of text and pulling out only the pieces that look relevant to the question.

When the document fits

Say the document is long but it still fits in the window. You can send it whole. That is the simplest path, so start there. But how you arrange the request matters.

The Anthropic docs, in their long context prompting advice, speak about documents of 20k or more tokens. A token is a small piece of text, a bit like a word. The Tokens lesson explains it, and the Tokenization lesson in Machine Learning Foundations goes deeper. For inputs that large, the page gives a few tips.

  1. 1

    Put the document first

    The docs say to place long documents near the top of the prompt, above the question, the instructions and the examples.

  2. 2

    Ask the question last

    The same page says that putting the query at the end can improve response quality by up to 30 percent in tests, especially with complex inputs made of several documents. Think of it as reading the book first, then hearing the question.

  3. 3

    Label each document

    With several documents, the docs suggest wrapping each one in its own tags, with the source name next to it, so the model can tell them apart.

  4. 4

    Ask for quotes first

    The docs also suggest asking the model to quote the relevant parts of the documents before it does the task. They say this helps it focus on the relevant content and ignore the rest.

Tip

The quotes step has a bonus. You can check the quotes against the document yourself. An answer with a quote next to it is easier to trust than a bare answer.

Your request might read like this: the document, then "First, copy the sentences that mention refunds. Then answer the question using only those sentences." The document comes first, and the question comes last.

Models can miss the middle

Even when everything fits, a long input is not read evenly. A research paper called "Lost in the Middle" tested models on two jobs: answering questions from many documents, and looking up a value from a list of key-value pairs. The abstract says performance "can degrade significantly" when the position of the relevant information changes.

More exactly, it reports that performance is often highest when the relevant information is at the beginning or the end of the input, and gets worse when the model must use information in the middle. This held even for models built for long inputs.

Picture a long bus ride. You remember the first stop and the last stop. The twenty stops in between blur together. So do not assume that "it fits" means "it will be read well". If one fact in the middle is the whole point, test for it, or help the model find it, for example by asking for quotes first.

When it does not fit

Now the harder case. The document is bigger than the window, or so close that there is no room left for instructions and the answer. Remember that the answer shares the window with the input. Sending more is not an option, so you split the document into chunks.

There are three common ways to get one answer out of many chunks. They differ in what each chunk is asked and what is passed along.

  • Map-reduce

    Ask the same question of each chunk on its own (map), then combine the small answers with one final request (reduce). The chunks do not depend on each other.

  • Refine

    Go through the chunks in order and carry a running answer. Each step gets the answer so far plus the next chunk, and improves the answer.

  • Retrieve

    Do not read every chunk. Search first, and send only the few chunks that look relevant to the question. Retrieval gets its own lessons later.

Map-reduce: answer each piece, then combine

The name comes from a well-known way to process big data. Wikipedia describes a MapReduce program as made of a map procedure, which does filtering and sorting, and a reduce method, which does a summary operation. Its example: sort students into queues by first name (map), then count each queue (reduce). Wikipedia adds that the model is a specialization of the split-apply-combine strategy.

Split-apply-combine is how you would share out a stack of exam papers. Cut the stack into piles, give each helper one pile, then add up the results. For a long document: split it into chunks, ask each chunk the question, then combine the small answers.

The good side: each chunk is small, and the calls do not depend on each other, so they can run side by side. The weak side: a fact that needs two far-apart chunks may be lost, because no single call sees both.

Refine: carry an answer along

Refine works like writing a summary while you read. After chapter one you have a note. After chapter two you fix the note. The LlamaIndex docs describe their refine mode this way: it creates and refines an answer by going through the chunks one after another, with a separate model call for each chunk. The first chunk makes a first answer. Then that answer, the next chunk and the original question go into another request, and so on until every chunk is read.

The good side: the answer can build on earlier chunks. The weak side: it cannot run side by side, because each step waits for the one before. So it is slower than map-reduce when the number of chunks is big.

The same LlamaIndex page also describes a mode it calls tree summarize. It answers groups of chunks, then treats those answers as new chunks and repeats, until only one answer is left. It is map-reduce, applied again and again.

Retrieve: read only what matters

A third idea is to skip most of the document. If your question is about refunds, you do not need the chapter on opening hours. Retrieval searches the chunks and picks the few that look relevant. Those few go into one request.

This saves the most cost and time, but it adds a new way to fail: if the search misses the right chunk, the model never sees it. How the search works is a big topic. The RAG lessons later in this course cover it, along with the details of how to chunk well.

Try it: split, map, reduce

Let us build map-reduce in plain Python. The document is made up: 30 sentences, four of which mention a refund. We cut it into chunks of ten sentences with an overlap of two. Then each chunk goes to a model that pulls out the sentences about refunds, and a final step joins the results.

A warning first. The function fake_model is a stand-in, not a real model. It only copies sentences that contain the keyword, so that this code runs free, with no key and no network. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. Treat this program as a picture of the flow, not of how a model thinks.

Python
# A made-up handbook with 30 sentences. Four of them mention "refund".
filler = [
    "The cafe opens at eight.",
    "Coffee beans arrive on Monday.",
    "Staff wear a green apron.",
    "The oven needs ten minutes to warm up.",
    "Milk is stored on the left shelf.",
    "Tables are wiped every hour.",
    "The playlist is chosen by the morning shift.",
    "Lost items go in the blue box.",
]
sentences = [filler[i % len(filler)] for i in range(30)]
sentences[3] = "A refund is possible within 14 days."
sentences[8] = "Refund requests need the original receipt."
sentences[14] = "The manager approves every refund over 20 dollars."
sentences[24] = "Refund money returns to the same card."

def make_chunks(items, size, overlap):
    step = size - overlap
    chunks = []
    for start in range(0, len(items), step):
        chunks.append(items[start:start + size])
        if start + size >= len(items):
            break
    return chunks

def fake_model(messages):
    # Stand-in: not a real model. It copies the sentences that
    # contain the keyword, so this code runs free and gives the same result every time.
    keyword, text = messages[0]["content"], messages[1]["content"]
    return [s for s in text.split("\n") if keyword in s.lower()]

chunks = make_chunks(sentences, size=10, overlap=2)
print("chunks:", len(chunks))

# Map step: one call per chunk.
partial = []
for i, chunk in enumerate(chunks):
    messages = [{"content": "refund"}, {"content": "\n".join(chunk)}]
    hits = fake_model(messages)
    print("chunk", i, "hits:", len(hits))
    partial.append(hits)

# Reduce step: join the partial answers, drop repeats, keep the order.
joined = [s for hits in partial for s in hits]
final = list(dict.fromkeys(joined))
print("before dedupe:", len(joined), "after:", len(final))
for s in final:
    print("-", s)

Output

chunks: 4
chunk 0 hits: 2
chunk 1 hits: 2
chunk 2 hits: 1
chunk 3 hits: 1
before dedupe: 6 after: 4
- A refund is possible within 14 days.
- Refund requests need the original receipt.
- The manager approves every refund over 20 dollars.
- Refund money returns to the same card.

Read the output. Four chunks were made. The overlap is why the first two chunks both found the sentence "Refund requests need the original receipt." and why the last two both found "Refund money returns to the same card." Before the reduce step there were 6 hits. After removing repeats there were 4, which matches the four refund sentences in the document.

That is the whole job of the reduce step here: join, then drop duplicates. With a real model, the reduce step is often one more request that asks the model to merge the small answers into one clear one.

Tip

Overlap is cheap insurance. Without it, a sentence that sits right on the cut could lose the sentence before it, and with it the meaning. The cost is that some text is read twice.

How many chunks do you get? Let N be the number of sentences, s the chunk size and o the overlap. Each new chunk moves forward by s minus o sentences. The formula below says: take the sentences left after the first overlap, divide by that step, and round up.

chunks=⌈N−os−o⌉\text{chunks} = \left\lceil \frac{N - o}{s - o} \right\rceil

Here N is 30, s is 10 and o is 2. That gives 28 divided by 8, which is 3.5, rounded up to 4. It matches the program.

Cost and time grow with length

Every token you send is read by the model, and you pay for each one. Longer input means a bigger bill and a longer wait. Chunking does not make that go away. It can make it a little worse, because overlap means some text is sent twice, and map-reduce adds one more call at the end.

The next program counts calls and tokens for a long document. Every number in it is made up, including the price. It only shows the shape of the sum. Real prices and limits change, so look up the current ones before you plan a real product.

Python
# All numbers are made up, to show the shape of the cost.
doc_tokens = 30000      # the document
window = 8000           # room per request, so chunks must be smaller
chunk = 6000            # leave room for instructions and the answer
overlap = 500
price_per_million = 2.0  # made-up dollars per million tokens

step = chunk - overlap
chunks = -(-(doc_tokens - overlap) // step)   # round up
sent = doc_tokens + (chunks - 1) * overlap    # overlap is sent twice

print("chunks:", chunks)
print("tokens sent in the map step:", sent)
print("model calls, map-reduce:", chunks + 1)
print("model calls, refine:", chunks)
print("made-up cost of the map step: $%.3f" % (sent / 1_000_000 * price_per_million))

Output

chunks: 6
tokens sent in the map step: 32500
model calls, map-reduce: 7
model calls, refine: 6
made-up cost of the map step: $0.065

The document of 30,000 tokens became 6 chunks. Because of the overlap, 32,500 tokens were sent, 2,500 more than the document itself. Map-reduce made 7 calls, and refine made 6, one after another. Retrieval would send only a few chunks, so it would send far fewer tokens.

So the question to ask is not only "how do I fit this?" but also "how much of it does the model really need to read?". Often the answer is less than all of it.

Which one should you pick?

Start with the plainest option and move down only when you must.

  1. 1

    It fits: send it whole

    Put the document first, the question last, and ask for quotes first. Test with a fact from the middle.

  2. 2

    It does not fit, and the answer is spread across it: map-reduce or refine

    Map-reduce when the chunks are independent and you want speed. Refine when each part builds on the one before.

  3. 3

    It does not fit, and the answer is in one spot: retrieve

    Search first, then read only the chunks that matter. It is the cheapest, but you must check that the search finds the right piece.

Quick check

A 400-page manual is too big for the window. You want the one paragraph about warranty returns. Which approach fits best?

Quick check

Your whole report fits in the window. Where should the question go?

Remember the limits of what you just read. The numbers from the docs and the paper come from their own tests, so check your own task with your own data. And the program you ran used a stand-in, so real models will behave differently, with more variety and more ways to fail.

Next: prompt caching, paying less for the parts that repeat.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.