esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.11

Chunking

How to cut long documents into pieces a search can find: fixed size, paragraphs, headings, and the labels each piece should carry.

14 min readFree

Why cut documents at all

The last lesson, Classify, recommend, spot odd ones, ended with a promise: next, chunking, cutting documents into pieces worth retrieving. Here it is.

Picture a 40-page handbook. A customer asks, "How long do refunds take?" The answer is one sentence on page 12. Hand the search the whole handbook as one block and it looks like a mix of everything, so it rarely looks like a good match for anything.

So we cut the handbook into small pieces first. Each piece is called a chunk. Search then compares the question with each chunk, and the chunk with the refund sentence has a fair chance to win. Cutting is the first step of the build in the RAG lesson, before embedding and storing.

Cutting sounds simple, but it is where many first search systems quietly go wrong. You will learn the main ways to cut, what to keep with each piece, how to pick a size, and the mistakes to avoid, and you will run a small Python program.

Key words

Chunk
One piece of a longer text, cut so that it can be stored and searched on its own.
Chunking
The cutting step: breaking large text into smaller chunks before you store it.
Chunk size
How big each chunk is, counted in words, characters or tokens.
Overlap
A few words repeated at the end of one chunk and the start of the next, so a sentence cut in half still appears whole somewhere.
Metadata
Labels stored next to a chunk, such as its title, its heading, its page and where it came from. The chunk is the content, metadata is the label on the box.
Heading path
The chain of headings above a piece of text, for example Refunds > Timing.

Tokens, the pieces models count text in, are covered in the Tokenization lesson in Machine Learning Foundations.

The size trade-off

Why not make chunks as big as possible, or as small as possible? Because each choice breaks something.

There is a hard limit on one side. An embedding model, the kind that turns text into a list of numbers for its meaning (the Embeddings lesson explains it), can only take in so much text at once. The Pinecone guide on chunking notes that a common chunk size is "the max context window size of the embedding model". A chunk larger than that cannot be embedded in one piece.

There is a softer limit on the other side. The Pinecone guide says that when a full paragraph or document is embedded, bigger inputs "may introduce noise or dilute the significance of individual sentences or phrases, making finding precise matches when querying the index more difficult". Think of mixing one spoonful of lemon juice into a bucket of water. The taste is still in there, but you cannot find it.

Tiny chunks fail the other way. The same guide says that if chunks are too small or too large, search results can be imprecise or miss relevant content. A chunk that is only half a sentence says almost nothing by itself. The Wikipedia page on retrieval-augmented generation (RAG) sums up the pull in one line: the size "involves a trade-off between focused retrieval and context".

  • Small chunks

    Match a question closely and are cheap to paste into a prompt. But a small piece can lose the context that makes it make sense.

  • Big chunks

    Keep the context. But they can bury the one useful sentence, and they use more of the prompt budget.

  • The goal

    A chunk big enough to be understood on its own, small enough to stay about one thing.

Tip

A test from the Pinecone guide: if a chunk makes sense to a human without the text around it, it will make sense to the language model too. If you would ask "what is this about?", the chunk is too small or cut in the wrong place.

Ways to cut

There is no single right way. Here are the main ones, from simplest to smartest.

Fixed size with overlap

Pick a size, say 40 words, and cut every 40 words, whatever the text says. This is the simplest method, and the Pinecone guide recommends starting here and moving on only if it turns out not to be enough.

Its weakness is that it is blind. The Wikipedia page says fixed-length methods "may interrupt sentences or separate pronouns from their antecedents". An antecedent is the thing a pronoun points back to. If one chunk ends with "the manager" and the next starts with "She said no", the second chunk no longer says who "she" is.

Overlap softens this. Each chunk starts a little before the previous one ended. The LangChain page puts it this way: "Overlapping chunks helps to mitigate loss of information when context is divided between chunks." With 40-word chunks and a 10-word overlap, the window moves forward 30 words each time, so every cut sentence appears whole in at least one chunk.

By sentence or paragraph

Instead of counting, cut where the writer already cut: at the end of a sentence or a paragraph. The Pinecone guide lists a naive way (split at full stops and new lines) and library tools that recognise sentences better. A blank line between paragraphs is an easy cut to find.

Paragraphs are natural, because a writer usually puts one idea in one paragraph. The catch is that sizes vary. One paragraph might be three words, the next three hundred. And a heading is also a "paragraph" of its own, which, as you will see, causes trouble.

By document structure

Many documents carry a map: headings, lists, tables, code blocks. The Pinecone guide says that for formats such as Markdown, recognising the syntax "(e.g., headings, lists, and code blocks)" lets you divide the content by its structure. Markdown is a plain-text way of writing headings with # signs.

The Wikipedia page gives an example from reports: a method "starts new chunks at titles and preserves tables as complete units". So a title begins a new chunk, and a table is never cut in the middle. This is the method our code will use, and it brings us to the most useful trick in this lesson: keeping the heading path.

Recursive splitting

Recursive splitting is a middle road for plain text with no clear headings. It has a list of separators in order of strength: first a blank line, then a new line, then a space, then anything. The LangChain page says the splitter "tries to split on them in order until the chunks are small enough". Its default list is blank line, new line, space, empty.

In plain words: try to cut at paragraphs. If a paragraph is still too big, cut it at line breaks. Still too big? Cut at spaces. Only as a last resort does it cut inside a word. The page explains the aim: this "has the effect of trying to keep all paragraphs (and then sentences, and then words) together as long as possible".

Libraries already ship this splitter. Pinecone calls the LangChain version a middle ground between always cutting on one character and a smarter splitter, and now you know how it thinks.

Semantic chunking

The smartest, and most costly, idea is to cut where the topic changes. Pinecone describes it as an experimental technique: break the text into sentences, group each sentence with its neighbours, embed the groups, and compare each group with the one before. "By comparing the semantic distance between each group and its predecessor, you can identify where the topic or theme shifts, which defines the chunk boundaries."

It needs embeddings for many small groups, so it takes more work than the other methods. Start simple, and try this only if your own tests show the simple ways cutting topics in half.

Cut one article three ways

Time to run it. Below is a made-up help article, cut three ways: fixed 40 words with a 10-word overlap, by paragraph, and by heading. The third way remembers the heading path and stores it as metadata. This is plain Python with no model and no network, so it runs free.

  1. 1

    Fixed

    Split the article into words and take 40 at a time, moving 30 words forward each step.

  2. 2

    Paragraph

    Split on blank lines. Every block becomes a chunk, headings included.

  3. 3

    Heading

    Walk the blocks. A heading updates the path; a body block becomes a chunk labelled with the path.

Python
ARTICLE = """# Refunds

Our shop offers refunds on most items. This page explains who can ask and how it works.

## Who can ask

Any customer can ask for a refund on an unused item. Sale items and gift cards cannot be refunded. Items that were used, washed or damaged after delivery are also not covered, so please check each item before you wear it.

## Timing

You have 30 days from delivery to ask. We check each returned item and email you when it passes. Once we receive the item, the money returns to your card in 5 to 7 working days.

# Shipping

We ship from one warehouse.

## Costs

Shipping is free above a set order size. Smaller orders pay a flat fee."""


def fixed_chunks(text, size=40, overlap=10):
    words = text.split()
    step = size - overlap
    out = []
    for start in range(0, len(words), step):
        out.append({"text": " ".join(words[start:start + size]), "meta": {"start_word": start}})
        if start + size >= len(words):
            break
    return out


def paragraph_chunks(text):
    return [{"text": p.strip(), "meta": {}} for p in text.split("\n\n") if p.strip()]


def heading_chunks(text):
    path = {}
    chunks = []
    body = []

    def flush():
        if body:
            trail = " > ".join(path[k] for k in sorted(path))
            chunks.append({"text": " ".join(body), "meta": {"heading_path": trail}})
            body.clear()

    for block in text.split("\n\n"):
        block = block.strip()
        if block.startswith("#"):
            flush()
            level = len(block) - len(block.lstrip("#"))
            for k in [k for k in path if k >= level]:
                del path[k]
            path[level] = block.lstrip("# ").strip()
        elif block:
            body.append(block)
            flush()
    return chunks


for name, fn in [("fixed 40 words, overlap 10", fixed_chunks), ("by paragraph", paragraph_chunks), ("by heading", heading_chunks)]:
    chunks = fn(ARTICLE)
    print(name, "->", len(chunks), "chunks")
    for c in chunks[:2]:
        print("  ", c["meta"], "|", c["text"])
    print()

fixed = [c for c in fixed_chunks(ARTICLE) if "5 to 7 working days" in c["text"]][0]
head = [c for c in heading_chunks(ARTICLE) if "5 to 7 working days" in c["text"]][0]
print("fixed chunk with the timing answer:", fixed["text"])
print("it mentions Timing:", "Timing" in fixed["text"])
print("heading chunk metadata:", head["meta"])

Output

fixed 40 words, overlap 10 -> 4 chunks
   {'start_word': 0} | # Refunds Our shop offers refunds on most items. This page explains who can ask and how it works. ## Who can ask Any customer can ask for a refund on an unused item. Sale items and gift cards cannot
   {'start_word': 30} | on an unused item. Sale items and gift cards cannot be refunded. Items that were used, washed or damaged after delivery are also not covered, so please check each item before you wear it. ## Timing You have 30 days

by paragraph -> 10 chunks
   {} | # Refunds
   {} | Our shop offers refunds on most items. This page explains who can ask and how it works.

by heading -> 5 chunks
   {'heading_path': 'Refunds'} | Our shop offers refunds on most items. This page explains who can ask and how it works.
   {'heading_path': 'Refunds > Who can ask'} | Any customer can ask for a refund on an unused item. Sale items and gift cards cannot be refunded. Items that were used, washed or damaged after delivery are also not covered, so please check each item before you wear it.

fixed chunk with the timing answer: the money returns to your card in 5 to 7 working days. # Shipping We ship from one warehouse. ## Costs Shipping is free above a set order size. Smaller orders pay a flat fee.
it mentions Timing: False
heading chunk metadata: {'heading_path': 'Refunds > Timing'}

Read the counts first. The same article became 4, 10 or 5 chunks, depending on the method.

Look at the paragraph method. Its first chunk is just "# Refunds", a heading with nothing under it. That is a useless piece to search: it matches the word "Refunds" and tells you nothing. Splitting by paragraph treats a heading as a paragraph, and it ended up as 10 chunks, nearly half of them one-line headings.

Now the important part, the last three lines. The fixed chunk that holds the answer, "the money returns to your card in 5 to 7 working days", does not say it is about timing. The heading "## Timing" sat in the chunk before it. Worse, the same chunk carries unrelated shipping text, so its meaning is muddled. A question like "how long do refunds take?" has less to grab onto.

The heading-aware chunk holds the same sentence, but its metadata says Refunds > Timing. You can store that label next to the chunk, or paste it in front of the text before embedding, so the words "Refunds" and "Timing" help the search.

Tip

The code cuts by counting words, not tokens, to stay simple. Real embedding models count tokens, and the Pinecone guide reminds you that different embedding models may tokenize text differently. Count the way your model does.

Keep metadata with each chunk

A chunk alone is a torn-out page. Always save a few labels next to it.

  • Where it came from

    The document title and a source link or file name. Without it you cannot show the reader where an answer comes from, and you cannot cite it.

  • Where it sat

    The heading path, and the page number for a PDF. This is the context the cut removed.

  • Where it fits in the list

    A chunk number, or the start position. It lets you fetch the chunk before or after when a piece is too short.

These labels also help later. The RAG and dynamic filters lesson showed how labels like year or access level let you filter chunks before ranking. You can only filter on labels you saved when you cut.

  1. 1

    Cut

    Split the document with the method you chose.

  2. 2

    Label

    Attach title, heading path, page and source to every chunk.

  3. 3

    Check

    Print ten random chunks with their labels and read them as a stranger would.

Tune on your own questions

Which size is right? Nobody can tell you from the outside. It depends on your documents, your questions and your embedding model. The Pinecone guide suggests picking a range of sizes, for example smaller chunks of 128 or 256 tokens for fine detail and larger ones of 512 or 1024 tokens for more context, then running real queries against each and comparing quality. It calls this an iterative process.

Make that concrete. Write down 20 real questions your users ask, and for each one note which chunk should come back. Then, for each chunk setting you try, count how often the right chunk lands in the top few results.

  1. 1

    Collect questions

    20 real questions, each with the passage that answers it.

  2. 2

    Try two or three settings

    For example 100 words, 200 words, and by heading.

  3. 3

    Score each

    How often does the right chunk appear in the top 3?

  4. 4

    Keep the winner

    Change one thing at a time, so you know what helped.

Quick check

A chunk holds the sentence "It takes 5 to 7 working days." but the heading "Refunds > Timing" was lost. What goes wrong?

Common mistakes

  • Splitting tables and code

    Cut a table in the middle and its rows no longer match their column names. Cut code in the middle of a function and neither half works. Keep tables and code blocks whole, even if the chunk runs long.

  • Losing the heading

    The heading often holds the topic. If the cut drops it, the body text loses its subject. Keep the heading path as metadata.

  • Chunks of one line

    Splitting by paragraph can leave lone headings or one-word chunks. Merge them into the text that follows.

Quick check

You must chunk a long Markdown guide full of headings and tables. Which first try makes the most sense?

Recap

  • Why we cut

    Search matches small pieces better than whole documents, and embedding models take only so much text at once.

  • How we cut

    Fixed size with overlap is the simple start. Paragraphs, headings and recursive splitting respect the text. Semantic chunking cuts where the topic changes.

  • What we keep

    Title, heading path, page and source go with every chunk. Then you test sizes on your own questions.

Next: indexing embeddings, building the store your search runs on.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.