esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.12

Indexing Embeddings

How to build the store your search runs on: embed your documents in batches, keep them fresh, and choose how the vectors are laid out.

14 min readFree

From chunks to a searchable store

The last lesson, Chunking, ended with a promise: next, indexing embeddings, building the store your search runs on. Here it is.

Think of a library. The books are your documents. Before anyone can find a book, a librarian has to read each one, write a catalogue card, and put the card in the right drawer. Nobody does this at the moment you ask a question. It is done ahead of time, once, and then kept up to date.

Indexing is that work for your AI app. You turn each chunk of text into a list of numbers (an embedding, which the Embeddings lesson explains) and save it in a place that search can read fast. By the end you will know the steps of the pipeline, how to avoid redoing work when a document changes, and what the two main kinds of index are. You will also run a small incremental indexer.

Key words

Index
The saved collection of vectors, plus the extra structure that lets you search it quickly.
Indexing
The ahead-of-time work of turning documents into vectors and saving them.
Stable id
A name for each chunk that stays the same between runs, so you can find and replace it later.
Metadata
Facts about a chunk that are not its text, like which document it came from or when it was saved.
Content hash
A short fingerprint made from some text. The same text always gives the same fingerprint. Change one letter and the fingerprint changes.
Batch
A group of chunks sent to the embedding model in one request.
Re-indexing
Redoing the indexing work for documents that changed.

The pipeline, step by step

Indexing is a line of small jobs. Each job hands its result to the next one, like a conveyor belt in a bakery.

  1. 1

    Load

    Read your documents from wherever they live: files, a database, a help centre. Keep a name for each one.

  2. 2

    Clean

    Remove the junk that carries no meaning: menus, page footers, repeated headers, odd spacing. Bad text in means bad matches out.

  3. 3

    Chunk

    Cut each document into pieces of a size that works for search. The Chunking lesson covered how.

  4. 4

    Embed in batches

    Send chunks to the embedding model a group at a time, not one by one.

  5. 5

    Store

    Save, for every chunk: the vector, the original text, its metadata and a stable id.

Why save the text next to the vector? Because a vector is only numbers. When search finds a close match, you still have to show the reader, or the model, the words. Without the text you would have to look it up again somewhere else.

Why a stable id? Imagine you number chunks 1, 2, 3 in the order you meet them. Add one document in the middle and every number after it shifts. Your saved ids now point at the wrong text. A better id is built from things that do not move, such as the document name plus the position of the chunk inside it, like refunds#0.

Tip

Save the original text and the metadata with the vector, always. It costs a little space and saves you from many painful lookups later.

Keeping the index fresh

Your documents will change. A price goes up, a policy is rewritten, a page is deleted. If your index never hears about it, your app will answer from old facts. This is one of the most common reasons a search feature slowly gets worse.

The lazy fix is to throw everything away and embed it all again every night. That works for five documents. For five hundred thousand it is slow, and an embedding model usually charges for every piece of text you send it. So we want to redo only what changed.

A content hash makes that easy. Python's built-in hashlib module can make one. Its docs describe creating a hash object with sha256(), feeding it bytes with update(), and asking for the result with digest() or hexdigest(). Here is the idea in a few lines.

Python
import hashlib

old = "Refunds are paid within 14 days."
new = "Refunds are paid within 7 days."

print(hashlib.sha256(old.encode("utf-8")).hexdigest()[:12])
print(hashlib.sha256(old.encode("utf-8")).hexdigest()[:12])
print(hashlib.sha256(new.encode("utf-8")).hexdigest()[:12])

Output

84c7af99b243
84c7af99b243
54271fa4a37b

The same sentence gave the same fingerprint twice. Changing 14 to 7 gave a completely different one. That is the whole trick. Save each document's fingerprint after you index it. Next time, compute it again. If it matches, skip the document. If not, re-index it.

There are three cases to handle, and the order matters.

  • New document

    No fingerprint saved yet. Chunk it, embed it, store it.

  • Changed document

    The fingerprint differs. Delete all its old chunks first, then add the new ones. Skipping the delete leaves old facts behind.

  • Removed document

    It is in the index but no longer in your source. Delete its chunks. Forgetting this is how an app keeps quoting a page you deleted months ago.

Tip

Delete a changed document's old chunks before adding the new ones. A document that shrank from three chunks to two would otherwise leave a stale third chunk behind.

Record which model made each vector

There is one more thing to save with every vector: the name and version of the embedding model that made it. Vectors from two different models are not comparable. They live in different spaces, like measuring one room in metres and another in feet and then comparing the numbers.

So when you switch to a new embedding model, every old vector is out of date, even if no document changed. Searching with a vector from the new model against stored ones from the old model gives nonsense. With the model version saved on each row, you can find the old ones and re-embed them, or refuse to mix the two by mistake.

Build an incremental indexer

Here is a small indexer that does all of the above. It has five documents. Two have two lines each, so they become two chunks. Chunking here just means one line is one chunk, to keep the code short.

The embedder is a stand-in. It turns the text's hash into four numbers. It is NOT a real embedding model. Equal text gives equal numbers, but similar meaning gives nothing useful, and it lets the code run free with no key and no network. A real embedding model is slower, costs money for every token you send, and can fail or refuse a request. Its numbers are also much longer and carry real meaning. The pipeline around it, which is what this lesson is about, would look the same.

Read the sync function in three parts: remove what is gone, find what is new or changed, then embed in batches of two and store.

Python
import hashlib

MODEL = "stand-in-v1"  # which embedding model made each vector


def fake_embed(text):
    # Stand-in so the code runs free. It is NOT a real embedding model:
    # it turns the text's hash into 4 numbers, so equal text gives equal numbers
    # but similar meaning gives nothing useful.
    digest = hashlib.sha256(text.encode("utf-8")).digest()
    return [round(b / 255, 2) for b in digest[:4]]


def embed_batch(texts):
    print(f"  embedding a batch of {len(texts)}")
    return [fake_embed(t) for t in texts]


def content_hash(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:8]


def chunk(text):
    return [line for line in text.split("\n") if line.strip()]


def sync(docs, store, seen, batch_size=2):
    """Bring the store in line with docs. store: chunk id -> row. seen: doc id -> hash."""
    embedded = removed = 0
    # 1. delete documents that are gone
    for doc_id in [d for d in seen if d not in docs]:
        for cid in [c for c in store if c.startswith(doc_id + "#")]:
            del store[cid]
            removed += 1
        del seen[doc_id]
        print(f"  removed {doc_id}")
    # 2. find new or changed documents
    todo = []
    for doc_id, text in docs.items():
        h = content_hash(text)
        if seen.get(doc_id) == h:
            continue
        for cid in [c for c in store if c.startswith(doc_id + "#")]:
            del store[cid]
            removed += 1
        for n, piece in enumerate(chunk(text)):
            todo.append((f"{doc_id}#{n}", piece, doc_id))
        seen[doc_id] = h
        print(f"  changed or new: {doc_id}")
    # 3. embed in batches, then store vector + text + metadata + stable id
    for i in range(0, len(todo), batch_size):
        part = todo[i:i + batch_size]
        vectors = embed_batch([t for _, t, _ in part])
        for (cid, text, doc_id), vec in zip(part, vectors):
            store[cid] = {"vector": vec, "text": text, "doc": doc_id, "model": MODEL}
            embedded += 1
    print(f"  embedded {embedded}, removed {removed}, store now holds {len(store)}")


docs = {
    "refunds": "Refunds are paid within 14 days.\nKeep your receipt.",
    "shipping": "Orders ship in 2 days.",
    "privacy": "We never sell your data.",
    "support": "Chat is open 9 to 5.\nEmail is answered daily.",
    "careers": "We are hiring designers.",
}
store, seen = {}, {}

print("Run 1: first build")
sync(docs, store, seen)

docs["refunds"] = "Refunds are paid within 7 days.\nKeep your receipt."
del docs["careers"]

print("Run 2: one edited, one deleted")
sync(docs, store, seen)

print("Run 3: nothing changed")
sync(docs, store, seen)

print(sorted(store))
print(store["refunds#0"])
print({row["model"] for row in store.values()})

Output

Run 1: first build
  changed or new: refunds
  changed or new: shipping
  changed or new: privacy
  changed or new: support
  changed or new: careers
  embedding a batch of 2
  embedding a batch of 2
  embedding a batch of 2
  embedding a batch of 1
  embedded 7, removed 0, store now holds 7
Run 2: one edited, one deleted
  removed careers
  changed or new: refunds
  embedding a batch of 2
  embedded 2, removed 3, store now holds 6
Run 3: nothing changed
  embedded 0, removed 0, store now holds 6
['privacy#0', 'refunds#0', 'refunds#1', 'shipping#0', 'support#0', 'support#1']
{'vector': [0.33, 0.15, 0.12, 0.64], 'text': 'Refunds are paid within 7 days.', 'doc': 'refunds', 'model': 'stand-in-v1'}
{'stand-in-v1'}

Run 1 saw five new documents and made seven chunks. They went out in four batches: three of two and one of one. Run 2 did far less. The careers document was removed, which deleted one row. The refunds document changed, so its two old rows were deleted too (three removals in all) and its two lines were embedded again in one batch. The other three documents were never touched. Run 3 found nothing to do and made no calls at all.

Look closely at run 2. The second line of the refunds document, Keep your receipt, did not change, but it was embedded again, because our fingerprint is per document. If you want to avoid even that, store a fingerprint per chunk instead. That is a small extra step you can add when documents are big and edits are small.

The last lines show the stored row for refunds#0: the vector, the text, the document it came from, and the model version. This is the shape of what you save.

Quick check

A document's text is edited, so its content hash changes. What should the indexer do with the chunks it saved earlier?

Batches, limits and retries

Why batch at all? Every request to an embedding service has a fixed cost in waiting: the trip across the network, the queue, the answer coming back. Sending one chunk per request pays that cost again and again. Sending a group pays it once. Services also cap how many requests you can make in a minute, and how big one request may be. These limits are called rate limits. Their exact numbers differ from service to service and change over time, so read them from the docs of the one you use.

Things will go wrong in a long indexing job. A request times out. A service tells you to slow down. If one failed batch crashes the whole run, you lose the hours of work before it. So wrap each batch in a retry: try again after a short wait, and wait longer after each failure. If it still fails after a few tries, stop with a clear error and record which batch failed.

Below, a stand-in that fails twice and then works, and a retry loop around it. The failures are made up to show the loop. A real service can fail in more ways, and a retry only helps for errors that are temporary.

Python
import time

calls = {"n": 0}


def flaky_embed(batch):
    # Stand-in that fails on its first two calls, like a busy service would.
    calls["n"] += 1
    if calls["n"] <= 2:
        raise RuntimeError("rate limit: slow down")
    return [[len(t)] for t in batch]


def embed_with_retry(batch, tries=4):
    wait = 0.01  # a real wait would be seconds; kept tiny so this runs fast
    for attempt in range(1, tries + 1):
        try:
            return flaky_embed(batch)
        except RuntimeError as err:
            print(f"  attempt {attempt} failed: {err}")
            if attempt == tries:
                raise
            time.sleep(wait)
            wait *= 2


print(embed_with_retry(["hello", "hi there"]))

Output

  attempt 1 failed: rate limit: slow down
  attempt 2 failed: rate limit: slow down
[[5], [8]]

Tip

Because the indexer from the last section only changes what it has to, a run that crashes halfway can simply be run again. Only the documents whose hash was never saved will be redone.

Index types: exact or approximate

So far we saved vectors in a plain dictionary. Real stores add structure so that search stays fast when you have millions of vectors. There are two big families.

A flat index keeps every vector in a simple list and compares your question against all of them. It is exact: it always finds the true closest ones. The FAISS wiki, a vector search library's docs, lists its flat indexes as exact search and describes them as brute-force. Building is almost free, because there is nothing to build. The cost comes at query time, which grows with the number of vectors.

An approximate index builds extra structure ahead of time so that a query only looks at a small part of the data. It may miss the true closest vector now and then. In return it is much faster on large collections. Here are the two well-known kinds, next to flat:

  • HNSW (a graph)

    Vectors are linked to near neighbours in layers. The arXiv paper that introduced it says it incrementally builds a multi-layer structure of proximity graphs, and that starting the search from the upper layer allows logarithmic complexity scaling.

  • IVF (partitions)

    The vector space is cut into cells. The FAISS wiki says a query only compares against the vectors in a few chosen cells (the nprobe setting), so only a fraction of the data is compared. The failure case is when the cell holding the true nearest neighbour is not chosen.

  • Flat (nothing extra)

    No structure and no misses. Fine for a few thousand vectors, slow for many millions.

The trade is between build time, query time and accuracy. An approximate index spends effort when you build it so queries are cheap. Wikipedia notes for HNSW that a larger search candidate list can improve accuracy but makes queries slower, and a larger construction candidate list can improve the quality of the graph but makes index building slower. It also says that because HNSW is approximate, its results are not always identical to a full exact search.

Tip

Start with a flat index. If your search is fast enough and you have under a few tens of thousands of chunks, you may never need anything fancier. Reach for an approximate index when query time becomes a real problem.

Quick check

What is the main thing you give up with an approximate index?

What to remember

  • Do the work ahead of time

    Load, clean, chunk, embed in batches, store. Search then only reads.

  • Save four things per chunk

    The vector, the text, the metadata and a stable id. Add the model version too.

  • Redo only what changed

    Content hashes tell you which documents are new, changed or gone. Delete before you re-add.

Next: choosing a vector store, where your vectors live.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.