esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.17

Build RAG by hand

Put every piece you have learned into one small plain program: chunk, embed, retrieve, prompt, generate and check, with no framework.

14 min readFree

Every piece, in one program

The last lesson, Generation, ended with a promise: next, build RAG by hand, every piece in plain code. This is that lesson.

You have met each part one at a time. You cut documents into chunks, turned text into numbers, searched those numbers, wrote a prompt and asked a model. Today they all go into one program of about sixty lines. No framework, no library, nothing to install.

By the end you will have read every function, run the program on three questions, and seen what a real product adds on top.

Key words

RAG
Retrieval-augmented generation. First find the pieces of your own documents that match the question, then hand them to a model so it answers from them.
Chunk
One small piece of a document, here one paragraph. Each chunk gets an id such as refunds-1.
Vector
A list of numbers that stands for a piece of text. Two texts about the same thing get similar lists.
Cosine similarity
A score for how alike two vectors are. Higher means closer.
Grounded prompt
A prompt that carries the facts the answer must come from, plus an order to use only those facts.
Stand-in
A tiny fake that takes the place of a real model so the code runs free on your computer.
Citation check
A small test that the answer only names sources the program actually supplied.

Why build it once by hand

Frameworks exist so you do not write all this yourself. So why do it by hand first?

Because a framework hides the pipeline, and a hidden pipeline is hard to fix. When an answer is wrong, the cause sits in one of a handful of places. Was the document cut badly? Did the search return the wrong pieces? Did the prompt leave something out? Did the model ignore what it was given? If you have built each step, you know where to look and what to print. If you have not, every wrong answer is a mystery.

The Wikipedia page on retrieval-augmented generation says RAG "improves LLMs by incorporating information retrieval before generating responses". That one sentence is the whole program. Retrieval first, generation second.

  • You can debug it

    You know the seven steps by heart, so you can print what each one produced and find which one went wrong.

  • You can judge a framework

    When a tool offers a setting, you can tell which step it changes and whether you need it.

  • You can leave one

    Nothing in the program is magic, so moving to another tool later means swapping parts, not starting over.

The seven parts

Here is the whole journey of one question. The first three parts happen once, before anyone asks anything. The last four happen on every question.

Text
Once:   chunk -> embed -> index
Always: retrieve -> prompt -> generate -> check
  1. 1

    Chunk

    Cut every help article into paragraphs and give each an id.

  2. 2

    Embed

    Turn each chunk into a vector. We use a toy way, explained below.

  3. 3

    Index

    Keep the chunks and their vectors together in a list, so you can search them.

  4. 4

    Retrieve

    Turn the question into a vector and keep the two chunks that score highest, if they score high enough.

  5. 5

    Prompt

    Paste those chunks in, each tagged with its id, and tell the model to answer only from them.

  6. 6

    Generate

    Send the prompt to a model and get words back.

  7. 7

    Check

    Make sure every id in the answer is one you really supplied.

The whole program

Here it is, in full. It holds four made-up help articles for an imaginary app. The next sections take it apart.

Python
import math, re

ARTICLES = {
    "password": "Open Settings and choose Security.\n\nTap Reset password. We email you a reset link.\n\nThe link works for 30 minutes.",
    "refunds": "You can ask for a refund within 14 days of paying.\n\nRefunds go back to the card you paid with.\n\nA refund takes 5 working days to arrive.",
    "export": "Open a project and tap Export.\n\nChoose PDF or PNG for your file.\n\nExports of large projects can take a minute.",
    "team": "Owners can invite teammates from the Team page.\n\nEach teammate gets an email invite.\n\nOnly owners can remove a teammate.",
}
STOP = {"a", "the", "to", "of", "can", "i", "my", "do", "how", "we", "you", "your", "for", "is", "with", "from", "and", "or", "an", "it", "in", "what", "much"}

def chunk(articles):  # one chunk per paragraph, each with an id
    return [(f"{name}-{n}", para) for name, text in articles.items()
            for n, para in enumerate(text.split("\n\n"), 1)]

def vectorize(text):  # toy bag-of-words: word -> count
    counts = {}
    for w in re.findall(r"[a-z0-9]+", text.lower()):
        if w not in STOP:
            counts[w] = counts.get(w, 0) + 1
    return counts

def cosine(a, b):
    dot = sum(n * b.get(w, 0) for w, n in a.items())
    size = math.sqrt(sum(n * n for n in a.values())) * math.sqrt(sum(n * n for n in b.values()))
    return dot / size if size else 0.0

def retrieve(question, index, k=2, min_score=0.2):
    q = vectorize(question)
    scored = [(round(cosine(q, vec), 2), cid, text) for cid, text, vec in index]
    return [s for s in sorted(scored, reverse=True)[:k] if s[0] >= min_score]

def build_prompt(question, hits):
    sources = "\n".join(f"[{cid}] {text}" for _, cid, text in hits)
    return ("Answer only from the sources. Cite the id in brackets. "
            "If the sources do not answer, say you do not know.\n"
            f"Sources:\n{sources}\nQuestion: {question}")

def fake_model(prompt):  # stand-in for a real model: quotes the first source it was given
    sources = re.findall(r"^\[([\w-]+)\] (.*)$", prompt, re.M)
    if not sources:
        return "I do not know. The help articles do not say."
    cid, text = sources[0]
    return f"{text} [{cid}]"

def check(answer, hits):
    cited = set(re.findall(r"\[([\w-]+)\]", answer))
    allowed = {cid for _, cid, _ in hits}
    if not cited:
        return "no citation (fine only when the model says it does not know)"
    return "ok" if cited <= allowed else f"BAD citation {sorted(cited - allowed)}"

chunks = chunk(ARTICLES)
index = [(cid, text, vectorize(text)) for cid, text in chunks]
print(len(chunks), "chunks indexed\n")

for question in ["How long do I have to ask for a refund?",
                 "How many minutes does the reset link work?",
                 "Can I pay with cryptocurrency?"]:
    hits = retrieve(question, index)
    print("Q:", question)
    print("retrieved:", [(score, cid) for score, cid, _ in hits])
    prompt = build_prompt(question, hits)
    print("--- prompt ---\n" + prompt + "\n--------------")
    answer = fake_model(prompt)
    print("A:", answer)
    print("check:", check(answer, hits), "\n")

Output

12 chunks indexed

Q: How long do I have to ask for a refund?
retrieved: [(0.41, 'refunds-1'), (0.2, 'refunds-3')]
--- prompt ---
Answer only from the sources. Cite the id in brackets. If the sources do not answer, say you do not know.
Sources:
[refunds-1] You can ask for a refund within 14 days of paying.
[refunds-3] A refund takes 5 working days to arrive.
Question: How long do I have to ask for a refund?
--------------
A: You can ask for a refund within 14 days of paying. [refunds-1]
check: ok 

Q: How many minutes does the reset link work?
retrieved: [(0.43, 'password-2'), (0.41, 'password-3')]
--- prompt ---
Answer only from the sources. Cite the id in brackets. If the sources do not answer, say you do not know.
Sources:
[password-2] Tap Reset password. We email you a reset link.
[password-3] The link works for 30 minutes.
Question: How many minutes does the reset link work?
--------------
A: Tap Reset password. We email you a reset link. [password-2]
check: ok 

Q: Can I pay with cryptocurrency?
retrieved: []
--- prompt ---
Answer only from the sources. Cite the id in brackets. If the sources do not answer, say you do not know.
Sources:

Question: Can I pay with cryptocurrency?
--------------
A: I do not know. The help articles do not say.
check: no citation (fine only when the model says it does not know) 

Tip

The generator here is a stand-in, called fake_model, so the code runs free with no key and no network. It does not think. It copies the first source it was given. A real model is slower, costs money for every token it reads and writes, can give a different answer each time you ask, and can fail or refuse. Nothing you read in the output below shows those things.

Taking it apart

One function at a time, in the order the data flows.

Data and chunk

ARTICLES is a dictionary: each key is an article name and each value is its text, with a blank line between paragraphs. The function chunk walks through the articles and splits each text on the blank lines. For every paragraph it makes a pair: an id such as refunds-1 (the article name plus the paragraph number) and the paragraph itself. Four articles with three paragraphs each give 12 chunks, which is the first line of the output.

Cutting by paragraph is the simplest honest choice, because a paragraph is usually one idea. Wikipedia notes that fixed-length cutting "may interrupt sentences or separate pronouns from their antecedents", which cutting at paragraph breaks mostly avoids. The Chunking lesson covers the other ways.

Embed: a toy on purpose

The function vectorize turns text into numbers. It lowercases the text, picks out the words, drops tiny filler words listed in STOP (like "the" and "to"), and counts how often each remaining word appears. "A refund takes 5 working days to arrive." becomes a tally: refund 1, takes 1, 5 1, working 1, days 1, arrive 1.

This is a bag of words. Wikipedia describes the bag-of-words model as one that "disregards word order (and thus most of syntax or grammar) but captures multiplicity", meaning it keeps how many times a word appears.

To be plain: this is a toy. It only matches identical words. "pay" and "paying" count as different words, and it has no idea that "money back" means "refund". A real embedding model fixes exactly that, because it turns meaning into numbers, not spelling. The Embeddings lesson explains how.

Cosine: comparing two vectors

The function cosine gives two tallies a score. For every word they share, it multiplies the two counts and adds the results up. Then it divides by the size of each tally, so that a long chunk does not win just by being long.

cos⁡(θ)=a⋅b∥a∥ ∥b∥\cos(\theta) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}

In words: multiply matching counts and add them (a dot b), then divide by the length of a times the length of b. Wikipedia puts it as "the dot product of the vectors divided by the product of their lengths". It also says the score "does not depend on the magnitudes of the vectors, but only on their angle". For counts, which are never negative, the score runs from 0 (nothing in common) to 1 (the same mix of words).

Retrieve: top two, with a floor

The function retrieve embeds the question, scores it against every chunk in the index, sorts from best to worst, keeps the first k (here 2) and then drops any that score under min_score (here 0.2).

That second rule is the safety net. Without it, a question with no good match would still get the two "least bad" chunks, and a model would happily build an answer from unrelated text. With it, a weak match returns nothing, and the program can say so. Too high a floor loses useful chunks. Too low lets junk in.

Prompt: a grounded prompt with ids

The function build_prompt writes the instruction, then one line per chunk in the form [id] text, then the question. Look at the instruction: answer only from the sources, cite the id in brackets, and say you do not know if the sources do not answer. That last clause gives the model a polite way out, so it does not feel pushed to invent something.

Generate: the stand-in

The function fake_model reads the prompt, finds the lines that start with [id], and returns the first one with its id added at the end. If there are no source lines, it says it does not know.

That is all it does. It is a stand-in for a real model, put here so the program runs free and gives the same answer every time. A real model writes new sentences, is slower, costs money for every token, can differ from run to run, and can fail or refuse.

Check: are the citations real?

The function check finds every [id] in the answer and compares that set with the ids that were retrieved. If the answer names an id you never supplied, it prints BAD citation. If there is no id at all, it says so, which is fine only when the model says it does not know.

Wikipedia notes RAG lets models "include sources in their responses, so users can verify the cited sources". This check is the first step of that: make sure the sources named are ones that were really given.

Reading the output

Now look at the three runs. Each shows the question, what was retrieved with its score, the full prompt, the answer and the check.

  • The refund question

    Two chunks passed the 0.2 floor: refunds-1 at 0.41 and refunds-3 at 0.2. The best one holds the 14 days, and that is the one quoted, with its id. The check says ok.

  • The reset link question

    The two chunks scored 0.43 and 0.41, almost a tie. The better answer is password-3, but password-2 scored a hair higher, so the stand-in quoted that. A real model sees both chunks and would likely use the right one. The check still says ok, because the id was real. A check on ids proves the source exists. It does not prove the answer is right.

  • The question with no match

    Nothing cleared the floor, because "pay" does not match "paying" in this toy. The sources section of the prompt is empty, and the answer says it does not know. This is the behaviour you want from a help bot.

Quick check

A question has no good match in your articles. What should the program do?

Quick check

The check printed ok, but the answer was the wrong sentence. Why?

What a production version adds

Our program is a skeleton. The skeleton is the same in a real product. What changes is what sits around each step. A production version adds six things, in three pairs.

  • Better search: a real embedding model and a vector store

    A real embedding model matches meaning, not just spelling, so "money back" finds the refund article. A vector store keeps the vectors and finds the closest ones, instead of scoring every chunk in a loop. The arXiv paper that named RAG used "a dense vector index of Wikipedia" for its memory.

  • Better experience and safety: streaming and permissions

    Streaming shows the answer word by word as it is written, so people do not stare at a blank box. Permissions make sure someone can only get chunks from documents they are allowed to read. Filter before the prompt is built, because anything in the prompt can end up in the answer.

  • Knowing it works: logging and evals

    Logging saves each question, the chunks retrieved and the answer, so you can find out later why something went wrong. Evals are a fixed set of questions with known good answers that you rerun after every change, to see if you made things better or worse.

None of the six change the shape of the program. Each swaps or wraps one function.

Tip

Log what retrieve returned, scores included, before anything else. Most wrong RAG answers are retrieval problems, and a retrieval log shows it in seconds.

Using frameworks well

When you pick up a framework, match its words to your program. Its loader and splitter are your chunk. Its embedding class is your vectorize. Its vector store and retriever are your index and retrieve. Its prompt template is your build_prompt. Its chain or pipeline is the loop at the bottom.

When something breaks, ask the same questions you would ask of your own code. What did the splitter return? What did the retriever return, and with what scores? What exactly was in the final prompt? You will know to ask because you have printed all of them yourself.

What to remember

  • RAG is a short loop

    Chunk and embed once. Then for each question: retrieve, prompt, generate, check.

  • Each part is one small function

    If you can print what a function returned, you can find the bug.

  • The rest is upgrades

    A real embedding model, a vector store, streaming, logging, evals and permissions all slot into the same shape.

Next: RAG use cases, where answering from your documents pays off.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.