AI engineering / Beginner / 01.31
Retrieval-augmented generation
How to make a model answer from your own documents: find the right pieces first, hand them over with the question, and show where the answer came from.
14 min readFree
Answering from your own documents
The last lesson, Vector DBs, ended with a promise: next, retrieval-augmented generation, answering from your own documents. Here it is.
Picture an exam. In a closed-book exam you answer from memory, and when memory fails you guess. In an open-book exam you may look things up first, then write the answer with the page open in front of you. Retrieval-augmented generation, called RAG for short, turns a model's closed-book exam into an open-book one.
By the end you will know what RAG is, why people use it, the two phases it runs in, and where it breaks. You will also run a whole tiny RAG pipeline in plain Python, with no keys and no network.
Key words
- RAG
- Retrieval-augmented generation. Before the model answers, you fetch the most useful pieces of your own documents and put them in front of it.
- Retrieval
- Finding the pieces of text that best match a question.
- Chunk
- One small piece of a document, such as a sentence or a paragraph. We search chunks, not whole files.
- Embedding
- A list of numbers that stands for the meaning of a piece of text. Texts with similar meaning get similar lists. The Vector DBs lesson covers this.
- Index
- A store where every chunk sits next to its embedding, ready to be searched.
- Context
- Everything the model can see when it answers: your question plus anything you paste in beside it.
- Citation
- A label that tells the reader which document a statement came from.
Tip
This is the overview lesson. The Core lessons that come after it go step by step through chunking, indexing, retrieval and generation, one piece at a time. Here you meet the whole machine once, so every later piece has a place to land.
Why a model needs your documents
A language model learns from a huge pile of text, and then the learning stops. After that its knowledge is frozen. It has never seen your company's help pages, last week's price change or the contract you signed this morning. Ask it anyway and it may still answer, politely and with total confidence, and be wrong.
The AWS page on RAG makes the same point with a person: think of a model as an over-eager new employee who ignores the news yet answers every question as if sure. RAG is the fix where you hand that employee the right folder before each question.
Wikipedia describes the idea in one sentence. RAG is "a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources". The model first looks at a set of documents you choose, then replies. Those documents add to what it learned in training.
People use it for three reasons.
Fresh and private knowledge
The model can use facts that are newer than its training, or that were never public, such as your own help centre. You change the documents, not the model.
Fewer made-up answers
Wikipedia says RAG helps reduce hallucinations, the made-up answers a model gives with full confidence. Google's page says that giving the model facts as part of its input can reduce them.
Sources you can show
Because you know which pieces went in, the answer can point back to them. AWS says RAG lets the model present information with source attribution, so users can look the document up themselves.
Tip
RAG does not change the model. It changes what the model is shown. That is why you can update your knowledge in minutes: edit a document, and the next answer already uses it.
Where the idea comes from
The name comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401). It was accepted at NeurIPS 2020. Its abstract starts with a problem: models store facts in their own numbers, and their ability to "access and precisely manipulate knowledge is still limited".
The same abstract says that "providing provenance for their decisions and updating their world knowledge remain open research problems". Provenance means where an answer came from. In everyday words: the paper wanted models that can show their sources and learn new facts without a full re-teach.
Its answer is in this sentence of the abstract: "We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation."
Two odd words, so here they are in plain terms. Parametric memory is what the model carries inside its own numbers, its parameters. Non-parametric memory is a library outside the model that it can look things up in. The paper's library was "a dense vector index of Wikipedia, accessed with a pre-trained neural retriever". A dense vector index is a store of embeddings, and a retriever is the part that searches it.
Most people now do it more simply: search a store of their own chunks and paste the best ones into the prompt. Wikipedia calls this "prompt stuffing": extra context is added to the user's input so the model leans on the supplied data over what it learned in training.
The two phases
Every RAG system runs in two phases. The first happens ahead of time and the second happens for every question.
Think of a library. First a librarian sorts the books and writes a card for each. Later a visitor asks a question, and the librarian brings back two or three pages to read the answer from.
Phase one: indexing
Indexing is the librarian's sorting. You do it once for your documents, and again whenever they change. Wikipedia and AWS both describe it the same way: the data is turned into embeddings, numbers that stand for meaning, and kept in a vector database so it can be searched.
- 1
Split
Cut each document into chunks. Small pieces are easier to match to a question than a whole file.
- 2
Embed
Turn each chunk into an embedding, a list of numbers for its meaning.
- 3
Store
Save every chunk with its embedding and its document id, so you can find it and cite it later.
Phase two: answering
Answering runs every time someone asks a question. The AWS page lays it out in the same order: the question is turned into numbers, matched against the store, and the matching text is given to the model beside the question.
- 1
Embed the question
Turn the question into numbers in the same way you did for the chunks. Only then can the two be compared.
- 2
Retrieve
Find the few chunks whose numbers sit closest to the question's. This is the top few, often called top-k, where k is how many you keep.
- 3
Build the prompt
Paste those chunks into the prompt with a label on each, then add the question and an instruction to answer from them only.
- 4
Generate
The model writes the answer from the prompt. If it kept the labels, the answer can cite them.
In short: Documents → Split → Embed → Store, then, per question: Question → Retrieve → Prompt → Model → Answer with sources.
A whole RAG pipeline in plain Python
Now the whole machine in one small program. It has six short help-centre notes. The program splits them into sentences, turns each sentence into a toy vector, takes the top 2 for a question, builds a prompt with [id] labels and asks a model.
Two parts are toys on purpose. The "vector" is only a count of the useful words in a sentence (a real system uses an embedding that captures meaning, as in the Vector DBs lesson). And the model is a stand-in.
That stand-in, fake_model, is not a real model. It exists so the code runs free, with no key and no network. All it does is copy the first labelled piece out of the prompt and put "According to our notes:" in front. A real model is slower, costs money for every token it reads and writes, can give a different answer on each run, and can fail or refuse. The stand-in does none of that, so do not mistake its tidy answer for what a real one does.
import math
import re
# 1. Our documents: six short help-centre notes.
notes = {
"refunds": "You can ask for a refund within 30 days of buying. Refunds go back to your original payment method.",
"shipping": "Standard shipping takes 5 working days. Express shipping takes 2 working days.",
"password": "To reset your password, click Forgot password on the sign-in page. We will email you a reset link.",
"invoices": "Invoices are emailed on the first day of each month. You can also download them from the Billing page.",
"cancel": "You can cancel your plan at any time from the Billing page. Your access stays on until the end of the month.",
"support": "Our support team answers emails from Monday to Friday. Replies usually arrive within one working day.",
}
STOP = {"a", "an", "the", "to", "of", "in", "on", "at", "is", "are", "do", "i", "my",
"you", "your", "we", "can", "will", "from", "or", "and", "for", "how", "what",
"when", "does", "it", "each", "any", "also", "back", "go", "us", "our"}
def words(text):
return [w for w in re.findall(r"[a-z]+", text.lower()) if w not in STOP]
def vectorize(text):
counts = {}
for w in words(text):
counts[w] = counts.get(w, 0) + 1
return counts
def similarity(a, b):
dot = sum(a[w] * b.get(w, 0) for w in a)
size = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
return dot / size if size else 0.0
# 2. Indexing: split into sentences, turn each into a vector, store.
pieces = []
for doc_id, text in notes.items():
for sentence in re.split(r"(?<=\.)\s+", text):
pieces.append({"id": doc_id, "text": sentence, "vec": vectorize(sentence)})
print("INDEXING:", len(notes), "notes ->", len(pieces), "pieces")
print("example piece:", pieces[0]["id"], pieces[0]["vec"])
# 3. Answering: embed the question, retrieve the top 2, build the prompt, generate.
def fake_model(messages):
# Stand-in so this runs free. It does not think: it quotes the first piece in the prompt.
first = [line for line in messages[-1].split("\n") if line.startswith("[")][0]
return "According to our notes: " + first
def answer(question):
q = vectorize(question)
print("\nQUESTION:", question)
print("question vector:", q)
ranked = sorted(pieces, key=lambda p: similarity(q, p["vec"]), reverse=True)
top = ranked[:2]
for p in top:
print("retrieved: %.2f [%s] %s" % (similarity(q, p["vec"]), p["id"], p["text"]))
context = "\n".join("[%s] %s" % (p["id"], p["text"]) for p in top)
prompt = "Answer using only the notes below. Cite the [id].\n" + context + "\nQuestion: " + question
print("PROMPT:\n" + prompt)
print("ANSWER:", fake_model([prompt]))
answer("How long do I have to ask for a refund?")
answer("Can I get my money back?")Output
INDEXING: 6 notes -> 12 pieces
example piece: refunds {'ask': 1, 'refund': 1, 'within': 1, 'days': 1, 'buying': 1}
QUESTION: How long do I have to ask for a refund?
question vector: {'long': 1, 'have': 1, 'ask': 1, 'refund': 1}
retrieved: 0.45 [refunds] You can ask for a refund within 30 days of buying.
retrieved: 0.00 [refunds] Refunds go back to your original payment method.
PROMPT:
Answer using only the notes below. Cite the [id].
[refunds] You can ask for a refund within 30 days of buying.
[refunds] Refunds go back to your original payment method.
Question: How long do I have to ask for a refund?
ANSWER: According to our notes: [refunds] You can ask for a refund within 30 days of buying.
QUESTION: Can I get my money back?
question vector: {'get': 1, 'money': 1}
retrieved: 0.00 [refunds] You can ask for a refund within 30 days of buying.
retrieved: 0.00 [refunds] Refunds go back to your original payment method.
PROMPT:
Answer using only the notes below. Cite the [id].
[refunds] You can ask for a refund within 30 days of buying.
[refunds] Refunds go back to your original payment method.
Question: Can I get my money back?
ANSWER: According to our notes: [refunds] You can ask for a refund within 30 days of buying.Read it stage by stage. The first line is indexing: 6 notes became 12 pieces, and each piece got a vector. The example piece is the first sentence of the refunds note. Its vector keeps five useful words: ask, refund, within, days, buying. The small words like "you" and "of" were dropped on purpose, because they are in almost every sentence and tell us nothing.
Then the first question. "How long do I have to ask for a refund?" became four useful words. Retrieval scored every piece against them. The best one scored 0.45 and is the refund sentence. The score runs from 0 for no shared words to 1 for the same words. It is a cosine similarity, which in plain terms measures how much two word lists overlap.
Next comes the prompt: each piece on its own line with its note id in square brackets, then the question. Those brackets let an answer cite its source. The stand-in answered by quoting the top piece with its id.
Tip
Try it yourself: add a seventh note, then ask about it. You changed only the documents.
Quick check
Quick check
A shop changes its refund window from 30 days to 14. In a RAG setup, what do you change?
Where it goes wrong
RAG is only as good as its retrieval. If the wrong pieces come back, the model answers from the wrong pieces. Google's page says it directly: if the retrieved information is irrelevant, the generation "could be grounded but off-topic or incorrect".
Our program shows it. The second question was "Can I get my money back?" A person knows that means a refund, but the notes say "refund", not "money back", and our toy vectors only match shared words. The question kept just two useful words, get and money. Both scores were 0.00. Nothing matched, yet the program still passed the top 2 pieces on, and the stand-in still answered with the refund sentence.
That answer is right by luck: the sort kept the original order of the pieces, and the refund note happens to come first. Change the order of the notes and it would quote something unrelated, with a confident citation. A real embedding would likely link "money back" to "refund", but it has misses of its own, only harder to see.
Weak retrieval
The right piece is not among the top few, because the words differ, the chunk was cut badly, or the question is vague. The model never sees the answer.
Weak reading
The right piece is in the prompt but the model misreads it. Wikipedia's example: a model pulled a chapter title phrased as a question from a book and stated it as a fact. It had no sense of the context.
Weak documents
The notes are old, wrong or contradict each other. RAG faithfully repeats them. Google's page lists curating the source data as part of getting RAG quality up.
Wikipedia sums up the limit: RAG "can reduce factual errors, but it does not solve all the problems". Treat a cited answer as a lead you can check, not as proof.
Tip
When a RAG answer is bad, look at what was retrieved before you blame the model. Print the retrieved pieces, as our program does. Most bad answers are bad retrieval.
Quick check
A RAG system gives a wrong answer. What should you check first?
What the Core lessons improve
Every weak spot above has a fix, and each fix is a lesson to come. Chunking decides how documents are cut. Indexing decides how chunks are stored. Retrieval decides how the best ones are found. Generation decides how the prompt is built and how the model is told to use it.
One more thing: the pieces you paste in use space in the model's context window, and you pay for them in tokens. The Tokenization lesson in Machine Learning Foundations goes deeper on tokens.
What to remember
It is an open book
RAG fetches pieces of your own documents and puts them in the prompt, so the model answers from them.
Two phases
Index once: split, embed, store. Answer each time: embed the question, retrieve, build the prompt, generate.
Retrieval decides
Wrong pieces in means a wrong answer out. Always look at what was retrieved.
Next: RAG with dynamic filters, narrowing the search before it starts.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.