AI engineering / Core / 02.15
From question to context
What your app does between a person's question and the model: clean, rewrite, search, filter, trim, cite and log, step by step.
13 min readFree
From the question to the right context
Last time we ended on a promise: the retrieval process, from question to the right context. Similarity search gave you one tool, a way to ask a store for its nearest pieces. This lesson puts that tool in its place. Between a person pressing Enter and a model seeing any text, your app does a short list of jobs. Together they are the retrieval step.
Think of a good librarian. You walk up and say, "that book about the thing with the bridges, I think?" They ask what you mean, try a few searches, pull a pile, set aside books you may not borrow, remove copies, and hand you a short stack that fits your bag. Then they write down what they gave you. A retrieval step does the same, in code.
Key words
- Retrieval
- Looking up pieces of text from your own collection that may help answer a question.
- Query
- The text you search with. It may be the user's question, or a changed version of it.
- Query rewriting
- Changing the question before searching, so the search has a better chance of matching.
- Hybrid search
- Using word search and meaning search together, then combining their results.
- Re-ranking
- Taking a short list of results and putting it in a better order with a more careful check.
- Token budget
- The limit on how much text you can send to a model in one go. The Tokenization lesson in Machine Learning Foundations goes deeper on tokens.
- Citation id
- A short label, such as ret-1, that travels with each piece, so an answer can say where it came from.
The whole path in seven steps
- 1
Decide
Does this message need a lookup at all? "Thanks!" does not.
- 2
Clean
Tidy the question: trim it, fix spacing, drop filler words if your search likes that.
- 3
Rewrite (optional)
Turn it into one better query, or several, or a made-up example answer.
- 4
Search
Run word search, meaning search or both. Take more results than you need.
- 5
Filter and re-rank
Remove what this user may not see or what is clearly weak. Put the rest in a better order.
- 6
Deduplicate and fit
Drop repeats, then keep only as much text as the budget allows.
- 7
Pass on, with ids
Send the pieces and their ids onward, and log what you found.
Not every app needs every step. Add one when you can name the problem it solves.
Decide, then clean
The cheapest search is the one you do not run. If someone types "hello" or "thanks", there is nothing to look up. A search would return the least-bad pieces anyway, and your app would send unrelated text to the model. A simple check, such as a short list of greetings, is often enough.
Cleaning is plain housekeeping. Trim spaces, lower-case the text for word search, and remove punctuation. For word search you may also drop very common words such as "the" and "do", often called stop words, because they match almost everything. For meaning search, keep the question as the person wrote it. The model that turns text into numbers was trained on normal sentences.
Rewrite the question
People ask short, vague or oddly worded questions. The wording in your documents is often different. Rewriting tries to close that gap. It is an old idea in search: Wikipedia describes query expansion as "the process of reformulating a given query to improve retrieval performance in information retrieval operations". Its list of techniques includes finding synonyms and searching for them too, and fixing spelling errors.
Multi-query: ask in several ways
One question can hide several meanings. "How long do I wait?" could be about delivery or about returns. The multi-query idea is to write a few versions of the question, search with each, and merge the results. Below, the question has no useful words for our made-up help pages, but the versions do.
A note on the code: fake_model is a stand-in so the code runs free. It returns fixed text and does not read the question. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. Your code must cope with all of that, for example by falling back to the original question.
CHUNKS = {
"ret-1": "You can return any item within 30 days for a full refund.",
"shp-1": "Standard delivery takes 3 to 5 working days.",
"war-1": "Every product has a one year warranty against faults.",
}
def fake_model(messages):
# A stand-in, not a real model: it returns fixed text so this code runs free.
return "How long does delivery take?\nHow long do I have to return an item?\nHow soon is express delivery?"
def words(text):
return set("".join(c if c.isalnum() else " " for c in text.lower()).split())
def best(query):
scores = {cid: len(words(query) & words(text)) for cid, text in CHUNKS.items()}
return max(scores, key=scores.get) if max(scores.values()) > 0 else None
question = "how long do I wait?"
variants = [question] + fake_model([{"role": "user", "content": "Rewrite: " + question}]).split("\n")
found = []
for v in variants:
cid = best(v)
print(f"{v!r:48} -> {cid}")
if cid and cid not in found:
found.append(cid)
print("merged ids:", found)Output
'how long do I wait?' -> None 'How long does delivery take?' -> shp-1 'How long do I have to return an item?' -> ret-1 'How soon is express delivery?' -> shp-1 merged ids: ['shp-1', 'ret-1']
The original question found nothing, each version found something, and the shared id was kept once. The cost is one extra model call and more searches.
HyDE: search with a made-up answer
Another trick flips the problem. A question and its answer often look different, but two answers look alike. So you ask a model to write a pretend answer, and search with that. The paper that introduced this is called "Precise Zero-Shot Dense Retrieval without Relevance Labels" (arXiv 2212.10496). Its abstract says: "Given a query, HyDE first zero-shot instructs an instruction-following language model (e.g. InstructGPT) to generate a hypothetical document. The document captures relevance patterns but is unreal and may contain false details."
That last sentence matters. The pretend answer is only a search key. You never show it to the user or treat it as true. The abstract goes on: "This second step ground the generated document to the actual corpus, with the encoder's dense bottleneck filtering out the incorrect details." So the real documents close to the pretend one are what you keep.
Search: words, meaning, or both
Keyword search
Matches the actual words. Great for names, codes and exact phrases. Misses a question that says "money back" when the page says "refund".
Vector search
Matches meaning, using the numbers from an embedding model. Handles different wording. It can drift when an exact term matters, such as a product code.
Hybrid search
Runs both and combines the two ranked lists. You get exact matches and meaning matches. You also get two systems to run and tune.
Whichever you pick, ask for more than you plan to keep. If the model will see three pieces, fetch twenty. The later steps (filter, re-rank, dedupe) need room to throw things away. The Performing Similarity Search lesson covers k, minimum scores and re-ranking.
Quick check
Your documents hold product codes like QX-4471. Which search is safest to include?
Filter, re-rank, dedupe, fit
Now you have a pile. Four short jobs turn it into a good stack.
Filter
Remove what must not be sent: pages this user may not read, old versions, other languages, weak scores. Do this in your code, not by asking the model to ignore it.
Re-rank
A more careful check on the short list, such as a model that reads the question and each piece together. It is slower, so run it on the short list only.
Deduplicate
Drop pieces with the same text, or nearly the same. Pages copied between folders cause this. Repeats waste your budget.
The fourth job is the budget. A model can only read so much per request, and every token costs money and time. So keep pieces in order of rank until the next one would not fit, then stop. Words are a rough count. A real app counts tokens with the model's own tokenizer, because a word can be more than one token, and leaves room for the instructions and the answer.
Pass the pieces with ids
Send each piece with a short id, such as [ret-1]. When the model writes an answer, you ask it to name the ids it used. Your app can then show "Source: returns page" and let people check. It also lets you test the answer later: did it use the piece you expected? The original RAG paper lists "providing provenance for their decisions" among the open problems of language models. Citation ids are the everyday, small version of that: a trail from the answer back to a piece of text.
A full retrieval function you can run
Time to put it all in one place. The code below uses ten made-up chunks from a pretend shop's help pages. It has no model and no network. Two parts are stand-ins on purpose. The "rewriter" is a fixed table of expansions (refund adds return, money and back), not a model. The "search" counts shared words, which is far simpler than a real vector search. It runs the whole path: skip, clean, rewrite, search, filter, dedupe and fit to a budget of 20 words, and it prints a trace of every step.
# 10 made-up chunks from a pretend shop's help pages: (id, public?, text)
CHUNKS = [
("ret-1", True, "You can return any item within 30 days for a full refund."),
("ret-2", True, "To return an item, print the label from your account page."),
("ret-3", True, "You can return any item within 30 days for a full refund."), # a copy
("shp-1", True, "Standard delivery takes 3 to 5 working days."),
("shp-2", True, "Express delivery arrives the next working day."),
("pay-1", True, "We accept cards and bank transfer at checkout."),
("acc-1", True, "Reset your password from the sign in page."),
("stf-1", False, "Staff only: a refund of money back above 500 needs a manager."),
("war-1", True, "Every product has a one year warranty against faults."),
("sup-1", True, "Our support team answers messages from 9 to 5 on weekdays."),
]
STOP = {"a", "an", "the", "do", "does", "how", "i", "my", "to", "can", "you", "we", "is", "for", "of", "get"}
SMALL_TALK = {"hi", "hello", "hey", "thanks", "thank"}
# Stand-in rewriter: fixed expansions, no model. A real rewriter would be a model call.
EXPAND = {"refund": ["return", "money", "back"], "arrive": ["delivery"], "wait": ["delivery"]}
def clean(question):
words = "".join(c if c.isalnum() else " " for c in question.lower()).split()
return [w for w in words if w not in STOP]
def rewrite(words):
extra = [e for w in words for e in EXPAND.get(w, [])]
return words + [e for e in extra if e not in words]
def search(words, chunks):
scored = []
for cid, public, text in chunks:
text_words = set("".join(c if c.isalnum() else " " for c in text.lower()).split())
score = len(set(words) & text_words)
if score > 0:
scored.append((score, cid, public, text))
return sorted(scored, key=lambda s: (-s[0], s[1]))
def retrieve(question, budget=20, min_score=1):
trace = [f"question : {question}"]
words = clean(question)
if not words or set(words) <= SMALL_TALK:
trace.append("skip : small talk, no search")
return [], trace
trace.append(f"cleaned : {words}")
words = rewrite(words)
trace.append(f"rewritten: {words}")
hits = search(words, CHUNKS)
trace.append(f"search : {[(cid, score) for score, cid, _, _ in hits]}")
hits = [h for h in hits if h[2] and h[0] >= min_score] # public chunks only, at least one shared word
trace.append(f"filtered : {[h[1] for h in hits]}")
seen, unique = set(), []
for h in hits: # same text twice: keep the first
if h[3] not in seen:
seen.add(h[3])
unique.append(h)
trace.append(f"deduped : {[h[1] for h in unique]}")
kept, used = [], 0
for score, cid, _, text in unique: # fit the budget, counted in words
size = len(text.split())
if used + size <= budget:
kept.append((cid, text))
used += size
trace.append(f"budget : kept {[c for c, _ in kept]}, {used} of {budget} words")
return kept, trace
for q in ["Hello!", "How do I get a refund?", "Do you sell bike helmets?"]:
found, trace = retrieve(q)
print("\n".join(trace))
if found:
print("to model :", [f"[{cid}] {text}" for cid, text in found])
elif "skip" in trace[-1]:
print("answer : (no lookup needed, just say hello back)")
else:
print("answer : I could not find that in our help pages.")
print()Output
question : Hello!
skip : small talk, no search
answer : (no lookup needed, just say hello back)
question : How do I get a refund?
cleaned : ['refund']
rewritten: ['refund', 'return', 'money', 'back']
search : [('stf-1', 3), ('ret-1', 2), ('ret-3', 2), ('ret-2', 1)]
filtered : ['ret-1', 'ret-3', 'ret-2']
deduped : ['ret-1', 'ret-2']
budget : kept ['ret-1'], 12 of 20 words
to model : ['[ret-1] You can return any item within 30 days for a full refund.']
question : Do you sell bike helmets?
cleaned : ['sell', 'bike', 'helmets']
rewritten: ['sell', 'bike', 'helmets']
search : []
filtered : []
deduped : []
budget : kept [], 0 of 20 words
answer : I could not find that in our help pages.Cleaning left one word, refund. The rewrite gave it three friends, so the search found four chunks. The top one, stf-1, is a staff page, and the filter removed it. The filter kept ret-1, ret-3 and ret-2. The dedupe dropped ret-3, because its text is the same as ret-1. The budget then kept ret-1 (12 words) and left out ret-2, which would have taken the total to 23 words, over 20.
The bike helmet question found nothing, and the app said so. That is a good outcome. A made-up answer would be worse, and your app should be able to say "I could not find that" and stop, or pass the empty result on so the model is told it has no source.
Log what you retrieved
When an answer is wrong, you need to know where it went wrong. Was the right chunk never found? Found, but filtered out? Found, but cut by the budget? Or was everything fine and the model misread it? Without a record you can only guess.
So log one line per question, much like the trace above: the original question, the rewritten queries, the ids and scores found, what each step removed, and what was finally sent. Questions can hold private details, so decide how long to keep logs and who can read them.
Quick check
A user says the answer was wrong. The log shows the right chunk was found, then dropped. Which step should you look at first?
Recap
Do the least
Skip small talk, clean the question, and only rewrite when it measurably helps.
Search wide, keep little
Fetch extra, then filter, re-rank, dedupe and fit the budget.
Keep a trail
Send ids with the pieces, and log every step so wrong answers can be traced.
Next: generation, turning retrieved pieces into a grounded answer.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.