AI engineering / Beginner / 01.28
Context Evaluation
Check the context, not just the answer: did you fetch the right pieces, is the answer backed by them, was anything missing, and what did it cost?
12 min readFree
Did the context help?
The last lesson, Context Failure Modes, ended with a promise: next, context evaluation, measuring whether the context helped. Here it is.
Picture a student who hands in a correct answer. Did they study the right pages, or did they guess and get lucky? If you only check the answer, you cannot tell. The next exam will find out.
AI products work the same way. A right answer today can hide a bad context that fails tomorrow. A wrong answer can come from a good context that the model ignored. So you check two things: the answer, and the context that produced it.
By the end you will know four things worth measuring, how to build a small test set by hand, and how to compare two ways of building context with real numbers. You will run one short Python program with no keys and no network.
Key words
- Context piece
- One chunk of text you can put in the prompt: a help page, a note, a past message.
- Context builder
- The code that picks which pieces go into the prompt for a given question.
- Test set
- A small list of questions where you already know the right answer, and which pieces are needed to give it.
- Label
- A note you write by hand saying which pieces a question needs. It is your answer key.
- Precision
- Of the pieces you picked, how many were actually needed.
- Recall
- Of the pieces that were needed, how many you picked.
- Groundedness
- How much of the answer can be traced back to the context you gave.
Four things to check
Think of packing a bag for a trip. You can pack the wrong things, forget something, pack so much it is heavy, or pack well and still not use it. Context has the same four problems, and each has a check.
Right pieces?
Did the builder fetch the pieces the question needs, and not many others? This is precision and recall of the pieces.
Grounded?
Can every claim in the answer be traced to a piece in the context? A claim with no source is a made-up fact.
Anything missing? How heavy?
Was a needed piece left out, and how many tokens did the context cost? More tokens mean more money and more waiting.
The Tokens lesson explains tokens. In short, a token is a small piece of text that a model counts. Every token in your context costs something, so a long context is not free.
Tip
Check the context first. If the right pieces never reached the model, no prompt trick will save the answer.
Precision and recall
These two words come from a very old idea, finding the right items in a big pile. The Wikipedia page on precision and recall defines them for any data retrieved from a collection. It says precision is "the fraction of relevant instances among the retrieved instances" and recall is "the fraction of relevant instances that were retrieved".
Here that means: instances are your context pieces, relevant means your label says the question needs it, and retrieved means your builder picked it.
In words: precision asks "how much of what I picked was useful?" Recall asks "how much of what I needed did I get?" Think of a fishing net. Precision is how much of the catch is fish and not boots. Recall is how many of the fish in the lake you caught.
The two pull against each other. Pick everything and recall is perfect, but precision is poor and the context is heavy. Pick almost nothing and precision may look fine, but you missed pieces. The Ragas docs put the second case plainly: "recall is about not missing anything important."
The Precision and Recall lessons in Machine Learning Foundations go deeper, including how the two combine into one score. Here you only need the two fractions.
Is the answer grounded?
Good pieces do not guarantee a good answer. The model can still add things that no piece says. Groundedness checks this. Take each claim in the answer and ask: which piece backs this up?
The Ragas docs call a related score faithfulness. They say it measures "how factually consistent a response is with the retrieved context", and that "a response is considered faithful if all its claims can be supported by the retrieved context". The score is the number of supported claims divided by the total number of claims.
Those docs use another model to judge each claim. We will use something much simpler and cruder, because it runs free: a sentence counts as grounded if it shares at least two content words with some piece we included. This is a rough test. It can be fooled by a sentence that reuses words and says something false, and it can miss a true sentence that uses other words. It is good enough to show the idea.
Tip
A sentence that is not grounded is not always wrong. It just has no source in your context. Treat it as a flag to read, not as a verdict.
Build a small test set
To measure anything, you need an answer key. The good news: it can be tiny. Five to twenty questions you really care about beat a thousand you never read.
- 1
Collect real questions
Use questions people actually asked, or ones you expect. Mix easy ones with awkward ones.
- 2
Label by hand
For each question, write down the ids of the pieces needed to answer it. Some questions need two pieces.
- 3
Freeze it
Do not edit the test set while you compare builders. A moving answer key makes every result meaningless.
- 4
Change one thing
Make one change to the builder, such as how many pieces it picks. Run the whole test set again.
- 5
Compare the numbers
Look at precision, recall, groundedness and cost side by side. Keep the change only if the numbers say it helped.
Why one change at a time? If you change two things and the score moves, you do not know which one did it. This is the same habit as testing a recipe: change the salt or the oven, never both at once.
Do not stop at averages. Look at the single worst question. An average of 0.8 can hide one question that fails every time.
Compare two builders
Time to run it. We have 8 context pieces, each with a day number. We have 5 test questions, each labelled with the pieces it needs. Question 5 needs two pieces.
Two builders compete. The keyword builder scores each piece by how many words it shares with the question and takes the best 3 that share at least one. The newest-3 builder ignores the question and always takes the 3 newest pieces. Newest-3 sounds silly, but it is what many apps do when they paste in the latest messages and hope.
The model is a stand-in. The function fake_model just copies the best-matching piece, or returns a fixed made-up sentence when nothing matches. It runs free and instantly. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The scores below are about the context builders only.
For cost, we count words as a rough stand-in for tokens. Real tokenizers count differently, so treat the number as a size comparison, not a bill.
import re
PIECES = {
"p1": (1, "Refunds are allowed within 30 days of purchase."),
"p2": (2, "Shipping to Canada takes 5 to 8 days."),
"p3": (3, "Password reset emails expire in one hour."),
"p4": (4, "The free plan includes 3 projects."),
"p5": (5, "Paid plans include unlimited projects and priority support."),
"p6": (6, "Support is open Monday to Friday."),
"p7": (7, "Invoices are emailed on the first day of each month."),
"p8": (8, "The office is closed on public holidays."),
}
TESTS = [
("How many days do I have to ask for a refund?", {"p1"}),
("How long does shipping to Canada take?", {"p2"}),
("When does a password reset email expire?", {"p3"}),
("How many projects are on the free plan?", {"p4"}),
("When is support open, and when are invoices sent?", {"p6", "p7"}),
]
STOP = {"the", "a", "is", "are", "do", "does", "to", "of", "on", "in", "i", "and",
"when", "how", "many", "long", "have", "ask", "for", "take", "each"}
def words(text):
return set(re.findall(r"[a-z0-9]+", text.lower())) - STOP
def keyword_builder(question):
q = words(question)
scored = [(len(q & words(t)), pid) for pid, (day, t) in PIECES.items()]
scored = [s for s in scored if s[0] > 0]
scored.sort(key=lambda s: (-s[0], s[1]))
return [pid for _, pid in scored[:3]]
def newest_builder(question):
by_day = sorted(PIECES, key=lambda pid: -PIECES[pid][0])
return by_day[:3]
def fake_model(question, piece_ids):
# Stand-in: copies the included piece that shares most words with the question.
q = words(question)
best = max(piece_ids, key=lambda p: len(q & words(PIECES[p][1])), default=None)
if best is None or not q & words(PIECES[best][1]):
return "We usually sort this out within a week."
return PIECES[best][1]
def grounded(answer, piece_ids):
sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer) if s]
ok = [any(len(words(s) & words(PIECES[p][1])) >= 2 for p in piece_ids) for s in sentences]
return sum(ok) / len(sentences)
def run(builder):
rows = []
for question, needed in TESTS:
got = builder(question)
hit = needed & set(got)
precision = len(hit) / len(got) if got else 0
recall = len(hit) / len(needed)
answer = fake_model(question, got)
tokens = sum(len(PIECES[p][1].split()) for p in got)
rows.append((precision, recall, grounded(answer, got), tokens))
return rows
results = {"keyword": run(keyword_builder), "newest-3": run(newest_builder)}
print("builder question precision recall grounded words")
for name, rows in results.items():
for i, (p, r, g, t) in enumerate(rows, 1):
print(f"{name:<11} q{i:<8} {p:<10.2f} {r:<7.2f} {g:<9.2f} {t}")
print()
print("builder avg precision avg recall avg grounded avg words")
for name, rows in results.items():
n = len(rows)
avg = [sum(row[k] for row in rows) / n for k in range(4)]
print(f"{name:<11} {avg[0]:<14.2f} {avg[1]:<11.2f} {avg[2]:<13.2f} {avg[3]:.1f}")Output
builder question precision recall grounded words keyword q1 0.50 1.00 1.00 16 keyword q2 1.00 1.00 1.00 8 keyword q3 1.00 1.00 1.00 7 keyword q4 0.50 1.00 1.00 14 keyword q5 0.67 1.00 1.00 24 newest-3 q1 0.00 0.00 0.00 23 newest-3 q2 0.00 0.00 0.00 23 newest-3 q3 0.00 0.00 0.00 23 newest-3 q4 0.00 0.00 0.00 23 newest-3 q5 0.67 1.00 1.00 23 builder avg precision avg recall avg grounded avg words keyword 0.73 1.00 1.00 13.8 newest-3 0.13 0.20 0.20 23.0
Reading the table
Start with the keyword builder. Its recall is 1.00 on every question: it never missed a needed piece, including both pieces for question 5. Its average precision is 0.73. It sometimes picked an extra piece. In q1 it grabbed the shipping page too, because both mention "days". In q4 it grabbed the paid plans page, because both mention "projects". That extra piece is noise. The cost is a longer context, 13.8 words on average.
Now the newest-3 builder. Average precision is 0.13 and average recall is 0.20. It only passed q5, and by luck: the two pieces q5 needs happen to be among the newest. On q1 to q4 it returned nothing useful, so the stand-in model fell back to its made-up sentence, and groundedness dropped to 0.00 on those. It was also heavier: 23.0 words every time, because it always sends the same 3 pieces.
This is why you check the context. The answers on q1 to q4 were wrong for newest-3, and you could see why without reading a single answer: the right pieces never arrived.
Quick check
A builder has recall 1.00 but precision 0.50 on a question. What happened?
What the numbers hide
Look at q5 again. Both builders got recall 1.00 there, yet our stand-in model only returned one sentence, about support. It never mentioned invoices. So full recall of the pieces does not mean a full answer. The pieces were there and the model did not use them.
That is a fifth thing to watch, and it needs a different check: is the answer complete? One simple way is to write down the facts the answer must contain, then look for each one. The Ragas docs use a similar idea for context recall. They break a reference answer into claims and check whether the retrieved context supports each one.
This is also why the groundedness score of 1.00 for the keyword builder deserves a pinch of salt. A grounded answer can still be a partial one. Each score answers one question, and none of them answers all.
Where this goes next
Our check is tiny on purpose. The Ragas paper, on arXiv, describes a framework for "reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines". It lists dimensions to check, including "the ability of the retrieval system to identify relevant and focused context passages" and the ability of the model to use those passages in a faithful way. That matches the first two checks in this lesson.
Full evaluation gets its own lessons in the advanced level: bigger test sets, scoring with another model, and watching quality over time. For now, one habit is enough: write a small answer key, change one thing, and compare the numbers.
Quick check
You change your builder and also shorten your prompt, and the score goes up. What is the problem?
Tip
Keep your test set and your results in a file. Next month, when you change the builder again, you can compare against today.
Recap
Check the context
Precision and recall of the pieces tell you if the right context arrived and how much noise came with it.
Check the answer
Groundedness asks if each claim traces to a piece. Completeness asks if the needed facts are all there.
Test like a scientist
A small hand-labelled set, one change at a time, and the numbers side by side, including cost.
Next: embeddings, turning text into numbers that capture meaning.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.