AI engineering / Core / 02.18
Where RAG fits, and where not
Which products gain from answering out of your own documents, which do not, and a small program that scores an idea before you build it.
14 min readFree
Is this a job for RAG?
The last lesson, Build RAG by hand, ended with a promise: next, RAG use cases, where answering from your documents pays off. Here it is.
You now know how to build the machine. The harder skill is knowing when to build it. A hammer is a fine tool, but it is no help with a screw. RAG is the same. Some products get a lot from it. Others get a slow, costly system that does worse than a simple page of text.
By the end you will know three signs that RAG fits, three signs that it does not, six common uses with their risks, and how to measure whether one works. You will also run a small program that scores project ideas.
Key words
- RAG
- Retrieval-augmented generation. Find the right pieces of your documents first, then hand them to the model together with the question, so it answers from them.
- Knowledge base
- The pile of documents the system searches: help pages, manuals, policies, code.
- Citation
- A note in the answer that says which document it came from, so a reader can check it.
- Stale
- Out of date. A stale document still sits in the pile but no longer tells the truth.
- Permissions
- Rules about who may see what. The finance team may read a salary file. Most people may not.
- Context window
- The limit on how much text a model can take in at one time. The Context Window lesson goes deeper.
- Fit
- How well a tool matches a problem. A good fit means the tool's strengths are what the problem needs.
Three signs RAG fits
A model on its own answers from what it learned in training. The IBM page describes the gap this leaves: training datasets "are finite and limited to the information the AI developer can access". RAG fills that gap by letting the model look things up. So the first question is simple: is there something to look up?
The answers live in documents
A person could find the answer by opening the right page and reading a few lines. If that is true, a program can do the same and then explain it in plain words.
People need to see the source
The IBM page says that when RAG models cite their sources, "human users can verify those outputs to confirm accuracy". Think of a student who writes the book and page next to a fact. A reader can trust it because they can check it.
The content changes often
The Wikipedia page quotes Ars Technica: when new information becomes available, "all that's needed is to augment the model's external knowledge base with the updated information". You change the pile, not the model.
The AWS page makes the same point about fresh information. It says developers can use RAG to connect a model to "frequently-updated information sources". A shop that changes its returns rule every season is a good example. You do not retrain anything. You swap one page.
Tip
The more of the three signs a project has, the safer the bet. One sign alone can be enough, but then check the other two before you build.
Three signs RAG does not fit
RAG hands the model only the few pieces it finds. That is its strength, and it is also its limit. These are three ways the limit shows up.
It needs maths over a whole table
"What was our total sales last quarter?" needs every row added up. A search finds a handful of rows, not all of them, and a model is not a calculator. A database query or a spreadsheet formula does this job exactly, every time.
It needs all the documents at once
"Summarise the themes across all 900 contracts" cannot be answered from the best five pieces. The answer is spread over everything. Retrieval picks a few, so it would miss most of the story.
The content is tiny and never changes
If your whole knowledge fits on a page, paste it into the prompt. The Google Cloud page says a long context window is "a great way to provide source materials", and that RAG is for when you "need to provide more information than fits into the LCW", or need to scale up performance.
That last point deserves a moment. The Google page also says a RAG approach "will reduce the number of tokens, saving you time and cost". So RAG wins on a large pile. On a tiny pile, the extra search step is just extra work.
Even a good fit has limits. The IBM page says: "While RAG can reduce the risk of hallucinations, it cannot make a model error-proof." The Wikipedia page adds that RAG "does not prevent hallucinations". A hallucination is when a model states something made up as though it were fact. Plan for wrong answers even when everything is set up well.
Six common uses
The IBM page lists the kinds of work where RAG has been applied: specialised chatbots and virtual assistants, research, content generation, market analysis and product development, knowledge engines and recommendation services. It also sums up the idea well: RAG systems "essentially enable users to query databases with conversational language".
Below are six patterns you will meet again and again. They are general patterns, not claims about any one company. For each one we ask four things: who uses it, what the documents are, what can go wrong, and how you know it works.
Support and help centres
The IBM page describes this one: RAG plugs models into internal data "to equip customer support chatbots with the latest knowledge about a company's products, services and policies".
Users: customers with a question, and support staff who want a quick draft. Documents: help articles, past tickets, product pages. Biggest risk: stale pages. An old article says "free returns for 60 days" and the new rule is 30. The bot repeats the old one with a straight face. A second risk is a ticket that holds one customer's private details being shown to another.
How to measure: the share of questions the bot answers correctly with a citation, how often customers still contact a person afterwards, and how often a person has to fix the draft.
Internal knowledge search
Every company has facts hidden in wikis, chat threads and shared folders. People waste hours asking "who knows how we do X?". A RAG search lets them ask in plain words. The IBM page says responses with citations "point users directly toward the materials they need".
Users: staff. Documents: wikis, meeting notes, slide decks. Biggest risk: permissions. The same pile holds the public handbook and the private pay review. If the search ignores who is asking, it will happily show the private page to anyone. The fix is to filter by the asker's rights before the search, not after the answer is written.
How to measure: how often the right document is among the pieces found, and how long people take to find an answer, compared with before.
Policy and legal look-ups
Rulebooks, contracts and compliance guides are long, and people need one clause, not the whole book. Citations matter most here, because someone will act on the answer.
Users: staff who need to know "may I do this?", and the lawyers or specialists who check. Documents: policies, contracts, regulations. Biggest risks: an old version of a policy, two versions that disagree, and a model that sounds sure while misreading a clause. The Wikipedia page notes that when sources conflict, a model may "merge outdated and updated information in a misleading manner".
How to measure: ask a specialist to mark a sample of answers as right, partly right or wrong, and check that every answer points to the exact clause. Keep a person in the loop for anything with real consequences.
Product docs assistants
A new developer wants to know how to call a feature. Your guide has the answer, but on page 40 of a long site. An assistant that finds it and quotes it saves the trip.
Users: developers and customers. Documents: guides, API pages, release notes. Biggest risk: version mix-ups. The answer is right for last year's version and wrong for this one. Store the version with each piece and filter on it.
How to measure: ask real questions from your support inbox and check that the answer matches the current docs. Track which questions find nothing. Those are gaps in the docs themselves.
Sales enablement
A salesperson is on a call and needs the answer to a security question, or a case study for a similar customer. RAG can pull it from the pile of decks, one-pagers and answered questionnaires.
Users: sales and pre-sales staff. Documents: decks, case studies, past replies. Biggest risks: old claims. A slide that promised a feature the product dropped is now a promise nobody can keep. Also risky: private details from one customer's deal appearing in another's.
How to measure: how often people use the suggested answer without rewriting it, how quickly questionnaires get finished, and whether anyone reports a wrong claim.
Coding assistants over your code
A general model has never seen your code. Search over your repository lets it find the function you mean and the style your team uses, so its suggestions fit.
Users: developers. Documents: source files, tests, design notes. Biggest risks: code that changed since it was indexed, secrets such as keys left in files, and files a person is not allowed to see. Keep keys out of the pile, and keep it fresh.
How to measure: how often suggestions compile and pass tests, and how often a developer keeps them. Code gives you a clear signal because a test either passes or it does not.
Measuring success
Every use above ends with the same advice: measure it with real questions. Build a small test set, say a hundred questions people really ask, with the right document noted next to each. Then check two things separately: did search find the right document, and was the answer built from it correctly?
The first check has a name, the hit rate. In words: out of all the questions you asked, how many times was the right document among the pieces found? As a formula:
A made-up example: you ask 100 questions and the right document turns up for 82 of them. The hit rate is 82 divided by 100, or 0.82. Those numbers are invented to show the sum. The point is to keep the test set and run it again after every change, so you can see whether a change helped or hurt.
Also watch the dull things. How many documents are stale? How many answers lack a citation? How often does the system say "I don't know" when it should? The Wikipedia page warns that a model may "generate answers even when they should indicate uncertainty". Testing for that is part of the job.
Keeping the pile healthy is ongoing work. The IBM page states it flatly: "Knowledge bases must be continually updated to maintain the RAG system's quality and relevance." Treat it as a chore with an owner, not a one-off setup.
A fit checker in plain Python
Let us turn the six signs into a small program. Each sign gets a weight: a plus for signs that favour RAG, a minus for signs against it. A project description is a Python dictionary, which is a list of labels with values. The program adds up the weights and prints a verdict with its reasons.
There is no model in this code, so nothing is faked and nothing costs money. The weights are our own choices for teaching. They are a thinking aid, not a law.
- 1
Write down the weights
Three plus signs and three minus signs, each with a number and a plain reason.
- 2
Add up what applies
For each sign that is true for the project, add its weight and remember its reason.
- 3
Turn the score into a verdict
5 or more is a good fit, 2 to 4 is maybe, below 2 is a poor fit.
# Each sign gets a weight. Plus signs push toward RAG, minus signs push away.
SIGNS = {
"answers_in_documents": (3, "the answers live in documents"),
"needs_sources": (2, "people need to see where an answer came from"),
"changes_often": (2, "the content changes often"),
"needs_calculation": (-4, "it needs maths over whole tables"),
"needs_all_documents": (-4, "it needs every document at once"),
"tiny_and_static": (-3, "the content is tiny and never changes, so it fits in the prompt"),
}
def check(project):
score = 0
reasons = []
for sign, (weight, text) in SIGNS.items():
if project.get(sign):
score += weight
reasons.append(f"{weight:+d} {text}")
if score >= 5:
verdict = "good fit"
elif score >= 2:
verdict = "maybe"
else:
verdict = "poor fit"
return score, verdict, reasons
projects = [
{"name": "Help centre bot", "answers_in_documents": True, "needs_sources": True, "changes_often": True},
{"name": "Policy look-up", "answers_in_documents": True, "needs_sources": True},
{"name": "Sales numbers Q&A", "answers_in_documents": True, "needs_calculation": True, "changes_often": True},
{"name": "Summarise all 900 contracts", "answers_in_documents": True, "needs_all_documents": True, "needs_sources": True},
{"name": "Staff handbook helper", "answers_in_documents": True, "needs_sources": True, "tiny_and_static": True},
]
for p in projects:
score, verdict, reasons = check(p)
print(f"{p['name']}: {verdict} (score {score})")
for r in reasons:
print(" ", r)Output
Help centre bot: good fit (score 7)
+3 the answers live in documents
+2 people need to see where an answer came from
+2 the content changes often
Policy look-up: good fit (score 5)
+3 the answers live in documents
+2 people need to see where an answer came from
Sales numbers Q&A: poor fit (score 1)
+3 the answers live in documents
+2 the content changes often
-4 it needs maths over whole tables
Summarise all 900 contracts: poor fit (score 1)
+3 the answers live in documents
+2 people need to see where an answer came from
-4 it needs every document at once
Staff handbook helper: maybe (score 2)
+3 the answers live in documents
+2 people need to see where an answer came from
-3 the content is tiny and never changes, so it fits in the promptRead the output. The help centre bot scores 7: all three plus signs. The policy look-up scores 5, because its documents change slowly, which is still enough. The sales numbers project has the right documents and changing content, but the minus 4 for maths over whole tables drags it down to 1. The contracts summary lands on 1 for the same size of penalty. The staff handbook helper is the interesting one: a score of 2 means maybe. A handbook that small might just fit in the prompt, so paste it in first and only add search if it outgrows that.
Tip
Change a weight and run it again. Raise the maths penalty to 2 and watch the sales project move. Seeing a verdict shift is how you learn which signs matter most for you.
A real project has more shades than yes or no, and a score cannot replace talking to the people who will use the product. But writing the signs down forces the right questions early, before you spend weeks building.
Quick check
Quick check
Your team wants a bot that answers "What was our total revenue last year?" from a big spreadsheet. What is the best advice?
Quick check
An internal search shows a private pay review to the wrong person. Which risk is that?
To recap: RAG fits when answers live in documents, when people need sources and when content changes often. It fits poorly when the job needs maths over whole tables, needs every document at once, or when the content is tiny and fixed. In every case, test with real questions and keep the pile fresh and permission-aware.
Next: RAG vs fine-tuning, which tool for which problem.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.