esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.19

RAG vs Fine-tuning

Three ways to make a model do better work: a better prompt, retrieval, or more training. How to pick one, and when to use two.

14 min readFree

Which tool for which problem

The last lesson, RAG Usecases, ended with a promise: next, RAG vs fine-tuning, which tool for which problem. Here it is.

Picture a new shop assistant who gives wrong answers. You could hand them a thick folder of current price lists and policies. You could send them on a training course so they speak the way your brand speaks. Or you could simply write them clearer instructions. All three fix something, and they fix different things.

The folder is RAG, the course is fine-tuning, and the clearer instructions are a better prompt. Picking the wrong one wastes weeks.

By the end you will know what each tool fixes, a short table to compare them, when to use two together, and a small Python helper that turns six yes/no questions into a recommendation.

Key words

Prompt
The text you send to the model: your instructions, any examples, and the question.
RAG (retrieval-augmented generation)
Before the model answers, your program looks up useful text and puts it in the prompt. The model answers with that text in front of it.
Fine-tuning
Giving a model extra training on your own examples, so its behaviour changes.
Knowledge
Facts the model needs: prices, rules, names, dates.
Behaviour
How the model answers: tone, layout, format, and how steadily it follows your rules.
Citation
A pointer to where an answer came from, so a person can check it.
Example (training example)
One sample of a good input and the answer you want back.

Two different problems

Most mix-ups come from one mistake: treating all wrong answers as one problem. They are two.

  • A knowledge problem

    The model does not know the fact. Maybe it is new, private to your company, or just not in what the model learned.

  • A behaviour problem

    The model knows enough but answers in the wrong shape: wrong tone, wrong format, or rules followed only some of the time.

  • Both at once

    It happens often. The model lacks the facts and also answers in a messy way. Then you may need two fixes.

OpenAI's guide on optimizing accuracy makes the same split. It says you need to optimize for context when the model lacks contextual knowledge because it was not in its training set, its knowledge is out of date, or it needs proprietary information. It says you need to optimize the model itself when results are inconsistent with incorrect formatting, the tone or style is not correct, or the reasoning is not followed consistently.

Tip

Before you build anything, collect ten real examples where the model got it wrong. Next to each, write why it was wrong: a missing fact, or a bad shape? The reasons point at the tool.

Start with the prompt

The cheapest thing to try is a better prompt. You change some words, run it again, and see the result in minutes. No search system to build, no training run to pay for.

The OpenAI guide agrees on the order of attack, with a warning attached. It says prompt engineering "is typically the best place to start". It also says it is often the only method needed for tasks like summarization, translation and code generation.

But the same guide warns against reading this as a fixed ladder. It says many guides paint optimization as a simple linear flow, from prompt to RAG to fine-tuning, and that this is often not the case: they are levers that solve different things, so you need to pull the right lever.

So the plan is: start with a plain prompt, test it, look at why answers fail, then choose the lever that matches the failure.

  1. 1

    Write a simple prompt

    Say what you want, in plain words, with the answer you expect in mind.

  2. 2

    Add examples

    Show two or three good answers inside the prompt. This is called few-shot prompting, and it often fixes tone and format.

  3. 3

    Test on real cases

    Run it on the ten wrong answers you collected. Keep the score somewhere, even a plain list.

  4. 4

    Name the failure

    For each miss, decide: missing fact, or bad behaviour?

  5. 5

    Pull the matching lever

    Missing facts point to RAG. Behaviour that prompts cannot hold points to fine-tuning.

The Prompt Engineering lesson covers these first steps in detail.

When RAG is the right fit

RAG is the folder of papers on the assistant's desk. The model does not need to remember the facts. It needs to read them when asked.

The Wikipedia page on RAG describes it as a technique that enables models "to retrieve and incorporate new information from external data sources". It says this lets a chatbot reach internal company data or answer from authoritative sources.

Three situations point to RAG.

  • Knowledge that changes

    Prices, opening hours, policies, product lists. With RAG you edit the document. The same page quotes Ars Technica: when new information appears, you do not retrain the model, you update its outside knowledge base. Wikipedia adds that RAG reduces the need to retrain a model with new data, saving computing and money.

  • Answers that must be checked

    Wikipedia says RAG lets a model include sources in its answers, so users can verify them.

  • Private information

    The facts live in your own files and were never in the model's training. Retrieval brings them in at question time. The OpenAI guide lists needing proprietary information as a reason to optimize context.

RAG has limits too. Wikipedia notes that it does not stop a model from making things up: the model can still hallucinate around the source material. Retrieval can also fetch the wrong text or too much noise, which the OpenAI guide says can drown out the real information.

When fine-tuning is the right fit

Fine-tuning is the training course. Wikipedia describes it as adapting a model trained for one task to perform a different, usually more specific, task, by applying additional training to the model's parameters, its stored numbers.

It changes how the model behaves. That makes it a good match when your trouble is shape, not facts.

  • A fixed style

    Your brand voice, a formal tone, a set sign-off. The OpenAI guide lists a wrong tone or style among the reasons to optimize the model.

  • A strict format

    Always the same fields, the same layout. The same guide names inconsistent results and incorrect formatting.

  • A narrow, repeated task

    The same small job done thousands of times, such as sorting messages into five labels.

A fourth benefit is shorter prompts. The OpenAI guide says fine-tuning can minimize the tokens used for prompt engineering, because you replace instructions and few-shot examples with many training examples. The Tokens lesson explains why length is counted.

The data matters more than the amount. The guide says the quality of training data is more important than quantity. It suggests starting with 50 or more examples, evaluating, and growing the set only if you still miss your target and the cause is behaviour, not context.

Fine-tuning is a weaker tool for adding new facts. The next section shows a study on exactly this.

What the research found

A paper called "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" put the two tools head to head on facts. Its abstract says: "We evaluate both approaches on a variety of knowledge-intensive tasks across different topics."

The abstract reports the result in one line: "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge." Here "it" means unsupervised fine-tuning, the plain kind where the model just keeps reading text.

It also says: "LLMs struggle to learn new factual information through unsupervised fine-tuning." And it adds a hint: exposing the model to numerous variations of the same fact during training could alleviate this problem.

Tip

Read the limits. This paper tested one kind of fine-tuning on knowledge tasks. It does not say fine-tuning is poor at style or format, and the abstract makes no claim about that. Use it as a reason to prefer RAG for facts, not as a verdict on all training.

Quick check

Your assistant must answer from a price list that changes every week. What is the first tool to try?

Side by side

Here is the comparison in one place. The columns are general rules of thumb from the sources above and plain reasoning, so check them against your own case. Prompting is included because it comes first.

Text
Question        Prompting        RAG                    Fine-tuning
--------------  ---------------  ---------------------  ----------------------
Freshness       Edit the prompt  Edit the documents     Needs more training
Cost to change  Very low         Low: update the index  Higher: a new run
Data needed     A few examples   Your documents         Good examples (50+)
Citations       No built-in way  Yes: show the source   No built-in way
Latency         Short prompt     An extra search step   Shorter prompt, no step
Privacy         Text goes out    Chosen text goes out   Examples go into training
Skills needed   Writing, testing Search, chunking       Clean data, evaluation

Three rows deserve a closer look.

  • Latency

    RAG adds a lookup before the model answers, and longer prompts take longer to handle. A fine-tuned model can use a shorter prompt. If every second counts, keep retrieved text short.

  • Privacy

    With RAG, the text you retrieve is sent along with each question. With fine-tuning, your examples are used to train a model. In both cases ask who holds the data and whether you may use it that way. This is our reasoning, so confirm it with whoever runs your model.

  • Skills needed

    RAG asks for search and document work, which the Chunking and Vector Stores lessons cover. Fine-tuning asks for tidy examples and a way to measure results, both easy to underestimate.

Using both together

Sometimes one tool is not enough. The OpenAI guide says the techniques stack on top of each other. If early tests show problems with both context and behaviour, you may end up with fine-tuning plus RAG in production, and the two balance each other's weaknesses.

Its example flow runs like this: begin with a prompt and evaluate it, add fixed few-shot examples, add a retrieval step so the examples come in based on the question, prepare 50 or more examples and fine-tune, then tune the retrieval and add a check for made-up facts.

One more tip from the guide is worth keeping. If you fine-tune a model that will run inside a RAG app, train it on examples that include the retrieved text, so it learns the same situation it will face for real.

The guide also shows that more is not always better. In its test of correcting sentences, a score out of 100 went from 62 with no examples to 70 with three examples in the prompt. One model fine-tuned on 1000 examples scored 78, and a second model tuned the same way scored 87. Adding RAG to that second model dropped it to 83. The guide's reading: this was a behaviour problem, so extra retrieved text mostly added noise.

One extra tool can help or hurt. Only your own examples will tell you which.

A small decision helper

Let's turn the lesson into a tool you can run. The helper below asks six yes/no questions about a project and prints a pick with reasons. It uses plain Python and does not call any model, so there is no stand-in needed and nothing to pay for.

The rules are a first pass, written by us. The 500+ examples cut-off is our own made-up round number, not a figure from the sources (the OpenAI guide suggests starting a test with 50 or more). Treat the output as a conversation starter, not a verdict.

Python
QUESTIONS = [
    ("changes", "Does the knowledge change often?"),
    ("cites", "Do answers need to point to a source?"),
    ("style", "Do you need one fixed style or format?"),
    ("examples", "Do you have 500+ good examples?"),
    ("latency", "Is speed strict (every extra second hurts)?"),
    ("budget", "Is the budget small?"),
]


def decide(project):
    a = project["answers"]
    reasons = []
    need_rag = a["changes"] or a["cites"]
    need_tune = a["style"] and a["examples"]

    if a["changes"]:
        reasons.append("Facts change, and editing a document is cheaper than retraining.")
    if a["cites"]:
        reasons.append("Only retrieved text can be shown as a source.")
    if a["style"] and a["examples"]:
        reasons.append("A fixed style plus 500+ examples is what fine-tuning is for.")
    if a["style"] and not a["examples"]:
        reasons.append("You want a fixed style, but there are too few examples to train on yet.")
    if a["latency"] and need_rag:
        reasons.append("Speed is strict: keep the retrieved text short.")
    if a["budget"] and need_tune:
        reasons.append("The budget is small: try prompts first and fine-tune only if they fall short.")

    if need_rag and need_tune and not a["budget"]:
        pick = "RAG + fine-tuning"
    elif need_rag:
        pick = "RAG (with a good prompt)"
    elif need_tune and not a["budget"]:
        pick = "Fine-tuning"
    else:
        pick = "Better prompting first"
        if not reasons:
            reasons.append("Nothing here asks for more than a good prompt.")
    return pick, reasons


projects = [
    {"name": "Shop help bot",
     "answers": dict(changes=True, cites=True, style=False, examples=False, latency=False, budget=True)},
    {"name": "Brand voice writer",
     "answers": dict(changes=False, cites=False, style=True, examples=False, latency=False, budget=True)},
    {"name": "Contract clause helper",
     "answers": dict(changes=True, cites=True, style=True, examples=True, latency=True, budget=False)},
]

for p in projects:
    yes = [q for key, q in QUESTIONS if p["answers"][key]]
    pick, reasons = decide(p)
    print(p["name"])
    print("  yes to:", len(yes), "of", len(QUESTIONS))
    print("  pick:", pick)
    for r in reasons:
        print("  -", r)

Output

Shop help bot
  yes to: 3 of 6
  pick: RAG (with a good prompt)
  - Facts change, and editing a document is cheaper than retraining.
  - Only retrieved text can be shown as a source.
Brand voice writer
  yes to: 2 of 6
  pick: Better prompting first
  - You want a fixed style, but there are too few examples to train on yet.
Contract clause helper
  yes to: 5 of 6
  pick: RAG + fine-tuning
  - Facts change, and editing a document is cheaper than retraining.
  - Only retrieved text can be shown as a source.
  - A fixed style plus 500+ examples is what fine-tuning is for.
  - Speed is strict: keep the retrieved text short.

Walk through the three runs. The shop bot says yes to 3 of 6 questions: its facts change and it must cite sources, so it gets RAG. Its small budget does not matter, because RAG needs no training.

The brand voice writer says yes to 2 of 6. It wants a fixed style but has too few examples, so the helper says to improve the prompt first. Writing three good examples inside the prompt costs almost nothing, and collecting more examples can start now.

The contract helper says yes to 5 of 6. Its facts change, it needs sources, it wants a fixed style, it has the examples and the budget. So it lands on both. The latency question adds a warning to keep retrieved text short.

Tip

Change one answer and run it again. Flip "small budget" for the contract helper and watch the pick change. The helper is small enough to bend to your own rules.

Recap and what to do on Monday

  1. 1

    Prompt first

    It is the cheapest to try, and it shows you what a good answer is.

  2. 2

    Sort the failures

    Missing or changing facts, or a wrong shape of answer?

  3. 3

    Facts: RAG

    Add retrieval, show sources, and test the search as well as the answer.

  4. 4

    Behaviour: fine-tune

    Use a small, clean set of examples, and keep your test set apart.

  5. 5

    Both, if needed

    Add the second tool only after a test shows the first is not enough.

Quick check

Your tone is right, but the bot invents return policies. What is the likeliest fix?

Next: agent use cases, where letting a model act is worth it.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.