esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.24

Evaluation Metrics

The few numbers an AI engineer reports: retrieval scores, overlap scores and their limits, RAG scores, and the product numbers users feel.

13 min readFree

Numbers that sum up quality

The last lesson, Regression Testing, ended with a promise: next, evaluation metrics, the numbers that sum up quality. Here they are.

Think of a school report card. A grade fits on one line, and you can compare it with last term. A metric is a grade for your AI feature. It turns "feels good" into a number you can track and improve.

Most metrics are not for you. This lesson picks the few that matter for a product, and shows where each one can fool you. You will also run small programs that compute them.

Key words

Metric
A number that sums up how well something works, such as 0.75 or 83%.
Reference
The right answer that a person wrote down ahead of time. Your system's output is compared with it.
Relevant
Useful for the question. A relevant document is one that helps answer it.
Rank
A place in a list. Rank 1 is the top result.
Unigram
A single word.
Faithful
Sticking to the facts you were given, and not making things up.
Retrieval
Finding the right pieces of text for a question, before a model writes an answer.

Pick few numbers, tied to users

A dashboard with twenty numbers is one nobody reads. Start from what your users need, then pick one number per need.

  • Does it do the job?

    A task metric. For a classifier, that is accuracy, precision, recall and F1. For a whole product, it is task success.

  • Does it find the right material?

    A retrieval metric. Used when the model reads documents first: recall@k, MRR and nDCG.

  • Is it worth the money?

    A product metric. Time saved, thumbs-up rate and cost per success.

Tip

Write the user's need in one sentence before you pick a metric. "People find the refund rule fast" points to retrieval. "People get an answer they can trust" points to faithfulness. If you cannot link a number to a sentence like that, drop the number.

Task metrics: did it do the job?

Many AI features sort things: spam or not spam, urgent or not urgent, happy or angry. For these, the standard numbers are accuracy, precision, recall and F1. Accuracy is the share of answers that were right. Precision asks how many of the "yes" answers were real yeses. Recall asks how many of the real yeses were caught. F1 blends the last two into one number.

The Machine Learning Foundations course teaches these in depth, in its lessons on accuracy, precision, recall and the confusion matrix. You do not need to learn them again here. The AI-engineering angle is short: report precision and recall next to accuracy.

Retrieval metrics: did it find the right pages?

Many AI products first search a pile of documents, then let a model answer from what was found. If the search misses the right page, the model cannot save you. So you score the search by itself.

Picture a librarian who hands you a stack of books, best guess first. Is the right book in the stack? How near the top is it? Each question has a metric.

recall@k

Recall@k asks: of all the relevant documents, what share showed up in the top k results? If two documents are relevant and one is in the top 3, recall@3 is 0.5. It tells you if the right material even reached the model.

MRR

Mean reciprocal rank looks at where the first right result sits. The Wikipedia page on it explains that the reciprocal rank of a query response is "the multiplicative inverse of the rank of the first correct answer: 1 for first place, 1⁄2 for second place, 1⁄3 for third place and so on". The mean reciprocal rank is the average of those numbers over a sample of queries.

MRR=1∣Q∣∑i=1∣Q∣1ranki\mathrm{MRR} = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{\mathrm{rank}_i}

In words: for each query, find the position of the first right result and flip it upside down (1 divided by the position). Then average over all queries. The page adds a catch: further relevant answers are ignored. If no result is right, that query scores 0.

nDCG

Normalized discounted cumulative gain handles a list with several right results. The Wikipedia page says DCG "sums the usefulness, or gain, of the results discounted by their position in the result list".

DCGp=∑i=1prelilog⁡2(i+1)nDCGp=DCGpIDCGp\mathrm{DCG}_p = \sum_{i=1}^{p} \frac{\mathrm{rel}_i}{\log_2(i+1)} \qquad \mathrm{nDCG}_p = \frac{\mathrm{DCG}_p}{\mathrm{IDCG}_p}

In words: each result earns its relevance score, but divided by a number that grows as you go down the list. So a good page at rank 1 counts fully, and the same page at rank 3 counts half. The ideal DCG (IDCG) is the best score possible, if the results were in perfect order. Dividing by it gives a number between 0 and 1.

Run it: three retrieval scores

Here are four made-up searches. For each one, the code has the ranked list the search returned and the set of ids that were truly relevant. Every relevant document counts as 1 and every other as 0. It is plain Python with nothing to install.

  1. 1

    Recall@3

    Take the first three results and count how many are relevant, then divide by how many relevant ones exist.

  2. 2

    Reciprocal rank

    Find the first relevant result, and give 1 divided by its position. No relevant result gives 0.

  3. 3

    nDCG@3

    Add 1 divided by log2(position + 1) for each relevant result in the top 3, then divide by the same sum for a perfect order.

Python
import math

# Each query: what the search returned (best first) and which ids are truly relevant.
queries = [
    ("refund time", ["d3", "d7", "d1", "d9", "d2"], {"d3", "d1"}),
    ("reset password", ["d5", "d2", "d8", "d4", "d6"], {"d8"}),
    ("delete account", ["d6", "d4", "d9", "d1", "d3"], {"d2", "d7"}),
    ("change plan", ["d2", "d9", "d7", "d3", "d5"], {"d7", "d2", "d5"}),
]
K = 3

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & relevant) / len(relevant)

def reciprocal_rank(ranked, relevant):
    for position, doc in enumerate(ranked, start=1):
        if doc in relevant:
            return 1 / position
    return 0.0

def ndcg_at_k(ranked, relevant, k):
    dcg = sum(1 / math.log2(i + 1) for i, d in enumerate(ranked[:k], start=1) if d in relevant)
    best = min(len(relevant), k)
    ideal = sum(1 / math.log2(i + 1) for i in range(1, best + 1))
    return dcg / ideal

print(f"{'query':<16}{'recall@3':>9}{'RR':>7}{'nDCG@3':>8}")
totals = [0.0, 0.0, 0.0]
for name, ranked, relevant in queries:
    row = [recall_at_k(ranked, relevant, K), reciprocal_rank(ranked, relevant), ndcg_at_k(ranked, relevant, K)]
    totals = [t + r for t, r in zip(totals, row)]
    print(f"{name:<16}{row[0]:>9.2f}{row[1]:>7.2f}{row[2]:>8.2f}")
n = len(queries)
print(f"{'average':<16}{totals[0] / n:>9.2f}{totals[1] / n:>7.2f}{totals[2] / n:>8.2f}")

Output

query            recall@3     RR  nDCG@3
refund time          1.00   1.00    0.92
reset password       1.00   0.33    0.50
delete account       0.00   0.00    0.00
change plan          0.67   1.00    0.70
average              0.67   0.58    0.53

Read the table row by row. "Refund time" has both right pages in the top 3, so recall@3 is 1.00. Its nDCG is 0.92, not 1.00, because the second right page is third and a perfect list would put it second.

"Reset password" is the tricky one. Recall@3 is a perfect 1.00, yet the right page sits at rank 3, so its reciprocal rank is 0.33 and its nDCG is 0.50. The right page was found, but late. Only the rank-aware metrics show it.

"Delete account" scores 0 everywhere: the search missed. The average row shows 0.67 for recall@3, 0.58 for the RR column and 0.53 for nDCG@3. The 0.58 is the MRR for these four queries. Averages hide a miss, so also look at the zero rows.

Tip

Report a retrieval metric with its k, like recall@3. If the model only reads the top 3 results, recall@10 flatters you.

Text overlap metrics and their limits

Say a model writes a summary or a translation. One old idea for scoring it: compare it with a human-written reference and count shared words. Two well-known metrics do this.

  • BLEU

    Made for machine translation. Wikipedia says "Intelligibility or grammatical correctness are not taken into account".

  • ROUGE

    Made for summaries and translation. Wikipedia says it compares an automatically produced summary or translation against a human-produced reference. ROUGE-1 refers to the overlap of unigrams (each word).

  • The shared limit

    Both reward word overlap, not meaning. The BLEU page says there is no guarantee that a higher BLEU score means better translation quality, and that BLEU "penalizes fluent translations that don't precisely hit the reference translations".

That last point matters for chat models, whose answers are fluent almost every time. Fluent does not mean true. Let us see it happen.

Run it: a fluent wrong answer wins

The reference says you can return an item within 30 days for a full refund. Candidate A says the right thing in other words. Candidate B copies the sentence but gets two facts wrong: 3 days, and a partial refund. The code computes ROUGE-1 recall, which is the share of the reference's words that the candidate also has.

Python
from collections import Counter

reference = "you can return the item within 30 days for a full refund"
candidates = {
    "A: right, other words": "items can be sent back in 30 days and you get all your money back",
    "B: wrong, same words": "you can return the item within 3 days for a partial refund",
}

def rouge1_recall(reference, candidate):
    ref = Counter(reference.split())
    cand = Counter(candidate.split())
    overlap = sum(min(count, cand[word]) for word, count in ref.items())
    return overlap / sum(ref.values())

for name, text in candidates.items():
    print(f"{name:<24} ROUGE-1 recall = {rouge1_recall(reference, text):.2f}")

Output

A: right, other words    ROUGE-1 recall = 0.33
B: wrong, same words     ROUGE-1 recall = 0.83

The wrong answer scores 0.83. The right answer scores 0.33. This is why overlap metrics are weak for chat answers. Use them as a quick smell test, not as proof that an answer is good.

Quick check

Which statement about ROUGE-1 is true?

RAG metrics: grounded answers

RAG means retrieval-augmented generation: the model reads the retrieved text and answers from it. There are two parts to judge, the finding and the writing. You met the finding part above. For the rest, a paper called Ragas (arXiv 2309.15217) proposes metrics that work without human-written reference answers.

The paper's abstract says to judge whether retrieval can identify "relevant and focused context passages" and whether the model uses them "in a faithful way". The paper names three scores.

  • Faithfulness

    The answer should be grounded in the given context. The paper says an answer is faithful if the claims it makes can be inferred from the context.

  • Answer relevance

    The answer should address the actual question. The paper penalises answers that are incomplete or that contain redundant information.

  • Context relevance

    The retrieved context should be focused, with as little irrelevant information as possible. In the paper's wording, the context is relevant to the extent that it exclusively contains information needed to answer the question.

The paper measures these by prompting a language model as a judge. For faithfulness, the steps are below.

  1. 1

    Split the answer

    A model breaks the answer into short statements, one claim each.

  2. 2

    Check each statement

    For each statement, a model says whether it can be inferred from the retrieved context: yes or no.

  3. 3

    Divide

    Faithfulness is the supported statements divided by all statements.

F=∣V∣∣S∣F = \frac{|V|}{|S|}

In words: |V| is how many statements were supported, and |S| is how many statements there were. Context relevance works the same way: the number of sentences that help answer the question, divided by all sentences in the context. A low score means the model was handed padding.

We cannot call a real judge here, so the code below uses a stand-in. The fake_judge function looks answers up in a fixed table. It is not a real model, so it runs free. A real judge model is slower, costs money for every token, can give a different answer on each run, and can fail or refuse.

Python
# A stand-in for the judge model, so this runs free. It looks answers up in a fixed table.
# A real judge model is slower, costs money for every token, can answer differently each run,
# and can fail or refuse.
def fake_judge(statement, context):
    verdicts = {
        "Returns are free for 30 days.": "yes",
        "Refunds go back to the original card.": "yes",
        "Returns are free for 90 days.": "no",
        "Shipping is free worldwide.": "no",
    }
    return verdicts[statement]

context = "Returns are free for 30 days. Refunds go to the original card."
statements = [
    "Returns are free for 30 days.",
    "Refunds go back to the original card.",
    "Returns are free for 90 days.",
    "Shipping is free worldwide.",
]
supported = sum(1 for s in statements if fake_judge(s, context) == "yes")
print("supported:", supported, "of", len(statements))
print("faithfulness:", supported / len(statements))

Output

supported: 2 of 4
faithfulness: 0.5

Two of four statements were supported, so faithfulness is 0.5. The two unsupported ones, "90 days" and "free worldwide", are invented facts. Faithfulness catches them.

Tip

A judge model is itself a model, and it can be wrong. Check a handful of its verdicts by hand before you trust the score. The lesson A model as the judge goes deeper.

Product metrics: what users feel

Product metrics score the result for a person. These are the ones a manager or a client will ask about.

  • Task success

    Of the people who tried to do something, how many finished it? Define success in advance.

  • Time saved

    How much faster is the job with the feature than without?

  • Thumbs-up rate

    Of the people who rated an answer, how many pressed thumbs-up? Only some people rate, so it is a hint.

The fourth number joins quality to money: cost per success. Divide what you spent by the number of tasks that worked, not by all tasks. A cheap feature that fails a lot can cost more per success than a pricier one that works.

The numbers below are made up, to show the sum. They are not real prices.

Python
# All numbers are made up, to show the sum.
tasks = 200
solved = 150
total_cost = 12.00
print("task success rate:", solved / tasks)
print("cost per task:", round(total_cost / tasks, 3))
print("cost per success:", round(total_cost / solved, 3))

Output

task success rate: 0.75
cost per task: 0.06
cost per success: 0.08

Out of 200 tasks, 150 worked, so the success rate is 0.75. A task cost 0.06 on average, but a success cost 0.08, because the 50 failures still cost money.

Check yourself

Quick check

A chat answer is fluent but invents a fact. Which metric is built to catch it?

Quick check

Which metric rewards a search for putting the first right result higher up?

Next: eval tools, libraries that run all of this for you.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.