esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.01

Model evaluation: is it any good?

A model that sounds clever may still be wrong. Learn to measure it fairly, pick a score that fits the stakes, and beat a lazy baseline.

13 min readFree

How do you know a model is any good?

The last lesson ended with a question: how do you tell whether a model is any good? Here is the answer, and it starts with a small story. A friend says, "I built a model that is 96% right." It sounds great. But right about what? Measured on which data? And would you trust it with your health?

Model evaluation is the habit of answering those questions. It means measuring how well a model does its job, with a number, on examples it has not seen, using a score that fits the problem. A model is only as good as the evidence that it works.

Think of a driving test. Nobody trusts a driver because they say they are careful. The examiner watches them drive a route they have not practised, and marks specific mistakes. Evaluation is the driving test for a model.

Key words

Evaluation
Measuring how well a model does its job.
Metric
The number you use to score a model, such as accuracy or error. Scikit-learn's guide also calls it a scoring function.
Unseen data
Examples the model never learned from. Only these tell you how it will behave in real life.
Baseline
A very simple guess that a real model must beat to be worth using.
False positive
A false alarm: the model says yes, the truth is no.
False negative
A miss: the model says no, the truth is yes.
Imbalanced data
Data where one answer is far more common than the other, like healthy patients against ill ones.

By the end you will know why we test on fresh data, how the cost of mistakes picks the score, what the common scores are called, and a short checklist for judging any model.

Always test on data it has not seen

Imagine a student who memorises last year's exam, answers included. Give them last year's paper again and they score 100%. Give them a new paper and they may fail. A model can do the same: score very well on data it has already studied, and badly on new data.

That is why we hold some data back. The lesson on Train - Test Data shows how: the model learns from the training set, and we score it on a test set it never touched. We will not repeat the how here. The rule to carry forward is simple: a score on the training data tells you almost nothing about the future.

Tip

If a score looks too good to be true, ask one question first: was it measured on data the model had already seen?

The lesson on Model Selection added a second rule. Do not use the test set to choose between models, or the test score stops being honest. Tuning has the same rule. Keep the test set for one last, final check.

Mistakes cost different amounts

Every model gets some answers wrong. The real question is which wrong answers hurt. Google's Machine Learning Crash Course puts it plainly: which evaluation metrics are most meaningful depends on the specific model and the specific task, the cost of different misclassifications, and whether the dataset is balanced or imbalanced.

Two examples show how different the stakes can be.

  • Spam filter

    A false positive sends a real email to the spam folder. A false negative lets one junk email into the inbox. Both are annoying. Google's course notes it often makes sense to prioritise catching all the spam, or making sure spam-labelled emails really are spam, or a balance of the two.

  • Illness screening

    A false negative tells a sick person they are fine. A false positive sends a healthy person for another check. Google's course says that for disease prediction, correctly finding the positive cases is crucial, and a false negative typically has more serious consequences than a false positive.

  • Your own project

    Ask the same thing: which mistake would a real person complain about louder? Write the answer down before you pick a score. The score should follow the cost, never the other way round.

Scikit-learn's guide gives the same advice from another angle. If the scoring function is given, for example in a Kaggle competition or a business setting, use that one. If you are free to choose, start by considering the ultimate goal and application of the prediction.

A map of the common scores

There are many metrics. You do not need to learn them all today. Here is a map: each name once, with one line on what it tells you. Later lessons give each its own space.

For classification, where the answer is a group:

  • Accuracy

    The share of all answers that were correct. Simple, but it can mislead, as we will see.

  • Precision and recall

    Precision: of everything the model flagged, how much was truly positive. Recall: of everything truly positive, how much the model found.

  • F1

    One number that blends precision and recall. Scikit-learn and Wikipedia both describe it as their harmonic mean.

  • ROC-AUC

    Wikipedia describes the area under the ROC curve (AUC) as a useful single measure when a model's threshold can be adjusted. A threshold is the cut-off score that turns a model's score into a yes or no.

  • Log loss

    Scikit-learn's guide says log loss is defined on probability estimates, so it can score a classifier's probabilities, not just its final labels.

  • Confusion matrix

    Not a score but a table of hits and misses. Most of the scores above are built from its four counts. The Classification lesson introduced it.

For regression, where the answer is a number, the main scores are mean absolute error (MAE), mean squared error (MSE) and root mean squared error (RMSE). Scikit-learn describes RMSE as the square root of the MSE. The Regression lesson explains all three, so we will not repeat them.

For clustering, where nobody gave the right answers, scores like the silhouette coefficient look only at the groups themselves. Scikit-learn says a higher Silhouette Coefficient relates to a model with better defined clusters. The Clustering lesson covers it.

Tip

In scikit-learn's scoring convention, higher values are always better. Error scores are simply flipped to negative so that the rule holds. You will notice the minus signs.

A baseline: the lazy guess to beat

A score on its own means little. "93% accurate" could be brilliant or useless. You need something to compare it with: a baseline.

A baseline is the dumbest sensible answer. For a yes or no question, always say the most common answer. For a number, always say the average. The Prediction lesson shows this idea, and scikit-learn builds it in: its guide says dummy estimators are useful to get a baseline value of those metrics for random predictions. Its DummyClassifier can always predict the most frequent label.

If your clever model cannot beat the lazy guess, it has learned nothing useful. If it only barely beats it, the extra complexity may not be worth it.

Quick check

A model gets 90% accuracy. The most common answer in the data is right 90% of the time. What does that tell you?

Worked example: when accuracy misleads

Let us try it. We invent 1,000 patients and a rare illness. The data is made up, and a fixed random seed makes it repeatable, so you get the same numbers if you run the code.

First the data, and the laziest baseline possible: always say "healthy".

Python
import random

random.seed(42)

# 1000 made-up patients: 1 means "has the illness", 0 means "healthy".
truth = [1 if random.random() < 0.05 else 0 for _ in range(1000)]

# The baseline never thinks hard: it always says "healthy".
baseline = [0] * len(truth)

def accuracy(truth, guess):
    right = sum(1 for t, g in zip(truth, guess) if t == g)
    return right / len(truth)

print("patients:", len(truth))
print("really ill:", sum(truth))
print("baseline accuracy:", accuracy(truth, baseline))

Output

patients: 1000
really ill: 41
baseline accuracy: 0.959

A baseline that never finds a single sick person is 95.9% accurate. The data is imbalanced: only 41 of the 1,000 patients are ill. Google's course makes the same point with a rounder case: with a rare class at 1% of the data, a model that predicts negative all the time would score 99% on accuracy, despite being useless.

Now a made-up classifier. It spots about 70% of the ill patients, and raises a false alarm for about 4% of the healthy ones. It is not perfect, but it does try to find the sick.

Python
import random

random.seed(42)
truth = [1 if random.random() < 0.05 else 0 for _ in range(1000)]
baseline = [0] * len(truth)

# A made-up classifier: it spots 70% of ill patients,
# but it also raises a false alarm for 4% of healthy ones.
random.seed(7)
model = []
for t in truth:
    if t == 1:
        model.append(1 if random.random() < 0.70 else 0)
    else:
        model.append(1 if random.random() < 0.04 else 0)

def accuracy(truth, guess):
    right = sum(1 for t, g in zip(truth, guess) if t == g)
    return right / len(truth)

print("baseline accuracy:", accuracy(truth, baseline))
print("model accuracy:   ", accuracy(truth, model))

Output

baseline accuracy: 0.959
model accuracy:    0.944

By accuracy, the model (94.4%) loses to the baseline (95.9%). If accuracy were your only score, you would throw away the only one of the two that ever finds a sick person. Next, count the hits and misses.

Python
import random

random.seed(42)
truth = [1 if random.random() < 0.05 else 0 for _ in range(1000)]
baseline = [0] * len(truth)
random.seed(7)
model = []
for t in truth:
    if t == 1:
        model.append(1 if random.random() < 0.70 else 0)
    else:
        model.append(1 if random.random() < 0.04 else 0)

def counts(truth, guess):
    tp = sum(1 for t, g in zip(truth, guess) if t == 1 and g == 1)
    fn = sum(1 for t, g in zip(truth, guess) if t == 1 and g == 0)
    fp = sum(1 for t, g in zip(truth, guess) if t == 0 and g == 1)
    tn = sum(1 for t, g in zip(truth, guess) if t == 0 and g == 0)
    return tp, fn, fp, tn

for name, guess in [("baseline", baseline), ("model", model)]:
    tp, fn, fp, tn = counts(truth, guess)
    print(name)
    print("  caught", tp, "of", tp + fn, "ill patients")
    print("  false alarms:", fp)
    print("  recall:", round(tp / (tp + fn), 2))
    if tp + fp == 0:
        print("  precision: none, it never says ill")
    else:
        print("  precision:", round(tp / (tp + fp), 2))

Output

baseline
  caught 0 of 41 ill patients
  false alarms: 0
  recall: 0.0
  precision: none, it never says ill
model
  caught 31 of 41 ill patients
  false alarms: 46
  recall: 0.76
  precision: 0.4

The picture changes. The baseline caught 0 of 41 sick patients, so its recall is 0. The model caught 31 of 41, a recall of 0.76. The price is 46 false alarms, so its precision is 0.4: of every ten patients it flags, about four are really ill.

The baseline's precision can't be worked out because it never says "ill", and dividing by zero is meaningless. Google's course notes the same thing: a model that never predicts positive has no true or false positives, so its precision calculation has zero in the denominator.

So which model is better? That depends on the cost, as we said earlier. If a missed illness is far worse than a false alarm, the model wins easily. If every false alarm triggers an expensive, risky test, you might want fewer of them. Accuracy cannot tell you any of this. Recall and precision can.

Tip

Google's course also warns that precision and recall often pull in opposite directions: improving one tends to worsen the other. Choosing a metric is choosing which trade you prefer.

Blending two scores into one

Sometimes you want a single number, for example to compare many models in a table. F1 does that. It is the harmonic mean of precision and recall. In words: multiply them, double it, and divide by their sum.

F1=2⋅precision⋅recallprecision+recallF_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}

A harmonic mean is a kind of average that is pulled down by whichever number is smaller. So one great score cannot hide one terrible score. Here are our model's numbers.

Python
precision = 31 / (31 + 46)
recall = 31 / (31 + 10)

# F1 is the harmonic mean of precision and recall.
f1 = 2 * precision * recall / (precision + recall)
average = (precision + recall) / 2

print("precision:", round(precision, 2))
print("recall:", round(recall, 2))
print("plain average:", round(average, 2))
print("F1:", round(f1, 2))

Output

precision: 0.4
recall: 0.76
plain average: 0.58
F1: 0.53

The plain average is 0.58, but F1 is 0.53. It sits lower, closer to the weaker score, which is exactly the point.

Getting a fairer score from small data

A single test set has a weakness. If it is small, or you happen to cut it badly, the score can jump around by luck. Smarter splitting helps: instead of one test set, you test on several different slices of the data and look at all the scores.

Two names to remember for later: k-fold cross-validation and leave-one-out. Each has its own lesson coming up. For now, know that they exist because one split can be unlucky.

Quick check

A model scores 99% on the very data it learned from, but 70% on a fresh test set. Which number should you report?

A checklist for judging any model

You can now follow the same steps for any model, whether it sorts emails, predicts prices or groups customers. Keep this list near you.

  1. 1

    Say what a mistake costs

    Name the worst mistake in plain words: a missed illness, a lost email, a wrong price. This picks the score.

  2. 2

    Pick the metric

    Match it to the task: classification, regression or clustering. If someone gives you the score to use, use theirs.

  3. 3

    Hold data back

    Keep a test set the model never sees. Do not touch it while you build or tune.

  4. 4

    Build a baseline

    Score the lazy guess first: the most common answer, or the average.

  5. 5

    Score the model on unseen data

    Report the number from the test set, never from the training data.

  6. 6

    Look past one number

    Count the hits and misses. Check recall, precision or error size, not only accuracy.

  7. 7

    Compare and decide

    Does the model beat the baseline by enough, on the mistakes that matter? If yes, you have evidence. If not, go back to the data or the model.

Tip

Write the metric and the baseline score at the top of your notes before you train anything. It stops you moving the goalposts later.

What to remember

  • Fair test

    Score a model on data it has not seen. A good score on training data proves little.

  • Right score

    The best metric depends on what each kind of mistake costs, and on whether the data is balanced. Accuracy alone can mislead, as the 95.9% baseline showed.

  • Compare

    Always beat a baseline, and look at hits and misses, not one number.

Next: accuracy, the simplest score and its traps.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.