esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.16

Model selection: choosing between models

Many models could answer your question. Learn to pick one fairly: start with a baseline, compare on the validation set, and watch for models that memorise.

13 min readFree

Why you have to choose

Say you want a model that guesses the price of a flat. You could use a very simple one, a very flexible one, or ten in between. All can train on the same data. Each gives different answers. Which do you ship?

That choice has a name. Wikipedia says model selection is 'the task of selecting a model from among various candidates on the basis of performance criterion to choose the best one'. Think of hiring. You have several candidates and one job. You do not pick the one who talks best. You give them all the same task, mark it the same way, and compare the marks.

Key words

Candidate
One model you are thinking of using. Model selection compares candidates.
Baseline
A very simple model you must beat. If a clever model cannot beat it, the clever model is not worth having.
Metric
One number that scores a model, such as the average size of its mistakes. Later lessons cover metrics in depth.
Validation set
Data kept back while you build. You use it to compare candidates.
Test set
Data locked away until the very end. It gives one honest final score.
Underfitting
The model is too simple to catch the pattern. It does badly everywhere.
Overfitting
The model memorised its training data, so it does badly on new data.

Step one: build a baseline

Before any clever model, build the dullest one you can. It always gives the same answer. For a number, it always guesses the average. For a category, it always guesses the most common one.

This sounds silly. It is one of the most useful things you can do. The scikit-learn guide calls it a sanity check: 'When doing supervised learning, a simple sanity check consists of comparing one’s estimator against simple rules of thumb.' Its dummy estimators do exactly this. For categories, 'most_frequent always predicts the most frequent label in the training set'. For numbers, 'mean always predicts the mean of the training targets'. And they are truly dull: 'In all these strategies, the predict method completely ignores the input data!'

Why bother? A score alone means nothing. Is 80 right out of 100 good? Look at ten messages: eight normal, two spam. A model that says 'normal' every time, without reading any, already scores 80%.

Python
from collections import Counter

labels = ["ok", "ok", "ok", "ok", "ok", "ok", "ok", "ok", "spam", "spam"]
most_common = Counter(labels).most_common(1)[0][0]
guesses = [most_common] * len(labels)
correct = sum(1 for g, real in zip(guesses, labels) if g == real)

print("always guess:", most_common)
print("accuracy:", correct / len(labels))

Output

always guess: ok
accuracy: 0.8

That model cannot spot one spam message, yet it looks fine. The baseline shows it. Accuracy here means the share of answers that were right.

Tip

The same guide gives a warning sign: 'when the accuracy of a classifier is too close to random, it probably means that something went wrong'. A model that barely beats the baseline is a signal to check your data, not to celebrate.

Step two: compare fairly

Now you have candidates and a baseline. To compare them you need three things to stay the same.

  • The same data split

    Every model trains on the same training set and is scored on the same validation set. Different data gives different scores, and then you are comparing luck.

  • One agreed metric

    Pick the score before you look at results. For categories it is often accuracy, the share of right answers. For numbers it is often MAE, the mean absolute error: the average size of the mistakes, ignoring whether each was too high or too low.

  • Validation, not training

    Score on data the model did not learn from. A model's score on its own training data tells you how well it remembers, not how well it predicts.

The scikit-learn guide makes the same point about the score you get from your final choice. Once you pick settings by their validation score, 'the validation score is biased and not a good estimate of the generalization any longer'. Generalization means doing well on data the model has never seen. To measure that, the guide says, 'we have to compute the score on another test set'.

  1. 1

    Split the data

    Training, validation and test, as in the supervised learning lesson. Lock the test set away.

  2. 2

    Pick one metric

    Write it down before you train anything.

  3. 3

    Score the baseline

    On the validation set. This is the number to beat.

  4. 4

    Train each candidate

    On the training set only.

  5. 5

    Score each on the validation set

    Same metric, same data. Put the numbers in one table.

  6. 6

    Choose, then test once

    Pick the winner. Only then, one time, score it on the test set.

Too simple, too clever

Why do we not just choose the model that scores best on its training data? Because that model may simply have memorised the answers. Picture two students before an exam. One learned the ideas. The other learned the practice paper word for word. On the practice paper, the second one scores full marks. On the real exam, with new questions, they fall apart.

Google's crash course describes the same thing. Overfitting is 'creating a model that matches (memorizes) the training set so closely that the model fails to make correct predictions on new data'. And 'An overfit model is analogous to an invention that performs well in the lab but is worthless in the real world.'

The scikit-learn example shows the other side too. It fits a wavy curve with models of different flexibility. A straight line 'is not sufficient to fit the training samples. This is called underfitting.' Too much flexibility, and the model 'learns the noise of the training data'. Noise here means the random wobble in real data that has no pattern to learn.

Underfitting

Underfitting is the student who only read the chapter titles. The model is so simple that it misses the pattern. It makes big mistakes on the training data and the new data alike. Our baseline is the extreme case: it ignores the input completely.

Overfitting

Overfitting is the student with the memorised practice paper. The model is so flexible that it copies even the random wobble. Its training mistakes are tiny, and its mistakes on new data are not. Google gives one cause that matters here: 'The model is too complex.' (The other it names is training data that does not represent real life.)

Quick check

A model has almost no mistakes on its training data but many on the validation set. What is the likely problem?

Bias and variance, gently

Statisticians split a model's mistakes into two kinds. Wikipedia gives the short versions. 'The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).' And: 'The variance is an error from sensitivity to small fluctuations in the training set. High variance may result from an algorithm modeling the random noise in the training data (overfitting).'

Picture darts. High bias: every dart lands left of the bullseye, the same way each time. Consistent and wrong. High variance: darts all over the board. You want them close together and near the middle.

The scikit-learn guide says it in model terms. Its too-simple model 'can at best provide only a poor fit' because it is too simple, which is high bias. Its over-flexible one 'approximates the training data perfectly but does not fit the true function very well', because it is 'very sensitive to varying training data'. That is high variance.

Now the trade-off. Wikipedia describes it: as a model gets more flexible, it 'can better fit a training data set', so its bias goes down. But a flexible model will tend to change a lot each time you give it a new batch of training data, so its variance goes up. The bias–variance dilemma is 'the conflict in trying to simultaneously minimize these two sources of error'. You cannot drive both to zero by turning one dial. You look for a balance.

The complexity curve

You can see all this in one picture. Make a list of models of rising complexity. For each, measure the error on the training set and on the validation set. Two things happen.

  • Training error keeps falling

    A more flexible model can always fit the data it saw a little better. It can even reach zero.

  • Validation error falls, then turns back up

    At first extra flexibility helps. Then the model starts to copy the noise, and its mistakes on new data grow.

  • The best model sits at the dip

    That lowest point on the validation line is the balance between too simple and too clever.

Google's crash course describes the same shape in terms of loss, a number for how wrong the model is. On its generalization curve, 'after a certain number of iterations, loss declines or holds steady (converges) for the training set, but increases for the validation set. This suggests overfitting.' The scikit-learn guide offers a tool for it called a validation curve, which plots the training score and the validation score for different values of one setting, 'to find out whether the estimator is overfitting or underfitting for some hyperparameter values'. A hyperparameter is a setting you choose before training, like the k below.

A worked example

Let us build the curve ourselves, using only plain Python. Our model is the nearest-neighbour idea from the supervised learning lesson. To guess the answer for a new point, find the k closest known points and average their answers. Here k is the setting we choose. A small k trusts the nearest few points. A large k averages over many.

Wikipedia ties k to our bias and variance words: 'In k-nearest neighbor models, a high value of k leads to high bias and low variance.' So k = 1 should be the flexible, memorising end, and k = 25 the stiff, simple end. Let us check.

The data is made up. The answer rises with x, with a repeating wave and some random noise. A fixed seed means you get the same numbers as we did. We make 40 training points and 20 validation points, build the baseline, then try six values of k. The metric is MAE, the average size of the mistakes. Lower is better.

Python
import random

random.seed(7)

def make_rows(n):
    rows = []
    for _ in range(n):
        x = random.uniform(0, 10)
        y = 3 * x + 8 * (x % 3) + random.gauss(0, 4)
        rows.append((x, y))
    return rows

train = make_rows(40)
valid = make_rows(20)

def predict(rows, x, k):
    nearest = sorted(rows, key=lambda r: abs(r[0] - x))[:k]
    return sum(y for _, y in nearest) / k

def mae(rows, guesses):
    return sum(abs(y - g) for (_, y), g in zip(rows, guesses)) / len(rows)

mean_y = sum(y for _, y in train) / len(train)
print("baseline (always the mean):")
print("  train MAE", round(mae(train, [mean_y] * len(train)), 2),
      " valid MAE", round(mae(valid, [mean_y] * len(valid)), 2))

print("k-nearest neighbours:")
for k in [1, 3, 5, 9, 15, 25]:
    t = mae(train, [predict(train, x, k) for x, _ in train])
    v = mae(valid, [predict(train, x, k) for x, _ in valid])
    print(f"  k={k:<2}  train MAE {t:5.2f}  valid MAE {v:5.2f}")

Output

baseline (always the mean):
  train MAE 8.82  valid MAE 12.91
k-nearest neighbours:
  k=1   train MAE  0.00  valid MAE  5.77
  k=3   train MAE  3.13  valid MAE  4.95
  k=5   train MAE  3.67  valid MAE  5.43
  k=9   train MAE  4.77  valid MAE  6.46
  k=15  train MAE  5.95  valid MAE  7.80
  k=25  train MAE  7.08  valid MAE  9.69

Read the table like a detective.

  • The baseline

    Always guessing the mean is off by 12.91 on average on the validation set. Any model worth keeping must do better than this.

  • k = 1 memorised

    Its training error is exactly 0.00, because each training point is its own nearest neighbour. Yet on the validation set it is off by 5.77. That gap is overfitting.

  • k = 25 is too stiff

    Both errors are high and close to each other. It averages so many points that it flattens the wave. That is underfitting.

The best validation score is 4.95, at k = 3. As k grows from 1, the training error climbs every time, from 0.00 to 7.08, because the model is getting simpler. The validation error is different. It falls from 5.77 to 4.95, then rises again to 9.69. That is the turn-back-up shape of the curve. So we choose k = 3.

Tip

Do not trust the exact numbers too much. We used only 20 validation points. A different seed or a different split could move the winner to k = 5. A way to get steadier scores is called cross-validation, and it has its own lesson soon.

Quick check

In the table, why is the training error exactly 0.00 when k = 1?

When scores are close

Sometimes two models score almost the same. Wikipedia gives the usual tie-breaker: 'Given candidate models of similar predictive or explanatory power, the simplest model is most likely to be the best choice (Occam's razor).'

Occam's razor is a rule of thumb: when two explanations work equally well, prefer the simpler one. It is only a tie-breaker. It does not say simple always wins: our baseline lost by a mile.

Real life has more than one score

The best score is not the only thing in a real project. After the numbers, ask the plain questions.

  • Speed

    How fast must it answer? Behind a button people tap, it must be quick.

  • Size

    Where will it run, and how much room does it have?

  • Explainability

    Can you tell a person why it said what it said? For some jobs that matters as much as the score.

And think about how much data you have. We saw that a larger training set tends to decrease variance. So with little data, a very flexible model is a risk. With lots of data, you can afford more flexibility.

The golden rule: never choose on the test set

One last warning, and it is the most important. The test set is for one job: a single honest score for the model you already chose. If you use it to compare candidates, it stops being honest.

Google's crash course sets out the idea: while developing a model, 'your test set serves as a proxy for real-world data'. A proxy is a stand-in. It only works while it stays unseen. So use the validation set for every choice you make, and look at the test set once, at the end.

What to remember

Start with a baseline, because a score means little without something to compare it to. Compare every candidate on the same validation set with one metric you chose first. Too simple a model underfits and too flexible a model overfits, and the best one sits at the dip of the validation curve. When scores tie, prefer the simpler model. Weigh speed, size, explainability and how much data you have. And never choose using the test set.

Cross-validation, a steadier way to compare, gets its own lesson.

Next: tuning a model's settings.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.