Machine Learning Foundations / Core / 02.16
Model selection: choosing between models
Many models could answer your question. Learn to pick one fairly: start with a baseline, compare on the validation set, and watch for models that memorise.
13 min readFree
Why you have to choose
Say you want a model that guesses the price of a flat. You could use a very simple one, a very flexible one, or ten in between. All can train on the same data. Each gives different answers. Which do you ship?
That choice has a name. Wikipedia says model selection is 'the task of selecting a model from among various candidates on the basis of performance criterion to choose the best one'. Think of hiring. You have several candidates and one job. You do not pick the one who talks best. You give them all the same task, mark it the same way, and compare the marks.
Key words
- Candidate
- One model you are thinking of using. Model selection compares candidates.
- Baseline
- A very simple model you must beat. If a clever model cannot beat it, the clever model is not worth having.
- Metric
- One number that scores a model, such as the average size of its mistakes. Later lessons cover metrics in depth.
- Validation set
- Data kept back while you build. You use it to compare candidates.
- Test set
- Data locked away until the very end. It gives one honest final score.
- Underfitting
- The model is too simple to catch the pattern. It does badly everywhere.
- Overfitting
- The model memorised its training data, so it does badly on new data.
Step one: build a baseline
Before any clever model, build the dullest one you can. It always gives the same answer. For a number, it always guesses the average. For a category, it always guesses the most common one.
This sounds silly. It is one of the most useful things you can do. The scikit-learn guide calls it a sanity check: 'When doing supervised learning, a simple sanity check consists of comparing one’s estimator against simple rules of thumb.' Its dummy estimators do exactly this. For categories, 'most_frequent always predicts the most frequent label in the training set'. For numbers, 'mean always predicts the mean of the training targets'. And they are truly dull: 'In all these strategies, the predict method completely ignores the input data!'
Why bother? A score alone means nothing. Is 80 right out of 100 good? Look at ten messages: eight normal, two spam. A model that says 'normal' every time, without reading any, already scores 80%.
from collections import Counter
labels = ["ok", "ok", "ok", "ok", "ok", "ok", "ok", "ok", "spam", "spam"]
most_common = Counter(labels).most_common(1)[0][0]
guesses = [most_common] * len(labels)
correct = sum(1 for g, real in zip(guesses, labels) if g == real)
print("always guess:", most_common)
print("accuracy:", correct / len(labels))Output
always guess: ok accuracy: 0.8
That model cannot spot one spam message, yet it looks fine. The baseline shows it. Accuracy here means the share of answers that were right.
Tip
The same guide gives a warning sign: 'when the accuracy of a classifier is too close to random, it probably means that something went wrong'. A model that barely beats the baseline is a signal to check your data, not to celebrate.
Step two: compare fairly
Now you have candidates and a baseline. To compare them you need three things to stay the same.
The same data split
Every model trains on the same training set and is scored on the same validation set. Different data gives different scores, and then you are comparing luck.
One agreed metric
Pick the score before you look at results. For categories it is often accuracy, the share of right answers. For numbers it is often MAE, the mean absolute error: the average size of the mistakes, ignoring whether each was too high or too low.
Validation, not training
Score on data the model did not learn from. A model's score on its own training data tells you how well it remembers, not how well it predicts.
The scikit-learn guide makes the same point about the score you get from your final choice. Once you pick settings by their validation score, 'the validation score is biased and not a good estimate of the generalization any longer'. Generalization means doing well on data the model has never seen. To measure that, the guide says, 'we have to compute the score on another test set'.
- 1
Split the data
Training, validation and test, as in the supervised learning lesson. Lock the test set away.
- 2
Pick one metric
Write it down before you train anything.
- 3
Score the baseline
On the validation set. This is the number to beat.
- 4
Train each candidate
On the training set only.
- 5
Score each on the validation set
Same metric, same data. Put the numbers in one table.
- 6
Choose, then test once
Pick the winner. Only then, one time, score it on the test set.
Too simple, too clever
Why do we not just choose the model that scores best on its training data? Because that model may simply have memorised the answers. Picture two students before an exam. One learned the ideas. The other learned the practice paper word for word. On the practice paper, the second one scores full marks. On the real exam, with new questions, they fall apart.
Google's crash course describes the same thing. Overfitting is 'creating a model that matches (memorizes) the training set so closely that the model fails to make correct predictions on new data'. And 'An overfit model is analogous to an invention that performs well in the lab but is worthless in the real world.'
The scikit-learn example shows the other side too. It fits a wavy curve with models of different flexibility. A straight line 'is not sufficient to fit the training samples. This is called underfitting.' Too much flexibility, and the model 'learns the noise of the training data'. Noise here means the random wobble in real data that has no pattern to learn.
Underfitting
Underfitting is the student who only read the chapter titles. The model is so simple that it misses the pattern. It makes big mistakes on the training data and the new data alike. Our baseline is the extreme case: it ignores the input completely.
Overfitting
Overfitting is the student with the memorised practice paper. The model is so flexible that it copies even the random wobble. Its training mistakes are tiny, and its mistakes on new data are not. Google gives one cause that matters here: 'The model is too complex.' (The other it names is training data that does not represent real life.)
Quick check
A model has almost no mistakes on its training data but many on the validation set. What is the likely problem?
Bias and variance, gently
Statisticians split a model's mistakes into two kinds. Wikipedia gives the short versions. 'The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).' And: 'The variance is an error from sensitivity to small fluctuations in the training set. High variance may result from an algorithm modeling the random noise in the training data (overfitting).'
Picture darts. High bias: every dart lands left of the bullseye, the same way each time. Consistent and wrong. High variance: darts all over the board. You want them close together and near the middle.
The scikit-learn guide says it in model terms. Its too-simple model 'can at best provide only a poor fit' because it is too simple, which is high bias. Its over-flexible one 'approximates the training data perfectly but does not fit the true function very well', because it is 'very sensitive to varying training data'. That is high variance.
Now the trade-off. Wikipedia describes it: as a model gets more flexible, it 'can better fit a training data set', so its bias goes down. But a flexible model will tend to change a lot each time you give it a new batch of training data, so its variance goes up. The bias–variance dilemma is 'the conflict in trying to simultaneously minimize these two sources of error'. You cannot drive both to zero by turning one dial. You look for a balance.
The complexity curve
You can see all this in one picture. Make a list of models of rising complexity. For each, measure the error on the training set and on the validation set. Two things happen.
Training error keeps falling
A more flexible model can always fit the data it saw a little better. It can even reach zero.
Validation error falls, then turns back up
At first extra flexibility helps. Then the model starts to copy the noise, and its mistakes on new data grow.
The best model sits at the dip
That lowest point on the validation line is the balance between too simple and too clever.
Google's crash course describes the same shape in terms of loss, a number for how wrong the model is. On its generalization curve, 'after a certain number of iterations, loss declines or holds steady (converges) for the training set, but increases for the validation set. This suggests overfitting.' The scikit-learn guide offers a tool for it called a validation curve, which plots the training score and the validation score for different values of one setting, 'to find out whether the estimator is overfitting or underfitting for some hyperparameter values'. A hyperparameter is a setting you choose before training, like the k below.
A worked example
Let us build the curve ourselves, using only plain Python. Our model is the nearest-neighbour idea from the supervised learning lesson. To guess the answer for a new point, find the k closest known points and average their answers. Here k is the setting we choose. A small k trusts the nearest few points. A large k averages over many.
Wikipedia ties k to our bias and variance words: 'In k-nearest neighbor models, a high value of k leads to high bias and low variance.' So k = 1 should be the flexible, memorising end, and k = 25 the stiff, simple end. Let us check.
The data is made up. The answer rises with x, with a repeating wave and some random noise. A fixed seed means you get the same numbers as we did. We make 40 training points and 20 validation points, build the baseline, then try six values of k. The metric is MAE, the average size of the mistakes. Lower is better.
import random
random.seed(7)
def make_rows(n):
rows = []
for _ in range(n):
x = random.uniform(0, 10)
y = 3 * x + 8 * (x % 3) + random.gauss(0, 4)
rows.append((x, y))
return rows
train = make_rows(40)
valid = make_rows(20)
def predict(rows, x, k):
nearest = sorted(rows, key=lambda r: abs(r[0] - x))[:k]
return sum(y for _, y in nearest) / k
def mae(rows, guesses):
return sum(abs(y - g) for (_, y), g in zip(rows, guesses)) / len(rows)
mean_y = sum(y for _, y in train) / len(train)
print("baseline (always the mean):")
print(" train MAE", round(mae(train, [mean_y] * len(train)), 2),
" valid MAE", round(mae(valid, [mean_y] * len(valid)), 2))
print("k-nearest neighbours:")
for k in [1, 3, 5, 9, 15, 25]:
t = mae(train, [predict(train, x, k) for x, _ in train])
v = mae(valid, [predict(train, x, k) for x, _ in valid])
print(f" k={k:<2} train MAE {t:5.2f} valid MAE {v:5.2f}")Output
baseline (always the mean): train MAE 8.82 valid MAE 12.91 k-nearest neighbours: k=1 train MAE 0.00 valid MAE 5.77 k=3 train MAE 3.13 valid MAE 4.95 k=5 train MAE 3.67 valid MAE 5.43 k=9 train MAE 4.77 valid MAE 6.46 k=15 train MAE 5.95 valid MAE 7.80 k=25 train MAE 7.08 valid MAE 9.69
Read the table like a detective.
The baseline
Always guessing the mean is off by 12.91 on average on the validation set. Any model worth keeping must do better than this.
k = 1 memorised
Its training error is exactly 0.00, because each training point is its own nearest neighbour. Yet on the validation set it is off by 5.77. That gap is overfitting.
k = 25 is too stiff
Both errors are high and close to each other. It averages so many points that it flattens the wave. That is underfitting.
The best validation score is 4.95, at k = 3. As k grows from 1, the training error climbs every time, from 0.00 to 7.08, because the model is getting simpler. The validation error is different. It falls from 5.77 to 4.95, then rises again to 9.69. That is the turn-back-up shape of the curve. So we choose k = 3.
Tip
Do not trust the exact numbers too much. We used only 20 validation points. A different seed or a different split could move the winner to k = 5. A way to get steadier scores is called cross-validation, and it has its own lesson soon.
Quick check
In the table, why is the training error exactly 0.00 when k = 1?
When scores are close
Sometimes two models score almost the same. Wikipedia gives the usual tie-breaker: 'Given candidate models of similar predictive or explanatory power, the simplest model is most likely to be the best choice (Occam's razor).'
Occam's razor is a rule of thumb: when two explanations work equally well, prefer the simpler one. It is only a tie-breaker. It does not say simple always wins: our baseline lost by a mile.
Real life has more than one score
The best score is not the only thing in a real project. After the numbers, ask the plain questions.
Speed
How fast must it answer? Behind a button people tap, it must be quick.
Size
Where will it run, and how much room does it have?
Explainability
Can you tell a person why it said what it said? For some jobs that matters as much as the score.
And think about how much data you have. We saw that a larger training set tends to decrease variance. So with little data, a very flexible model is a risk. With lots of data, you can afford more flexibility.
The golden rule: never choose on the test set
One last warning, and it is the most important. The test set is for one job: a single honest score for the model you already chose. If you use it to compare candidates, it stops being honest.
Google's crash course sets out the idea: while developing a model, 'your test set serves as a proxy for real-world data'. A proxy is a stand-in. It only works while it stays unseen. So use the validation set for every choice you make, and look at the test set once, at the end.
What to remember
Start with a baseline, because a score means little without something to compare it to. Compare every candidate on the same validation set with one metric you chose first. Too simple a model underfits and too flexible a model overfits, and the best one sits at the dip of the validation curve. When scores tie, prefer the simpler model. Weigh speed, size, explainability and how much data you have. And never choose using the test set.
Cross-validation, a steadier way to compare, gets its own lesson.
Next: tuning a model's settings.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.