Machine Learning Foundations / Advanced / 03.09
Validation techniques: testing fairly
One split gives one score, and that score is noisy. Learn the map of fair tests: stratified, group, time-series and nested, and the leaks that spoil them.
13 min readFree
Why one test is not enough
The last lesson, on log loss, ended with a way to score a model's chances. A score is only useful if you can trust it. That is the job of validation: testing a model on data it did not learn from, in a way that is fair.
Think of a driving test. If you pass once, on one quiet morning, does that prove you are a good driver? Maybe the roads were empty. Validation is the same: the way you cut up your data decides whether the score means anything.
You already know the basics. The train and test data lesson explained why we hold data back, and the model selection lesson used a validation set to compare models. This lesson is a map. It shows the whole family of validation techniques, when each one fits, and the traps that make a good score look better than it is.
Key words
- Validation
- Scoring a model on data it did not learn from, to see how it will do on new data.
- Hold-out
- Keeping one slice of the data aside for scoring. One cut, one score.
- Fold
- One slice of the data. Several folds together make up the whole data.
- Cross-validation
- Scoring a model several times, each time with a different slice held back, then looking at all the scores.
- Imbalanced classes
- When one answer is rare, such as 5 fraud cases in 100 payments.
- Group
- Rows that belong together, such as all the visits by one patient.
- Leakage
- When information from the held-back data sneaks into training. The score then looks better than real life will be.
- Nested validation
- Tuning settings in an inner test and judging the result in an outer test, so one set of rows never does both jobs.
Hold-out, in one paragraph
Hold-out is the simplest scheme. Google's crash course recommends three subsets: a training set to learn from, a validation set for checking during building, and a test set for one final check. Wikipedia describes the two-part version this way: you randomly assign data points to two sets, usually called the training set and the test set, train on the first and evaluate on the second. What matters here is the word single.
Why a single split is noisy
Wikipedia warns about the hold-out method: 'It should be used with caution because without such averaging of multiple runs, one may achieve highly misleading results.' Why? Because which rows land in the test set is down to chance. A lucky split puts easy rows in the test set. An unlucky one puts the hard ones there. Your model did not change at all, yet the score did.
Let us see it. The data is made up: 60 flats, each with a size and a price in thousands, plus plenty of random wobble. The model is as simple as it gets, a straight line fitted to the training rows. We cut the data 10 times, each with a different seed (the starting number of the shuffle), keep 12 rows for the test and print the average size of the mistakes. Lower is better.
import random
# 60 made-up flats: size in 10 m2 steps (x) and price in thousands (y), with noise
random.seed(11)
rows = []
for _ in range(60):
x = random.uniform(3, 12)
y = 20 * x + 40 + random.gauss(0, 35)
rows.append((x, y))
def fit_line(train):
n = len(train)
mx = sum(x for x, _ in train) / n
my = sum(y for _, y in train) / n
slope = sum((x - mx) * (y - my) for x, y in train) / sum((x - mx) ** 2 for x, _ in train)
return slope, my - slope * mx
def mae(model, test):
slope, icpt = model
return sum(abs(slope * x + icpt - y) for x, y in test) / len(test)
scores = []
for seed in range(1, 11):
shuffled = rows[:]
random.Random(seed).shuffle(shuffled)
train, test = shuffled[12:], shuffled[:12]
score = mae(fit_line(train), test)
scores.append(score)
print(f"seed {seed:2d}: test error {score:.1f}")
print(f"lowest {min(scores):.1f}, highest {max(scores):.1f}, average {sum(scores) / len(scores):.1f}")Output
seed 1: test error 34.4 seed 2: test error 34.4 seed 3: test error 37.9 seed 4: test error 35.8 seed 5: test error 30.4 seed 6: test error 24.5 seed 7: test error 25.1 seed 8: test error 27.0 seed 9: test error 24.0 seed 10: test error 22.4 lowest 22.4, highest 37.9, average 29.6
The same model, the same data, and the test error moves from 22.4 to 37.9. The average over the ten tries is 29.6. If you had run only seed 10, you might have bragged. If you had run only seed 3, you might have given up on a perfectly fine model.
Quick check
You split your data once and the model scores 22. A friend splits it with another seed and gets 38. What is the likely cause?
The map: repeat the test
The first family answers noise by repeating the test on different slices, then looking at all the scores. Both ideas below have their own lessons, so here they get one line each.
K-fold
Cut the data into k equal folds. Train on k minus 1 of them, test on the one left out, and take turns until every fold has been the test. Wikipedia notes that when k equals the number of rows, it becomes leave-one-out. The next lesson covers it.
Leave-one-out
The extreme version. The scikit-learn guide: 'Each learning set is created by taking all the samples except one, the test set being the sample left out.' One test row per round, as many rounds as rows. It has its own lesson after that.
Repeated random splits
Do the hold-out above many times and average, as we just did. Wikipedia calls this repeated random sub-sampling validation, also known as Monte Carlo cross-validation.
Everything from here is about a different question: how to cut the data so each test is fair. It applies to hold-out, k-fold and the rest alike.
Stratified splits for rare answers
Say you check payments for fraud, and only 2 in 100 are fraud. Cut the data at random into small slices and some slices may hold no fraud at all. A test on such a slice cannot tell you if the model catches fraud.
A stratified split fixes this. It cuts each class on its own, so every slice keeps the same mix. The guide says: 'StratifiedKFold is a variation of K-fold which returns stratified folds: each set contains approximately the same percentage of samples of each target class as the complete set.' Wikipedia gives the binary case: each partition contains roughly the same proportions of the two types of class labels.
Use it whenever one answer is rare. It protects the mix of answers, and nothing else.
Group splits: keep people on one side
Many tables hold several rows per person: ten scans per patient, twelve orders per customer. Rows from one person look alike. If some land in training and some in the test, the model can learn to recognise the person, and the test becomes a memory game.
The guide explains it: 'The i.i.d. assumption is broken if the underlying generative process yields groups of dependent samples.' (i.i.d. means each row is independent of the others.) Its medical example is a patient with many samples, where 'we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold.' And GroupKFold 'ensures that the same group is not represented in both testing and training sets.'
The rule in daily words: pick whole people for the test, with all their rows. The test then answers the real question: how will the model do for someone it has never met? If a new patient will walk in tomorrow, that is exactly what you need to know.
Tip
Ask of every table: what is one 'person' here? A patient, a customer, a machine? Whatever it is, split on it.
Time-series splits: past trains, future tests
Data with dates has a direction. A shuffle mixes days, so the model can train on Thursday and Saturday and then be tested on Friday. In real life it will never see Saturday before predicting Friday.
The guide says: 'it is very important to evaluate our model for time series data on the “future” observations least like those that are used to train the model.' Its TimeSeriesSplit trains on the early part and tests on the part after it, and 'successive training sets are supersets of those that come before them.' In plain words: each round the training set grows to include the days just tested, and the next block is the test. Wikipedia agrees that random splits 'might be problematic for time-series models'.
- 1
Sort by date
Oldest first.
- 2
Train on the past
Take an early block of days.
- 3
Test on what follows
The next block, never an earlier one.
- 4
Slide forward
Add that block to training and test on the one after. Repeat.
Let us see how much a shuffle can cheat. The data is made up: 100 days of sales that keep growing, with a little noise. The model is a simple one: to guess a day, average the sales of the 3 closest training days. First we shuffle and test on 20 random days. Then we train on days 0 to 79 and test on the newest 20 days, 80 to 99.
import random
# 100 made-up days of sales that keep growing, plus a little noise
random.seed(5)
sales = [100 + 2 * day + random.gauss(0, 6) for day in range(100)]
days = list(range(100))
def predict(train_days, day):
# average the sales of the 3 training days closest to this day
near = sorted(train_days, key=lambda d: abs(d - day))[:3]
return sum(sales[d] for d in near) / 3
def mae(train_days, test_days):
return sum(abs(predict(train_days, d) - sales[d]) for d in test_days) / len(test_days)
# Shuffled split: 20 random days are tested, the other 80 train
shuffled = days[:]
random.Random(1).shuffle(shuffled)
print("shuffled split: test error", round(mae(shuffled[20:], shuffled[:20]), 1))
# Time split: days 0-79 train, the newest days 80-99 are tested
print("time split: test error", round(mae(days[:80], days[80:]), 1))
# How far is each test day from the closest training day, on average?
def gap(train_days, test_days):
return sum(min(abs(d - t) for t in train_days) for d in test_days) / len(test_days)
print("average gap to nearest training day, shuffled:", round(gap(shuffled[20:], shuffled[:20]), 1))
print("average gap to nearest training day, time: ", round(gap(days[:80], days[80:]), 1))Output
shuffled split: test error 6.6 time split: test error 18.7 average gap to nearest training day, shuffled: 1.1 average gap to nearest training day, time: 10.5
The shuffled test error is 6.6. The honest time split gives 18.7, nearly three times worse. Nothing about the model changed. The last two lines show why. In the shuffled split, a test day sat on average 1.1 days from a training day: the model could look at yesterday and tomorrow. In the time split the gap was 10.5 days: the model had to guess beyond the end of what it knew, as it will in real life.
Quick check
Why is the shuffled score too good for a model that forecasts next month?
Nested validation, in plain words
Suppose you try many settings and keep the one with the best cross-validation score. That best score is now a little too good: you picked it because it was the best, and part of that was luck. The scikit-learn example says the same: 'Model selection without nested CV uses the same data to tune model parameters and evaluate model performance. Information may thus “leak” into the model and overfit the data.' Wikipedia puts it as a rule: when cross-validation is used for both choosing hyperparameters and estimating error, 'a nested cross-validation is required.'
The idea is two loops. The inner loop tunes: it tries the settings on the training part only. The outer loop judges: it scores the winning setup on rows the tuning never touched. In the example's words, 'In the inner loop ... the score is approximately maximized by fitting a model to each training set,' and 'In the outer loop ... generalization error is estimated by averaging test set scores over several dataset splits.' Tuning inside, judging outside. It is the train, validation and test idea you know, repeated for every slice.
Tip
The rows that helped you choose cannot also be the rows that grade you.
Leakage inside the folds
A fair split can still be spoiled by what you do before it. The classic case is scaling, which means rewriting numbers onto a common range, using the average and spread of a column. If you scale all the data first and then cut it into folds, each test slice has already helped to set the average. The test rows have left a fingerprint on the training data.
The scikit-learn guide says it directly: preprocessing 'such as standardization, feature selection, etc.' and similar data transformations 'should be learnt from a training set and applied to held-out data for prediction'. With several folds, redo the learning inside every fold.
Cut first
Make the folds before you scale, fill gaps or pick features.
Learn on training rows only
Inside each fold, the average, the filler values and the chosen features come from its training part.
Apply to the test rows
Use those same numbers on the test part. Never recompute them there.
The data preparation lesson covered scaling and filling in detail. The new idea here is that the rule repeats for every fold.
How to choose a scheme
Work down this list. The first question that fits your data tells you what to protect. Several can apply at once.
- 1
Are there dates that matter?
Then test on the future only: a time-series split.
- 2
Do rows belong to people or things?
Then split by group, so no one is on both sides.
- 3
Is one answer rare?
Then stratify, so each slice holds some of it.
- 4
Is the data small or the score jumpy?
Then repeat the test: k-fold or repeated splits, covered in the next lessons.
- 5
Are you also tuning settings?
Then use nested validation, or a separate validation set plus a locked test set.
- 6
Do all prepare steps live inside the folds?
Scaling, filling and feature picks, learned from the training part of each fold.
Quick check
A shop has 5 years of weekly sales for 40 stores, and you must forecast next quarter for all stores. Which problem should the split protect against first?
In short: validation is a habit of asking what could make this score too good.
Next: k-fold cross-validation, testing on every slice of the data.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.