Machine Learning Foundations / Advanced / 03.11
LOOCV: leave one out, test every example
K-fold taken to its limit: every example takes one turn as the test set. See how it works, why small datasets love it, and what it costs.
14 min readFree
K-fold, pushed as far as it goes
The last lesson cut the data into k equal parts called folds. Each fold took a turn as the test set, and the other folds trained the model. You picked k yourself, often 5 or 10. So here is a fair question. What happens if you keep raising k until it cannot go any higher?
It stops when every fold holds exactly one example. If you have 12 examples, k is 12. That extreme has its own name: leave-one-out cross-validation, or LOOCV for short. The scikit-learn guide says it plainly: for k-fold, 'if k = n, this is equivalent to the Leave One Out strategy'. Here n is the number of examples.
Think of twelve pupils. The teacher hides one pupil's answer, predicts it from the other eleven, and checks. She does this twelve times, so everyone is the hidden pupil once.
Key words
- Cross-validation
- Testing a model several times, each time on a different slice of the data, and averaging the scores.
- LOOCV
- Leave-one-out cross-validation. Every example is the test set exactly once, all alone.
- Variance (of a score)
- How much a score would jump around if you repeated the test on slightly different data.
- Error
- How far a guess is from the real answer. Here we use the absolute error: the gap, ignoring whether the guess was too high or too low.
How it works
The rule is short. Take one example out. Train on all the rest. Guess the one you took out, and note how wrong the guess was. Put it back, take the next one out, and repeat. When every example has had its turn, average the errors.
- 1
Count your examples
Say you have n of them. You will train the model n times.
- 2
Hide example number 1
It is the whole test set. The other n minus 1 examples are the training set.
- 3
Train on the rest
Build a fresh model from scratch. Never reuse a model that has seen the hidden example.
- 4
Guess the hidden example
Compare the guess with its real answer and write down the error.
- 5
Repeat for every example
Hide number 2, then number 3, and so on, until each has been hidden once.
- 6
Average the n errors
That one number is your LOOCV score. Lower is better.
The scikit-learn guide describes the same loop: 'Each learning set is created by taking all the samples except one, the test set being the sample left out. Thus, for n samples, we have n different training sets and n different test sets.' Here is what the splits look like for just five examples, numbered 0 to 4.
for left_out in range(5):
train = [i for i in range(5) if i != left_out]
print("test:", [left_out], "train:", train)Output
test: [0] train: [1, 2, 3, 4] test: [1] train: [0, 2, 3, 4] test: [2] train: [0, 1, 3, 4] test: [3] train: [0, 1, 2, 4] test: [4] train: [0, 1, 2, 3]
Each example is tested once and trains four times.
In words, the score is the average of the gaps between each real answer and the guess made without it. Here it is as a formula. The symbol y with a little hat is a guess, and the small minus i means 'made without example i'.
Why small datasets love it
Imagine you have only 12 examples. Keep a quarter for testing and you train on 9. Those 3 test examples are also a tiny, shaky judge. Every example held back is one the model never learns from.
LOOCV wastes almost nothing. The guide puts it this way: 'This cross-validation procedure does not waste much data as only one sample is removed from the training set.' It also compares the sizes. Each LOO training set holds n − 1 samples, while a k-fold one holds (k − 1) n / k. With 12 examples, that is 11 against 9 for 4-fold.
Training sets are nearly full size
The model learns from all but one example every time, so each round looks like the real thing you will ship.
Every example gets tested
Each of the 12 is the exam paper once. Nothing is left out of the scoring.
No luck in the split
There is no shuffle and no random choice. Run it twice and you get the same score.
A worked example
The model is k-nearest neighbours for numbers (regression). To guess a new point, it finds the k closest training points on the x line and averages their answers. A small k trusts only the very nearest. A larger k averages over more. This k is a setting we choose, and it is not the k of k-fold, so keep the two apart.
The data is made up: 12 points, where y grows with x, with a little wiggle and random noise. A fixed seed means you get the same numbers we did.
import random
random.seed(11)
xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]
for i, (x, y) in enumerate(data):
print(i, x, y)Output
0 0.9 7.6 1 1.8 11.0 2 3.0 8.8 3 4.5 10.0 4 4.7 16.6 5 5.1 12.9 6 5.1 11.9 7 5.6 13.7 8 5.9 17.7 9 6.3 13.8 10 7.9 20.8 11 9.2 21.9
Next, LOOCV itself. The function loocv takes each point out in turn, trains on the other 11, guesses the hidden one and keeps the error. We run it for k = 1, 3 and 5, and print the error of every left-out point, then the mean.
import random
random.seed(11)
xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]
def predict(train, x, k):
nearest = sorted(train, key=lambda row: abs(row[0] - x))[:k]
return sum(y for _, y in nearest) / k
def loocv(data, k):
errors = []
for i, (x, y) in enumerate(data):
train = data[:i] + data[i + 1:]
errors.append(abs(predict(train, x, k) - y))
return errors
results = {k: loocv(data, k) for k in (1, 3, 5)}
print("left-out k=1 k=3 k=5")
for i in range(len(data)):
row = " ".join(f"{results[k][i]:4.1f}" for k in (1, 3, 5))
print(f"{i:>8} {row}")
print()
for k in (1, 3, 5):
print(f"k={k}: mean error {sum(results[k]) / len(data):.2f}, trainings {len(data)}")Output
left-out k=1 k=3 k=5
0 3.4 2.3 4.3
1 3.4 2.2 0.2
2 2.2 3.7 3.7
3 6.6 3.8 4.6
4 6.6 5.0 3.4
5 1.0 1.2 1.1
6 1.0 2.5 2.3
7 4.0 0.5 0.9
8 4.0 4.2 3.9
9 3.9 1.0 0.8
10 1.1 3.0 4.8
11 1.1 4.5 6.1
k=1: mean error 3.19, trainings 12
k=3: mean error 2.82, trainings 12
k=5: mean error 2.99, trainings 12Point 3 is hard for k = 1: once it is hidden, its nearest neighbour (point 4) sits much higher, 16.6 against 10.0, so the error is 6.6. For k = 3 it is 3.8. Point 5 is easy for all three, near 1.0. The last lines average each column: 3.19 for k = 1, 2.82 for k = 3 and 2.99 for k = 5. So LOOCV picks k = 3, with k = 1 clearly the weakest.
Quick check
In the run above, how many times was the model trained for each value of k?
LOOCV against 4-fold
Now the same data with 4-fold cross-validation: 12 points in 4 folds of 3. We shuffle the points once with a fixed seed so the folds are not just neighbours. We print the test sets and the run counts too.
import random
random.seed(11)
xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]
def predict(train, x, k):
nearest = sorted(train, key=lambda row: abs(row[0] - x))[:k]
return sum(y for _, y in nearest) / k
def folds_score(data, k, folds):
errors = []
for fold in folds:
train = [row for i, row in enumerate(data) if i not in fold]
for i in fold:
errors.append(abs(predict(train, data[i][0], k) - data[i][1]))
return sum(errors) / len(errors)
order = list(range(12))
random.seed(5)
random.shuffle(order)
four = [order[i::4] for i in range(4)]
loo = [[i] for i in range(12)]
print("4-fold test sets:", four)
for k in (1, 3, 5):
print(f"k={k}: 4-fold {folds_score(data, k, four):.2f} LOOCV {folds_score(data, k, loo):.2f}")
print("trainings per k: 4-fold", len(four), "LOOCV", len(loo))
print("trainings for 3 values of k: 4-fold", 3 * len(four), "LOOCV", 3 * len(loo))
for seed in (5, 6, 7, 8):
random.seed(seed)
o = list(range(12))
random.shuffle(o)
f = [o[i::4] for i in range(4)]
print("seed", seed, [round(folds_score(data, k, f), 2) for k in (1, 3, 5)])Output
4-fold test sets: [[10, 1, 8], [2, 3, 5], [11, 6, 4], [7, 0, 9]] k=1: 4-fold 3.19 LOOCV 3.19 k=3: 4-fold 2.73 LOOCV 2.82 k=5: 4-fold 3.11 LOOCV 2.99 trainings per k: 4-fold 4 LOOCV 12 trainings for 3 values of k: 4-fold 12 LOOCV 36 seed 5 [3.19, 2.73, 3.11] seed 6 [3.11, 2.61, 2.83] seed 7 [2.92, 2.75, 2.92] seed 8 [3.19, 2.56, 3.0]
Both methods agree that k = 3 is best (2.73 for 4-fold, 2.82 for LOOCV). The scores differ a little because 4-fold trains on 9 points, not 11. The counts are the cost. For one value of k, 4-fold trains 4 times and LOOCV 12. For three values of k, that is 12 trainings against 36.
There is one more difference. The 4-fold score depends on how the shuffle happened to fall. Let us change the shuffle seed four times and see.
import random
random.seed(11)
xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]
def predict(train, x, k):
nearest = sorted(train, key=lambda row: abs(row[0] - x))[:k]
return sum(y for _, y in nearest) / k
def folds_score(data, k, folds):
errors = []
for fold in folds:
train = [row for i, row in enumerate(data) if i not in fold]
for i in fold:
errors.append(abs(predict(train, data[i][0], k) - data[i][1]))
return sum(errors) / len(errors)
def four_fold(seed):
order = list(range(12))
random.seed(seed)
random.shuffle(order)
return [order[i::4] for i in range(4)]
loo = [[i] for i in range(12)]
print("shuffle k=1 k=3 k=5")
for seed in (5, 6, 7, 8):
row = " ".join(f"{folds_score(data, k, four_fold(seed)):.2f}" for k in (1, 3, 5))
print(f"4-fold {seed} {row}")
row = " ".join(f"{folds_score(data, k, loo):.2f}" for k in (1, 3, 5))
print(f"LOOCV {row}")Output
shuffle k=1 k=3 k=5 4-fold 5 3.19 2.73 3.11 4-fold 6 3.11 2.61 2.83 4-fold 7 2.92 2.75 2.92 4-fold 8 3.19 2.56 3.00 LOOCV 3.19 2.82 2.99
Look at k = 3: 4-fold gives 2.73, 2.61, 2.75 and 2.56 depending only on the shuffle. LOOCV gives 2.82 every time. So LOOCV has no split luck. That is not the same as being more trustworthy, as the next part explains.
What it costs
Training n models is the first bill. With 12 points it is nothing. With a million, it is a million trainings. The scikit-learn guide says it directly: 'When compared with k-fold cross validation, one builds n models from n samples instead of k models, where n > k.' So 'LOO is more computationally expensive than k-fold cross validation'.
from math import comb
for n in (12, 1000, 100000):
print(f"n={n}: LOOCV {n} trainings, 10-fold 10, leave-2-out {comb(n, 2)}")Output
n=12: LOOCV 12 trainings, 10-fold 10, leave-2-out 66 n=1000: LOOCV 1000 trainings, 10-fold 10, leave-2-out 499500 n=100000: LOOCV 100000 trainings, 10-fold 10, leave-2-out 4999950000
Wikipedia makes the same warning: 'n passes may still require quite a large computation time, in which case other approaches such as k-fold cross validation may be more appropriate.' The third figure in the output belongs to the next section.
The score swings a lot
The second bill is subtler. The guide warns: 'LOO often results in high variance as an estimator for the test error.' Its reason: 'since n − 1 of the n samples are used to build each model, models constructed from folds are virtually identical to each other and to the model built from the entire training set.'
Picture twelve judges who read the same notes. Their scores agree so closely that averaging them tells you little more than asking one. LOOCV's models are nearly twins, so its errors are not twelve fresh opinions.
There is a balance, and the guide gives it too: 'if the learning curve is steep for the training size in question, then 5 or 10-fold cross validation can overestimate the generalization error.' A learning curve shows how a model improves as it gets more training data. If the model is still learning fast, 10-fold trains on less, so it looks worse than the full model would. Then it ends with the advice: 'As a general rule, most authors and empirical evidence suggest that 5 or 10-fold cross validation should be preferred to LOO.'
Quick check
Your dataset has 5 million rows. Which is the sensible first choice?
Leave-p-out in one line
Leave-p-out takes out p examples at a time, in every possible way. Wikipedia defines it as 'using p observations as the validation set and the remaining observations as the training set', repeated 'on all ways to cut the original sample'. LOOCV is the case p = 1.
That formula counts the ways to choose p of n examples. For n = 12 and p = 2 it gives 66 trainings, and the table above shows how fast it grows. Wikipedia's own example: with n = 100 and p = 30, you need about 3 × 10^25 trainings, which it calls 'computationally infeasible'. In practice, p = 1 is the one you will meet.
A shortcut for some models
Does n trainings always mean n times the work? Not always. Wikipedia notes: 'In some cases such as least squares and kernel regression, cross-validation can be sped up significantly by pre-computing certain values that are needed repeatedly in the training'. And for plain linear regression, 'the results of cross-validation have a closed-form expression known as the prediction residual error sum of squares (PRESS)'. A closed form is a ready formula, so no retraining is needed.
It adds one caution: you "must be careful to preserve the 'total blinding' of the validation set from the training procedure, otherwise bias may result". In plain words, the model must never peek at the example it is tested on. Our sources give no such shortcut for k-nearest neighbours, so the worked example retrains every time.
When to pick it
Very small data
Dozens of examples, where a held-out slice would starve the model and 5 or 10 folds leave tiny, shaky test sets.
A cheap model
Training takes a blink, so n runs cost little. A neighbour-averaging model on 12 points is a good fit.
A repeatable score
You want the same number every run, with no shuffle seed to argue over.
Tip
Do not forget the rule from model selection: once you choose settings by a cross-validated score, that score is a little too hopeful. Keep a test set you never touch for the final check.
Quick check
Why does LOOCV give the same score every time you run it?
Next: back to reinforcement learning with Q-learning, learning the value of every move.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.