esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.11

LOOCV: leave one out, test every example

K-fold taken to its limit: every example takes one turn as the test set. See how it works, why small datasets love it, and what it costs.

14 min readFree

K-fold, pushed as far as it goes

The last lesson cut the data into k equal parts called folds. Each fold took a turn as the test set, and the other folds trained the model. You picked k yourself, often 5 or 10. So here is a fair question. What happens if you keep raising k until it cannot go any higher?

It stops when every fold holds exactly one example. If you have 12 examples, k is 12. That extreme has its own name: leave-one-out cross-validation, or LOOCV for short. The scikit-learn guide says it plainly: for k-fold, 'if k = n, this is equivalent to the Leave One Out strategy'. Here n is the number of examples.

Think of twelve pupils. The teacher hides one pupil's answer, predicts it from the other eleven, and checks. She does this twelve times, so everyone is the hidden pupil once.

Key words

Cross-validation
Testing a model several times, each time on a different slice of the data, and averaging the scores.
LOOCV
Leave-one-out cross-validation. Every example is the test set exactly once, all alone.
Variance (of a score)
How much a score would jump around if you repeated the test on slightly different data.
Error
How far a guess is from the real answer. Here we use the absolute error: the gap, ignoring whether the guess was too high or too low.

How it works

The rule is short. Take one example out. Train on all the rest. Guess the one you took out, and note how wrong the guess was. Put it back, take the next one out, and repeat. When every example has had its turn, average the errors.

  1. 1

    Count your examples

    Say you have n of them. You will train the model n times.

  2. 2

    Hide example number 1

    It is the whole test set. The other n minus 1 examples are the training set.

  3. 3

    Train on the rest

    Build a fresh model from scratch. Never reuse a model that has seen the hidden example.

  4. 4

    Guess the hidden example

    Compare the guess with its real answer and write down the error.

  5. 5

    Repeat for every example

    Hide number 2, then number 3, and so on, until each has been hidden once.

  6. 6

    Average the n errors

    That one number is your LOOCV score. Lower is better.

The scikit-learn guide describes the same loop: 'Each learning set is created by taking all the samples except one, the test set being the sample left out. Thus, for n samples, we have n different training sets and n different test sets.' Here is what the splits look like for just five examples, numbered 0 to 4.

Python
for left_out in range(5):
    train = [i for i in range(5) if i != left_out]
    print("test:", [left_out], "train:", train)

Output

test: [0] train: [1, 2, 3, 4]
test: [1] train: [0, 2, 3, 4]
test: [2] train: [0, 1, 3, 4]
test: [3] train: [0, 1, 2, 4]
test: [4] train: [0, 1, 2, 3]

Each example is tested once and trains four times.

In words, the score is the average of the gaps between each real answer and the guess made without it. Here it is as a formula. The symbol y with a little hat is a guess, and the small minus i means 'made without example i'.

LOOCV error=1n∑i=1n∣ yi−y^(−i) ∣\text{LOOCV error} = \frac{1}{n}\sum_{i=1}^{n}\left|\,y_i - \hat{y}_{(-i)}\,\right|

Why small datasets love it

Imagine you have only 12 examples. Keep a quarter for testing and you train on 9. Those 3 test examples are also a tiny, shaky judge. Every example held back is one the model never learns from.

LOOCV wastes almost nothing. The guide puts it this way: 'This cross-validation procedure does not waste much data as only one sample is removed from the training set.' It also compares the sizes. Each LOO training set holds n − 1 samples, while a k-fold one holds (k − 1) n / k. With 12 examples, that is 11 against 9 for 4-fold.

  • Training sets are nearly full size

    The model learns from all but one example every time, so each round looks like the real thing you will ship.

  • Every example gets tested

    Each of the 12 is the exam paper once. Nothing is left out of the scoring.

  • No luck in the split

    There is no shuffle and no random choice. Run it twice and you get the same score.

A worked example

The model is k-nearest neighbours for numbers (regression). To guess a new point, it finds the k closest training points on the x line and averages their answers. A small k trusts only the very nearest. A larger k averages over more. This k is a setting we choose, and it is not the k of k-fold, so keep the two apart.

The data is made up: 12 points, where y grows with x, with a little wiggle and random noise. A fixed seed means you get the same numbers we did.

Python
import random

random.seed(11)

xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]

for i, (x, y) in enumerate(data):
    print(i, x, y)

Output

0 0.9 7.6
1 1.8 11.0
2 3.0 8.8
3 4.5 10.0
4 4.7 16.6
5 5.1 12.9
6 5.1 11.9
7 5.6 13.7
8 5.9 17.7
9 6.3 13.8
10 7.9 20.8
11 9.2 21.9

Next, LOOCV itself. The function loocv takes each point out in turn, trains on the other 11, guesses the hidden one and keeps the error. We run it for k = 1, 3 and 5, and print the error of every left-out point, then the mean.

Python
import random

random.seed(11)
xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]


def predict(train, x, k):
    nearest = sorted(train, key=lambda row: abs(row[0] - x))[:k]
    return sum(y for _, y in nearest) / k


def loocv(data, k):
    errors = []
    for i, (x, y) in enumerate(data):
        train = data[:i] + data[i + 1:]
        errors.append(abs(predict(train, x, k) - y))
    return errors


results = {k: loocv(data, k) for k in (1, 3, 5)}

print("left-out  k=1   k=3   k=5")
for i in range(len(data)):
    row = "  ".join(f"{results[k][i]:4.1f}" for k in (1, 3, 5))
    print(f"{i:>8}  {row}")
print()
for k in (1, 3, 5):
    print(f"k={k}: mean error {sum(results[k]) / len(data):.2f}, trainings {len(data)}")

Output

left-out  k=1   k=3   k=5
       0   3.4   2.3   4.3
       1   3.4   2.2   0.2
       2   2.2   3.7   3.7
       3   6.6   3.8   4.6
       4   6.6   5.0   3.4
       5   1.0   1.2   1.1
       6   1.0   2.5   2.3
       7   4.0   0.5   0.9
       8   4.0   4.2   3.9
       9   3.9   1.0   0.8
      10   1.1   3.0   4.8
      11   1.1   4.5   6.1

k=1: mean error 3.19, trainings 12
k=3: mean error 2.82, trainings 12
k=5: mean error 2.99, trainings 12

Point 3 is hard for k = 1: once it is hidden, its nearest neighbour (point 4) sits much higher, 16.6 against 10.0, so the error is 6.6. For k = 3 it is 3.8. Point 5 is easy for all three, near 1.0. The last lines average each column: 3.19 for k = 1, 2.82 for k = 3 and 2.99 for k = 5. So LOOCV picks k = 3, with k = 1 clearly the weakest.

Quick check

In the run above, how many times was the model trained for each value of k?

LOOCV against 4-fold

Now the same data with 4-fold cross-validation: 12 points in 4 folds of 3. We shuffle the points once with a fixed seed so the folds are not just neighbours. We print the test sets and the run counts too.

Python
import random

random.seed(11)
xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]


def predict(train, x, k):
    nearest = sorted(train, key=lambda row: abs(row[0] - x))[:k]
    return sum(y for _, y in nearest) / k


def folds_score(data, k, folds):
    errors = []
    for fold in folds:
        train = [row for i, row in enumerate(data) if i not in fold]
        for i in fold:
            errors.append(abs(predict(train, data[i][0], k) - data[i][1]))
    return sum(errors) / len(errors)


order = list(range(12))
random.seed(5)
random.shuffle(order)
four = [order[i::4] for i in range(4)]
loo = [[i] for i in range(12)]

print("4-fold test sets:", four)
for k in (1, 3, 5):
    print(f"k={k}: 4-fold {folds_score(data, k, four):.2f}   LOOCV {folds_score(data, k, loo):.2f}")
print("trainings per k: 4-fold", len(four), "LOOCV", len(loo))
print("trainings for 3 values of k: 4-fold", 3 * len(four), "LOOCV", 3 * len(loo))

for seed in (5, 6, 7, 8):
    random.seed(seed)
    o = list(range(12))
    random.shuffle(o)
    f = [o[i::4] for i in range(4)]
    print("seed", seed, [round(folds_score(data, k, f), 2) for k in (1, 3, 5)])

Output

4-fold test sets: [[10, 1, 8], [2, 3, 5], [11, 6, 4], [7, 0, 9]]
k=1: 4-fold 3.19   LOOCV 3.19
k=3: 4-fold 2.73   LOOCV 2.82
k=5: 4-fold 3.11   LOOCV 2.99
trainings per k: 4-fold 4 LOOCV 12
trainings for 3 values of k: 4-fold 12 LOOCV 36
seed 5 [3.19, 2.73, 3.11]
seed 6 [3.11, 2.61, 2.83]
seed 7 [2.92, 2.75, 2.92]
seed 8 [3.19, 2.56, 3.0]

Both methods agree that k = 3 is best (2.73 for 4-fold, 2.82 for LOOCV). The scores differ a little because 4-fold trains on 9 points, not 11. The counts are the cost. For one value of k, 4-fold trains 4 times and LOOCV 12. For three values of k, that is 12 trainings against 36.

There is one more difference. The 4-fold score depends on how the shuffle happened to fall. Let us change the shuffle seed four times and see.

Python
import random

random.seed(11)
xs = sorted(round(random.uniform(0, 10), 1) for _ in range(12))
data = [(x, round(2 * x + 3 * (x % 2) + random.gauss(0, 2), 1)) for x in xs]


def predict(train, x, k):
    nearest = sorted(train, key=lambda row: abs(row[0] - x))[:k]
    return sum(y for _, y in nearest) / k


def folds_score(data, k, folds):
    errors = []
    for fold in folds:
        train = [row for i, row in enumerate(data) if i not in fold]
        for i in fold:
            errors.append(abs(predict(train, data[i][0], k) - data[i][1]))
    return sum(errors) / len(errors)



def four_fold(seed):
    order = list(range(12))
    random.seed(seed)
    random.shuffle(order)
    return [order[i::4] for i in range(4)]


loo = [[i] for i in range(12)]
print("shuffle   k=1   k=3   k=5")
for seed in (5, 6, 7, 8):
    row = "  ".join(f"{folds_score(data, k, four_fold(seed)):.2f}" for k in (1, 3, 5))
    print(f"4-fold {seed}  {row}")
row = "  ".join(f"{folds_score(data, k, loo):.2f}" for k in (1, 3, 5))
print(f"LOOCV    {row}")

Output

shuffle   k=1   k=3   k=5
4-fold 5  3.19  2.73  3.11
4-fold 6  3.11  2.61  2.83
4-fold 7  2.92  2.75  2.92
4-fold 8  3.19  2.56  3.00
LOOCV    3.19  2.82  2.99

Look at k = 3: 4-fold gives 2.73, 2.61, 2.75 and 2.56 depending only on the shuffle. LOOCV gives 2.82 every time. So LOOCV has no split luck. That is not the same as being more trustworthy, as the next part explains.

What it costs

Training n models is the first bill. With 12 points it is nothing. With a million, it is a million trainings. The scikit-learn guide says it directly: 'When compared with k-fold cross validation, one builds n models from n samples instead of k models, where n > k.' So 'LOO is more computationally expensive than k-fold cross validation'.

Python
from math import comb

for n in (12, 1000, 100000):
    print(f"n={n}: LOOCV {n} trainings, 10-fold 10, leave-2-out {comb(n, 2)}")

Output

n=12: LOOCV 12 trainings, 10-fold 10, leave-2-out 66
n=1000: LOOCV 1000 trainings, 10-fold 10, leave-2-out 499500
n=100000: LOOCV 100000 trainings, 10-fold 10, leave-2-out 4999950000

Wikipedia makes the same warning: 'n passes may still require quite a large computation time, in which case other approaches such as k-fold cross validation may be more appropriate.' The third figure in the output belongs to the next section.

The score swings a lot

The second bill is subtler. The guide warns: 'LOO often results in high variance as an estimator for the test error.' Its reason: 'since n − 1 of the n samples are used to build each model, models constructed from folds are virtually identical to each other and to the model built from the entire training set.'

Picture twelve judges who read the same notes. Their scores agree so closely that averaging them tells you little more than asking one. LOOCV's models are nearly twins, so its errors are not twelve fresh opinions.

There is a balance, and the guide gives it too: 'if the learning curve is steep for the training size in question, then 5 or 10-fold cross validation can overestimate the generalization error.' A learning curve shows how a model improves as it gets more training data. If the model is still learning fast, 10-fold trains on less, so it looks worse than the full model would. Then it ends with the advice: 'As a general rule, most authors and empirical evidence suggest that 5 or 10-fold cross validation should be preferred to LOO.'

Quick check

Your dataset has 5 million rows. Which is the sensible first choice?

Leave-p-out in one line

Leave-p-out takes out p examples at a time, in every possible way. Wikipedia defines it as 'using p observations as the validation set and the remaining observations as the training set', repeated 'on all ways to cut the original sample'. LOOCV is the case p = 1.

(np)=n!p! (n−p)!\binom{n}{p} = \frac{n!}{p!\,(n-p)!}

That formula counts the ways to choose p of n examples. For n = 12 and p = 2 it gives 66 trainings, and the table above shows how fast it grows. Wikipedia's own example: with n = 100 and p = 30, you need about 3 × 10^25 trainings, which it calls 'computationally infeasible'. In practice, p = 1 is the one you will meet.

A shortcut for some models

Does n trainings always mean n times the work? Not always. Wikipedia notes: 'In some cases such as least squares and kernel regression, cross-validation can be sped up significantly by pre-computing certain values that are needed repeatedly in the training'. And for plain linear regression, 'the results of cross-validation have a closed-form expression known as the prediction residual error sum of squares (PRESS)'. A closed form is a ready formula, so no retraining is needed.

It adds one caution: you "must be careful to preserve the 'total blinding' of the validation set from the training procedure, otherwise bias may result". In plain words, the model must never peek at the example it is tested on. Our sources give no such shortcut for k-nearest neighbours, so the worked example retrains every time.

When to pick it

  • Very small data

    Dozens of examples, where a held-out slice would starve the model and 5 or 10 folds leave tiny, shaky test sets.

  • A cheap model

    Training takes a blink, so n runs cost little. A neighbour-averaging model on 12 points is a good fit.

  • A repeatable score

    You want the same number every run, with no shuffle seed to argue over.

Tip

Do not forget the rule from model selection: once you choose settings by a cross-validated score, that score is a little too hopeful. Keep a test set you never touch for the final check.

Quick check

Why does LOOCV give the same score every time you run it?

Next: back to reinforcement learning with Q-learning, learning the value of every move.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.