Machine Learning Foundations / Advanced / 03.10
K-Fold Cross Validation: every slice counts
One split can be lucky. K-fold cross-validation tests on every slice of your data, then reports an average and a spread you can trust.
14 min readFree
One score is not enough
The last lesson ended with a promise: k-fold cross-validation, testing on every slice of the data. This is that lesson.
Start with the problem. You hold back some data, train on the rest, and score the model on the held-back part. That gives one number. But which rows landed in the held-back part was luck. It is like judging a student on a single pop quiz. Maybe the quiz was easy. Maybe it was hard.
Cross-validation runs several quizzes instead. Every row gets a turn, and you end up with a handful of scores. Their average is your estimate, and how far they wander tells you how much to trust it.
Key words
- Fold
- One slice of your data. K-fold cuts the data into k slices of about the same size.
- k
- The number of folds. With k = 5 you cut the data into five slices.
- Score
- One number for how a model did in one round, such as its average mistake.
- Spread
- How far the scores wander from their average. Small spread means the rounds agree. Here we measure it with the standard deviation.
- Stratified
- Split so each slice keeps the same mix of answers as the whole data.
- Setting
- A choice you make before training, such as how many neighbours to look at.
How k-fold works
The idea is small. Cut the data into k slices, then take turns.
- 1
Shuffle, then cut into k folds
Mix the rows, then cut them into k slices of equal size, or as equal as the row count allows.
- 2
Pick one fold to leave out
In round 1, fold 1 is the quiz. The model must not see it.
- 3
Train on the other k-1 folds
The guide puts it in one line: "The prediction function is learned using k - 1 folds, and the fold left out is used for test."
- 4
Score on the fold you left out
Run the model on the quiz fold and write down one score.
- 5
Repeat for every fold
Next round, fold 2 is the quiz. Go on until each fold has had a turn: k rounds, k scores.
- 6
Report the average and the spread
The guide says the figure reported by k-fold is "the average of the values computed in the loop". Add the spread, so people can see how steady the scores were.
Every row is in the quiz exactly once and in training k-1 times. Wikipedia says the same: "each observation is used for validation exactly once."
Tip
Shuffle before you cut. A file sorted by date or by customer would give folds that look nothing alike.
Average and spread
After k rounds you have k scores. The mean says where they sit. The standard deviation says how far they wander around it.
In words: add the k errors and divide by k. For the spread, take each error's gap from the mean, square the gaps, add them, divide by k minus 1, and take the square root.
Report both. The scikit-learn guide does: its example prints "0.98 accuracy with a standard deviation of 0.02". A mean of 3.1 with a spread of 0.2 says the model is steady. The same mean with a spread of 2 says one lucky or unlucky slice could have fooled you.
K-fold from scratch
Time to build it. We make 60 made-up points where the answer rises about 3 for each step in x, plus noise. The model is a straight line fitted by least squares: the slope and starting point that make the squared mistakes smallest. The score is the mean absolute error, the average size of a mistake. We run k = 2, 5 and 10 with a fixed seed. The last lines belong to the repeated k-fold section further down.
import random
import statistics
# 60 made-up points: price rises about 3 per step, plus noise
rng = random.Random(7)
data = []
for _ in range(60):
x = rng.uniform(0, 10)
data.append((x, 3 * x + 5 + rng.gauss(0, 4)))
def fit_line(rows):
# least squares: the slope and start that make the squared mistakes smallest
mx = statistics.mean(x for x, _ in rows)
my = statistics.mean(y for _, y in rows)
slope = sum((x - mx) * (y - my) for x, y in rows) / sum((x - mx) ** 2 for x, _ in rows)
return slope, my - slope * mx
def mean_abs_error(model, rows):
slope, start = model
return statistics.mean(abs(y - (slope * x + start)) for x, y in rows)
def k_fold_scores(rows, k, seed):
rows = rows[:]
random.Random(seed).shuffle(rows)
size = len(rows) // k
scores = []
for i in range(k):
test = rows[i * size:(i + 1) * size]
train = rows[:i * size] + rows[(i + 1) * size:]
scores.append(mean_abs_error(fit_line(train), test))
return scores
for k in (2, 5, 10):
scores = k_fold_scores(data, k, seed=1)
print(f"k = {k}")
for i, s in enumerate(scores, 1):
print(f" fold {i}: error {s:.2f}")
print(f" mean {statistics.mean(scores):.2f}, spread {statistics.stdev(scores):.2f}")
# Repeated k-fold: 5-fold again with four more shuffles
means = [statistics.mean(k_fold_scores(data, 5, seed)) for seed in range(1, 6)]
print("5-fold repeated, means:", [round(m, 2) for m in means])
print(f"mean of repeats {statistics.mean(means):.2f}, spread {statistics.stdev(means):.2f}")Output
k = 2 fold 1: error 2.77 fold 2: error 3.42 mean 3.10, spread 0.46 k = 5 fold 1: error 3.75 fold 2: error 2.17 fold 3: error 2.77 fold 4: error 2.60 fold 5: error 4.19 mean 3.10, spread 0.84 k = 10 fold 1: error 2.92 fold 2: error 4.53 fold 3: error 3.12 fold 4: error 1.24 fold 5: error 2.42 fold 6: error 3.43 fold 7: error 2.49 fold 8: error 2.76 fold 9: error 2.99 fold 10: error 5.47 mean 3.14, spread 1.17 5-fold repeated, means: [3.1, 3.1, 3.14, 3.21, 3.12] mean of repeats 3.13, spread 0.05
Read the output slowly. With k = 2, each round trains on 30 rows and tests on 30. With k = 5, it trains on 48 and tests on 12. With k = 10, it trains on 54 and tests on 6.
The means agree
3.10, 3.10 and 3.14. Whatever k we pick, the estimate lands near 3.1.
The folds do not agree
At k = 5 one fold scored 2.17 and another 4.19. A single split could have shown either.
The spread grew with k
0.46, then 0.84, then 1.17. Here each fold is smaller as k grows, and a quiz of 6 rows is a noisier judge than a quiz of 30. Your own data may behave differently.
Tip
Look at the worst fold. At k = 10, fold 10 scored 5.47 and fold 4 scored 1.24. If one fold is far from the rest, check the rows inside it.
Choosing k
There is no single right k. Wikipedia says "10-fold cross-validation is commonly used, but in general k remains an unfixed parameter." The scikit-learn guide leans the same way in its leave-one-out notes: "most authors and empirical evidence suggest that 5 or 10-fold cross validation should be preferred to LOO". Its own examples use 5 folds. So 5 and 10 are the usual starting points.
The choice is a trade-off.
Small k, like 2
Fast, but each model trains on half the data, so it may look worse than the model you will ship.
Middle k, like 5 or 10
Each model trains on 80% or 90% of the data, and five or ten trainings is a price most projects can pay.
Large k, up to k = n
Almost all the data trains each model, but you train many times. The guide notes that if k = n, k-fold "is equivalent to the Leave One Out strategy".
Data size matters too. Each fold needs enough rows for a quiz to mean something. With 60 rows, k = 10 gave folds of only 6. With 20 rows it would give folds of 2.
Quick check
You have only 30 rows. Which k is the worst fit?
Stratified k-fold
Plain k-fold shuffles and cuts, and it ignores the answers. If one answer is rare, chance can leave a fold with none of it. The scikit-learn guide warns that with rare classes, "cross-validation splitting can generate train or validation folds without any occurrence of a particular class."
Stratified k-fold fixes this. The guide says "each set contains approximately the same percentage of samples of each target class as the complete set." Below we do it by hand: 30 made-up patients, 5 sick and 25 well, five folds. Plain first. Then stratified, which shuffles each group alone and deals it out one at a time.
import random
# 30 made-up patients: only 5 are "sick", 25 are "well"
labels = ["sick"] * 5 + ["well"] * 25
k = 5
# Plain k-fold: shuffle everything, then cut into 5 folds of 6
plain = list(range(30))
random.Random(4).shuffle(plain)
plain_folds = [plain[i:i + 6] for i in range(0, 30, 6)]
print("plain k-fold, sick in each fold:", [sum(labels[i] == "sick" for i in f) for f in plain_folds])
# Stratified k-fold: shuffle each class alone, then deal round-robin
strat_folds = [[] for _ in range(k)]
for name in ("sick", "well"):
ids = [i for i in range(30) if labels[i] == name]
random.Random(4).shuffle(ids)
for n, i in enumerate(ids):
strat_folds[n % k].append(i)
print("stratified, sick in each fold: ", [sum(labels[i] == "sick" for i in f) for f in strat_folds])Output
plain k-fold, sick in each fold: [0, 0, 1, 3, 1] stratified, sick in each fold: [1, 1, 1, 1, 1]
Plain k-fold gave two folds with no sick patient and one fold with three. A model scored on a fold with no sick patients can never be caught missing one. The stratified folds hold exactly one sick patient each.
Tip
Use stratified k-fold when you predict categories and one is rare. For predicting numbers, as in our line example, plain k-fold is fine.
Repeated k-fold
Even k-fold depends on one shuffle. Repeated k-fold runs the whole thing several times with a new shuffle each time. The guide says it "repeats KFold n times, producing different splits in each repetition."
The last two lines of our earlier output did this: 5-fold five times on the same 60 points. The five means were 3.10, 3.10, 3.14, 3.21 and 3.12, with a spread of only 0.05. So the estimate of about 3.1 does not hinge on one lucky cut. The price is five times the training, and Wikipedia says averaging over many splits "is rarely desirable in practice". Use it when data is small and the score really matters.
What it costs
K-fold trains k times, not once. With k = 10, ten models. Wikipedia: "If the prediction method is expensive to train, cross-validation can be very slow since the training must be carried out repeatedly."
The guide admits it too: "This approach can be computationally expensive, but does not waste too much data". A single split is cheap but rests on one cut. K-fold spends compute to save data.
With a cheap model and small data, this costs nothing worth worrying about. With a slow model, ten trainings may take a day, so use a smaller k. Trying many settings multiplies it: ten settings at k = 5 is fifty trainings.
Comparing models and choosing settings
The best use of k-fold is a fair contest. Give each candidate the same folds, score each, and compare the means. If the spreads overlap a lot, it is a draw, and you can pick the simpler model. Settings work the same way: try each value, score it with k-fold, keep the best average.
Then comes the step people forget. The models trained inside the folds each saw only part of the data, and you throw them away. Once you have chosen, train the winner again on all the data you have, so the final model learns from every row.
One more fence. The guide says: "A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV." Cross-validation replaces the validation set. It does not replace the test set. Once you have picked the best of many candidates, its scores are no longer a clean grade. Only data nobody has looked at gives that.
Here is the whole routine on the same data. We lock away 10 rows as the final test set. We compare two candidates with 5-fold on the other 50: a baseline that always guesses the average, and the line. We retrain the winner on all 50 and touch the test set once.
import random
import statistics
rng = random.Random(7)
data = []
for _ in range(60):
x = rng.uniform(0, 10)
data.append((x, 3 * x + 5 + rng.gauss(0, 4)))
def fit_line(rows):
mx = statistics.mean(x for x, _ in rows)
my = statistics.mean(y for _, y in rows)
slope = sum((x - mx) * (y - my) for x, y in rows) / sum((x - mx) ** 2 for x, _ in rows)
return slope, my - slope * mx
def fit_average(rows):
return 0.0, statistics.mean(y for _, y in rows)
def mean_abs_error(model, rows):
slope, start = model
return statistics.mean(abs(y - (slope * x + start)) for x, y in rows)
def cv_mean(fit, rows, k):
size = len(rows) // k
scores = []
for i in range(k):
test = rows[i * size:(i + 1) * size]
train = rows[:i * size] + rows[(i + 1) * size:]
scores.append(mean_abs_error(fit(train), test))
return statistics.mean(scores)
# Lock away 10 rows as the final test set
random.Random(2).shuffle(data)
final_test, working = data[:10], data[10:]
# Compare two candidates with 5-fold on the other 50
for name, fit in (("average", fit_average), ("line", fit_line)):
print(f"{name}: 5-fold error {cv_mean(fit, working, 5):.2f}")
# Retrain the winner on all 50, then touch the test set once
final_model = fit_line(working)
print(f"line retrained on 50 rows, error on the 10 test rows: {mean_abs_error(final_model, final_test):.2f}")Output
average: 5-fold error 7.47 line: 5-fold error 3.01 line retrained on 50 rows, error on the 10 test rows: 3.38
The line's 5-fold error (3.01) beats the average-guesser (7.47) by far, so the line wins. The check on the 10 locked-away rows gave 3.38, close to what k-fold promised and a little higher. That is the grade you report.
Tip
When cross-validation both picks settings and estimates error, Wikipedia says "a nested cross-validation is required": an outer loop to score and an inner loop to choose. Just know it exists.
Where k-fold goes wrong
K-fold is trustworthy only when its folds look like the real future. Three traps to watch.
Rows that belong together
If one customer has many rows, plain k-fold puts them on both sides. Use a group split, as in the Train - Test Data lesson.
Data that moves with time
Wikipedia warns that cross-validation "only yields meaningful results if the validation set and training set are drawn from the same population". Shuffled folds that train on next month to predict last month break that.
Steps done before the split
Scaling on all the data first lets the quiz folds leak into training. The guide says such steps "should be learnt from a training set and applied to held-out data", so do them inside each round.
Quick check
You scored ten settings with 5-fold and picked the best. What do you do next?
Recap
Cut, rotate, average
Train on k-1 folds, test on the one left out, repeat k times, report the mean and the spread.
5 or 10, stratified if classes are rare
Common choices for k. Repeat the run for extra steadiness if you can pay for it.
Choose, retrain, then test once
Compare models with k-fold, retrain the winner on all working data, and keep a test set nobody has touched.
Next: leave-one-out cross-validation, k-fold taken to its limit.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.