Machine Learning Foundations / Core / 02.13
Train - Test Data: splitting your data
Hold some data back like a secret exam. Learn the three parts of a split, and how to split fairly: random, stratified, by date and by group.
14 min readFree
Why we hold data back
Picture a student who studies from a book of 100 practice questions. On exam day, the teacher hands out the same 100 questions. The student scores 100 out of 100. Does that prove they understand the subject? No. It may only prove they memorised the answers.
A model can do the same thing. If you test it on the rows it learned from, it can look brilliant and still fail on anything new. So we do what a fair teacher does: we keep some questions secret. We hold back part of the data, and the model never sees it while it learns. That held-back part is the exam.
The last lesson ended with loading data. This one is about the very next move: splitting it. Google's Machine Learning Crash Course says: 'You should test a model against a different set of examples than those used to train the model.' The scikit-learn guide, a well-known Python toolkit's manual, calls the alternative 'a methodological mistake': 'Learning the parameters of a prediction function and testing it on the same data is a methodological mistake'.
The earlier lessons on cleaning and preprocessing said to split first. Here we go deeper: how to split, how big each part should be, and the sneaky ways a split can still fool you.
Key words
- Training set
- The rows the model learns from. Its practice questions.
- Validation set
- Rows used while you build, to compare ideas and adjust settings. Its mock exams.
- Test set
- Rows kept locked away until the end, for one honest final check. The real exam.
- Hyperparameter
- A setting you choose before training, such as how complex the model may be. The model does not learn it; you pick it.
- Class
- One of the answers a model can give, such as 'yes' or 'no'. Class balance means how common each answer is.
- Stratified split
- A split that keeps the class balance the same in every part.
- Random seed
- A starting number for the shuffle. The same seed gives the same shuffle every time.
- Cross-validation
- Testing a model several times on different pieces of the data. It has its own lesson, coming up.
Tip
Hold-out thinking in one line: any row that helped you decide anything cannot also be the judge.
Training, validation and test
The simplest split is two parts: a training set and a test set. Google's course shows this as roughly 80% for training and 20% for testing. It then points out a trap. Say you test, tweak the model, test again, tweak again. Each tweak is guided by the test results. Google compares it to a teacher 'teaching to the test': the more often you reuse the same test set, the more the model fits that set.
The fix is a third part. Google says: 'Dividing the dataset into two sets is a decent idea, but a better approach is to divide the dataset into three subsets.' The scikit-learn guide describes the same flow: 'training proceeds on the training set, after which evaluation is done on the validation set, and when the experiment seems to be successful, final evaluation can be done on the test set.'
- 1
Train
The model learns from the training set.
- 2
Validate
Check it on the validation set. Not good enough? Change something (settings, features, even the whole model) and train again.
- 3
Repeat
Keep going until the validation results look good. Tweaking is allowed here, and only here.
- 4
Test once
Now, and only now, use the test set as a final double-check. Report that number.
Wikipedia's article on these data sets agrees on the roles. The validation set is used to tune the hyperparameters of a model. A test set that has never been used for anything else is called a holdout data set. It also warns that names vary: 'The term "validation set" is sometimes used instead of "test set" in some literature.' So when you read other material, check which role a name really means.
Even the three-part plan wears out. Google says: 'the more you use the same data to make decisions about hyperparameter settings or other model improvements, the less confidence that the model will make good predictions on new data.' Its advice: collect fresh data to refresh them when you can.
Training set
Used to learn. It is fine for the model to see these rows over and over.
Validation set
Used to choose between ideas. You look at its scores often, so it slowly wears out.
Test set
Used to report. Look as rarely as you can. The scikit-learn pitfalls page puts it plainly: 'Test data should never be used to make choices about the model.'
Quick check
You try five settings and keep the one that scores best on the test set. What went wrong?
How big should each part be?
There is no magic percentage. Wikipedia says: 'Deciding the sizes and strategies for data set division in training, test and validation sets is very dependent on the problem and data available.'
Here is what the sources actually state. Google's two-part picture is about 80% training and 20% test. In scikit-learn's train_test_split helper, if you give no sizes, the test part defaults to 25%. Its own guide example holds out 40%: of 150 rows, 90 train and 60 test. These are examples, not laws.
What matters more is that the held-back part is good. Google lists what a good test or validation set looks like. It is 'Large enough to yield statistically significant testing results.' It is 'Representative of the dataset as a whole.' It also matches the real-world data the model will meet, and has 'Zero examples duplicated in the training set.' 'Statistically significant' just means big enough that the score is not a lucky or unlucky fluke.
Tip
With very small data, a 20% test set is only a handful of rows, and its score jumps around. Cross-validation, a later lesson, helps with that.
Random and stratified splits
The usual way to split is to shuffle the rows, then cut. It is like shuffling a deck of cards and dealing the last few as the test hand. In scikit-learn, shuffling is on by default.
Shuffling matters. If your file is sorted by customer, the last 20% unshuffled looks nothing like the rest. Google asks for a test set that is 'Representative of the dataset as a whole'.
But random has a weakness when one answer is rare. By chance, your test set may get too many or too few of the rare rows. The scikit-learn guide notes that with rare classes, 'cross-validation splitting can generate train or validation folds without any occurrence of a particular class.'
The cure is a stratified split. Split each class on its own, so each part keeps the same mix. The guide says stratified splitters 'ensure that relative class frequencies are approximately preserved in each fold.' In train_test_split you switch it on by passing the class labels to the stratify setting.
Let us see it. We make 20 made-up rows, 5 'yes' and 15 'no', so the balance is 25% / 75%. We split 80/20 twice with seed 1. The random split just shuffles. The stratified split deals with 'yes' and 'no' separately.
import random
from collections import Counter
# 20 made-up rows: (day, label). 5 "yes" and 15 "no", so 25% / 75%.
labels = ["yes", "no", "no", "no", "yes", "no", "no", "no", "no", "yes",
"no", "no", "no", "yes", "no", "no", "no", "no", "yes", "no"]
rows = [(day, label) for day, label in enumerate(labels, start=1)]
def counts(part):
c = Counter(label for _, label in part)
return f"yes={c['yes']} no={c['no']}"
# 1. plain random 80/20 split
rng = random.Random(1)
shuffled = rows[:]
rng.shuffle(shuffled)
train, test = shuffled[:16], shuffled[16:]
print("random train:", counts(train), "| test:", counts(test))
# 2. stratified 80/20: split each class on its own, then join
rng = random.Random(1)
train, test = [], []
for label in ["yes", "no"]:
group = [r for r in rows if r[1] == label]
rng.shuffle(group)
cut = round(len(group) * 0.2)
test += group[:cut]
train += group[cut:]
print("stratified train:", counts(train), "| test:", counts(test))
# 3. date-ordered split: oldest 16 days train, newest 4 days test
ordered = sorted(rows)
train, test = ordered[:16], ordered[16:]
print("by date train: days", train[0][0], "to", train[-1][0], counts(train))
print("by date test: days", test[0][0], "to", test[-1][0], counts(test))Output
random train: yes=3 no=13 | test: yes=2 no=2 stratified train: yes=4 no=12 | test: yes=1 no=3 by date train: days 1 to 16 yes=4 no=12 by date test: days 17 to 20 yes=1 no=3
The whole data is 25% yes. The random test set got 2 'yes' out of 4, which is 50%, and the training set got only 3 out of 16. The stratified test set got 1 out of 4, which is 25%, and the training set got 4 out of 16, also 25%. Same seed, fairer mix.
Is the random split just unlucky with seed 1? This loop tries six seeds and counts the 'yes' rows that landed in the test set.
import random
labels = ["yes", "no", "no", "no", "yes", "no", "no", "no", "no", "yes",
"no", "no", "no", "yes", "no", "no", "no", "no", "yes", "no"]
rows = [(day, label) for day, label in enumerate(labels, start=1)]
for seed in range(1, 7):
shuffled = rows[:]
random.Random(seed).shuffle(shuffled)
test = shuffled[16:]
print("seed", seed, "-> yes rows in test:", sum(1 for _, l in test if l == "yes"))Output
seed 1 -> yes rows in test: 2 seed 2 -> yes rows in test: 1 seed 3 -> yes rows in test: 2 seed 4 -> yes rows in test: 1 seed 5 -> yes rows in test: 0 seed 6 -> yes rows in test: 1
It swings from 0 to 2. With a test set of 4 rows, one seed gave a test set with no 'yes' rows at all, so a model could not be checked on the answer you care about. Stratifying removes that lottery. And this works with a seed: we will come back to it.
Splits for data with dates
Some data has a clock. Think of daily sales, weekly sign-ups or monthly bills. If you shuffle these rows, a model may train on next month and be tested on last month. That is peeking at the future. In real life the model will always predict forward, never backward.
The scikit-learn guide says: 'If one knows that the samples have been generated using a time-dependent process, it is safer to use a time-series aware cross-validation scheme.' Its TimeSeriesSplit tool follows one pattern, shown in its example: train on the earlier rows, test on the next one, and 'successive training sets are supersets of those that come before them.'
The plain rule: sort by date, put the older rows in training, and the newest rows in test.
The first program above already did this in its last lines. The date-ordered split trained on days 1 to 16 (4 'yes', 12 'no') and tested on days 17 to 20 (1 'yes', 3 'no').
A date split is not random, so the class balance is whatever the calendar gave you. That is fine: it matches how the model will be used.
Quick check
You have two years of daily orders. Which split is fairest for predicting next week's orders?
Group splits: same customer, one side only
Sometimes rows are not independent. One patient may have ten scans. One customer may have many orders. If scans of the same patient land in both training and test, the model can recognise the patient instead of learning the illness. The test then looks easy for the wrong reason.
The guide says GroupKFold 'ensures that the same group is not represented in both testing and training sets', and warns that a flexible model that learns 'highly person specific features' 'could fail to generalize to new subjects.' The fix is to split by group: pick whole customers for the test set, with all their rows.
Here are 10 made-up visits from 5 customers, two each. First a normal row split with seed 3, then a group split with the same seed.
import random
# 10 made-up visits by 5 customers, two visits each
visits = [("Asha", 1), ("Asha", 2), ("Ben", 3), ("Ben", 4), ("Cy", 5),
("Cy", 6), ("Dee", 7), ("Dee", 8), ("Eli", 9), ("Eli", 10)]
# Row split: shuffle the rows, last 2 go to test
rows = visits[:]
random.Random(3).shuffle(rows)
train, test = rows[:8], rows[8:]
seen = {name for name, _ in train}
print("row split, test customers also in train:", sorted({n for n, _ in test} & seen))
# Group split: shuffle the customers, last 1 goes to test, with all its rows
names = sorted({name for name, _ in visits})
random.Random(3).shuffle(names)
test_names = set(names[4:])
train = [v for v in visits if v[0] not in test_names]
test = [v for v in visits if v[0] in test_names]
print("group split, test customers:", sorted({n for n, _ in test}))
print("group split, shared with train:", sorted({n for n, _ in test} & {n for n, _ in train}))Output
row split, test customers also in train: ['Ben', 'Eli'] group split, test customers: ['Ben'] group split, shared with train: []
The row split put Ben and Eli in the test set while the same two customers were also in training. The group split picked one customer, Ben, for the test set, and nobody is shared. The model must now predict for a customer it has never met, which is the real situation for a new customer.
Duplicates and other leaks
A leak is any way the answer reaches the model early. We have met one in the preprocessing lesson: learning an average from test rows. A second is duplicates. If the same row sits in both training and test, the model is being examined on a question it has already seen.
Google lists 'Zero examples duplicated in the training set' as a rule for a good test set, and says: 'We strongly recommend deleting duplicate examples from the test set before testing.' Its quiz notes that if the test score is suspiciously good, many test examples may be duplicates of training examples, which 'can be a problem in a dataset with a lot of redundant examples.'
Remove duplicates before splitting
Run the cleaning step first, so copies cannot end up on both sides.
Split by group when rows belong together
The same customer, patient or device goes to one side only.
Fit every step on training rows only
Scalers, fillers and encoders learn from training data, then are applied to the rest. The scikit-learn pitfalls page: 'The general rule is to never call fit on the test data.'
Tip
A score that looks too good to be true usually is. Check for duplicates, shared groups and peeking at test rows before you celebrate.
Fixed seeds: make the split repeatable
Shuffling uses chance. Without a fixed seed you get a different split on every run, so your scores move and you cannot tell whether a change helped or you were just lucky. scikit-learn says to pass a number to random_state: 'Pass an int for reproducible output across multiple function calls.' The programs above did the same with random.Random(1).
A seed is like writing down the order of a shuffled deck. Anyone with the same seed gets the same deal.
One caution: a seed makes one split repeatable, but it does not make it a better split. If a result only looks good for seed 1, that is a warning. Cross-validation, a later lesson, repeats the test over several different splits.
Putting it together
- 1
Clean and remove duplicates
Start from a tidy table, with copies gone.
- 2
Choose the split style
Random with stratify for plain rows, by date for time data, by group when rows belong together.
- 3
Fix the seed
Write the number down.
- 4
Split before learning anything
Training, validation and test, with the test set locked away.
- 5
Tune on validation only
Touch the test set once, at the end.
Quick check
Why fix a random seed when you split data?
Cross-validation, which tests a model on several different splits, gets its own lesson later.
Next: preparing the final training data.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.