Machine Learning Foundations / Core / 02.14
Data preparation: the final training table
The last step before training: join your cleaned, encoded and scaled columns into one numeric table, X, and one column of answers, y, built the safe way.
13 min readFree
The last step before training
Picture a chef before a busy dinner. The vegetables are washed and cut, the sauces are made. The last job is to lay everything on one tray, in the order the cook will reach for it. Data preparation is that tray. It is the final step before a model starts learning.
Wikipedia describes data preparation as the act of manipulating (or pre-processing) raw data into a form that can be readily and accurately analysed. You have already met the steps one by one: cleaning, feature engineering, scaling, selection and the train and test split. Here we join them into the shape a model wants, in the right order.
Key words
- Example (row)
- One thing you learn from, such as one house. scikit-learn calls it a sample.
- Feature (column)
- One fact you know about every example, such as its size in square metres.
- X
- The table of features. One row per example, one column per feature.
- y
- The answers you want the model to predict, one per row. Also called the label or target.
- Fit
- To learn a number from data, such as the median or the smallest value.
- Transform
- To apply a number you already learned, without learning anything new.
- Class imbalance
- When one answer is far more common than another in y.
The shape of X and y
scikit-learn, a popular Python toolkit for machine learning, names the two parts in its glossary. X is data that is observed at training and prediction time, used as independent variables in learning. y is data that may be observed at training time as the dependent variable in learning, but which is unavailable at prediction time, and is usually the target of prediction.
In plain words, X is what you know and y is what you want to guess. Our model will read a house's size and city (X) and guess whether it sells fast (y). For a new house you will have X but not y. That is the whole point.
Think of a spreadsheet. The glossary says it the same way: n_samples is the number of rows in a data array X, and in a data matrix, features are represented as columns.
X has rows and columns
Rows are examples. Columns are features. Ten houses with three features give ten rows and three columns.
y has one value per row
Row 4 of X goes with value 4 of y. Shuffle one and not the other and the answers no longer match.
Every cell is a number
scikit-learn says that nearly all its estimators assume that features are numeric, finite and not missing. No words, no gaps.
Building one numeric table
Raw data rarely arrives as a neat grid. It has gaps, words and columns of very different sizes. Each earlier lesson fixed one kind of mess. Now the fixes meet in one place.
Gaps become numbers
Missing cells are filled or flagged, as in data cleaning. No cell may stay empty.
Words become numbers
Categories such as a city name are encoded, for example one column per city with a 1 or a 0 (one-hot encoding).
Sizes become comparable
Numeric columns are scaled to similar ranges, as in feature scaling and normalization.
Feature selection and dimensionality reduction then decide which columns stay. The result is one table where every column is a number, with the label in its own variable, y.
One idea ties the steps together. Many learn something from the data: a median to fill gaps, a list of cities, a smallest and largest value. Each must come from the training rows only.
The order of steps
Order matters, because some steps learn from the data. The safe order is simple.
- 1
Split first
Set the test rows aside before you learn anything from the data. scikit-learn's advice: always split the data into train and test subsets first, particularly before any preprocessing steps.
- 2
Fit each step on the training rows
Learn the median, the category list, the smallest and largest values. Use training rows only.
- 3
Transform the training rows
Apply what you learned to the training table. This makes X_train.
- 4
Transform the test rows with the same numbers
Apply the very same numbers to the test table. Never learn new ones from it. This makes X_test.
- 5
Check, then train
Run the final checklist below. Only then give X_train and y_train to a model.
Why must the test rows teach nothing? scikit-learn defines data leakage as information that would not be available at prediction time being used when building the model. Its example is a normalization step: the average should be the average of the train subset, not the average of all the data, because if the test subset is included in the average calculation, information from the test subset is influencing the model. The scaling lesson showed the same rule.
In the library's words: never include test data when using the fit and fit_transform methods, and use transform on both train and test subsets, as the same preprocessing should be applied to all the data.
Quick check
You fill missing sizes with the median. Which median should you use?
A worked example in plain Python
Here is all of this on ten houses. Each row holds a size in square metres, a city and a label: 1 if the house sold within a month, 0 if not. None means the size is missing. Only Python's standard library is used.
We pick three test rows by hand, so you can follow them. Then we fit three things on the training rows only: the median size to fill gaps, the list of cities to one-hot, and the smallest and largest size for min-max scaling (which squeezes values to between 0 and 1). We keep them in a small dictionary called a recipe.
from statistics import median
# Raw table: size_m2, city, label (1 = sold within a month). None = missing.
raw = [
(50, "Leeds", 1), (62, "York", 0), (None, "Leeds", 1), (80, "Hull", 0),
(95, "York", 0), (70, "Leeds", 1), (120, "York", 0), (None, "Leeds", 1),
(85, "Hull", 0), (60, "Leeds", 1),
]
test_ids = [3, 7, 8] # chosen by hand so the output is easy to follow
train = [r for i, r in enumerate(raw) if i not in test_ids]
test = [r for i, r in enumerate(raw) if i in test_ids]
# Fit every step on the TRAINING rows only
sizes = [r[0] for r in train if r[0] is not None]
fill = median(sizes)
cities = sorted({r[1] for r in train})
filled = [fill if r[0] is None else r[0] for r in train]
lo, hi = min(filled), max(filled)
recipe = {"fill": fill, "cities": cities, "lo": lo, "hi": hi}
print("recipe:", recipe)
def transform(rows, recipe):
X, y = [], []
for size, city, label in rows:
size = recipe["fill"] if size is None else size
size = (size - recipe["lo"]) / (recipe["hi"] - recipe["lo"])
onehot = [1 if city == c else 0 for c in recipe["cities"]]
X.append([round(size, 2)] + onehot)
y.append(label)
return X, y
X_train, y_train = transform(train, recipe)
X_test, y_test = transform(test, recipe)
print("columns: size, " + ", ".join(cities))
print("X_train:", X_train)
print("y_train:", y_train)
print("X_test: ", X_test)
print("y_test: ", y_test)Output
recipe: {'fill': 66.0, 'cities': ['Leeds', 'York'], 'lo': 50, 'hi': 120}
columns: size, Leeds, York
X_train: [[0.0, 1, 0], [0.17, 0, 1], [0.23, 1, 0], [0.64, 0, 1], [0.29, 1, 0], [1.0, 0, 1], [0.14, 1, 0]]
y_train: [1, 0, 1, 0, 1, 0, 1]
X_test: [[0.43, 0, 0], [0.23, 1, 0], [0.5, 0, 0]]
y_test: [0, 1, 0]The recipe says: fill gaps with 66.0, the cities are Leeds and York, and sizes run from 50 to 120. All of it comes from the seven training rows.
Each row of X has three numbers: the scaled size, then a 1 or 0 for Leeds and for York. The label lives in y.
Reading the result
A gap was filled
The third training row had no size. It got the training median, 66.0, which scales to 0.23. The missing test size got the same 66.0.
Scaling used training numbers
The largest training size, 120, became 1.0. A test house of 85 square metres became 0.5, using the same 50 and 120. A test value above 120 would land above 1.0, which is honest.
An unseen city is all zeros
Hull never appears in the training rows, so it has no column. Both Hull test houses became 0, 0. The recipe knows only what the training rows showed it.
Saving the recipe
scikit-learn warns about this: if data transforms are used when training a model, they also must be used on subsequent datasets, whether it is test data or data in a production system. Otherwise, the feature space will change, and the model will not be able to perform effectively.
So the prepared table is not the only thing to keep. Keep the recipe too: the fill value, the category list, the smallest and largest values, and the order of the columns. Wikipedia makes the same point about data preparation tools: once the preparation work is complete, the underlying steps can be run on other datasets to perform the same operations.
In real projects you rarely write this by hand. scikit-learn offers Pipeline, which chains steps into a fixed sequence. Its documentation lists a Safety benefit: pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors. Its ColumnTransformer applies different transformations to different columns, such as filling and scaling the numbers while one-hot encoding the categories, inside a Pipeline that is safe from data leakage.
Class imbalance
There is one more thing to look at in y before you train. Count how often each answer appears. Google's Machine Learning Crash Course explains that in a class-balanced dataset, the number of positive classes and negative classes is about equal, for example 235 and 247. In a class-imbalanced dataset, one label is considerably more common than the other.
The course says class-imbalanced datasets are far more common than class-balanced datasets in the real world. In a dataset of credit card transactions, fraudulent purchases might make up less than 0.1% of the examples. In a medical diagnosis dataset, patients with a rare virus might be less than 0.01%. The more common label is the majority class, the less common one the minority class.
Imagine a bag of 990 red marbles and 10 blue ones. A model could say 'red' every time and be right 99 times out of 100. Google notes that accuracy is usually a poor metric for assessing a model trained on a class-imbalanced dataset.
Why it makes training hard
Models learn from small groups of examples called batches. Google says batches need a sufficient number of both positive classes and negative classes. A mild imbalance is no problem, but a severely class-imbalanced dataset might not contain enough minority class examples for proper training.
Its example has 200 labels in the majority class and 2 in the minority. If the batch size is 20, most batches won't contain any examples of the minority class. If the batch size is 100, each batch will contain an average of only one minority class example, which is insufficient for proper training.
Two ways to help
Google teaches a two-step technique: downsample the majority class, then upweight it.
- 1
Downsample the majority class
Train on a disproportionately low percentage of majority class examples. Google's example has 99% majority and 1% minority. Downsampling the majority class by a factor of 25 gives 80% majority to 20% minority, and each batch is much more likely to hold enough minority examples.
- 2
Upweight the majority class
Downsampling shows the model an artificial world where the classes are more balanced than in the real world. To correct this prediction bias, upweight by the same factor. Upweighting means treating the loss on a majority class example more harshly than the loss on a minority class example. For a factor of 25, a mistaken majority prediction counts as 25 errors.
How much should you downsample and upweight? Google says to experiment with different factors, just as you would with other hyperparameters (settings you choose before training). Its benefits: a better model that knows both the link between features and labels and the true distribution of the classes, and faster convergence.
scikit-learn offers a related tool called class weights. Its glossary says that class_weight='balanced' can be used to give all classes equal weight by giving each sample a weight inversely related to its class's prevalence in the training data: n_samples / (n_classes * np.bincount(y)). Here is that arithmetic, and the downsampling idea, in plain Python with the marble bag's 990 and 10.
# 990 "not fraud" (0) and 10 "fraud" (1)
y = [0] * 990 + [1] * 10
n, counts = len(y), {0: 990, 1: 10}
# Class weights: n_samples / (n_classes * count of that class)
weights = {c: round(n / (2 * k), 2) for c, k in counts.items()}
print(weights)
# Downsample the majority class by 10, then upweight it by 10
kept = 990 // 10
print("kept:", {0: kept, 1: 10}, "-> share of fraud:", round(10 / (kept + 10), 2))
print("weight of each kept majority example:", 10)Output
{0: 0.51, 1: 50.0}
kept: {0: 99, 1: 10} -> share of fraud: 0.09
weight of each kept majority example: 10Each common example weighs 0.51 and each rare one 50.0, so both classes pull with about the same total strength (990 times 0.51 and 10 times 50 are both about 500). In the second part, keeping 99 of the 990 common examples lifts the rare class from 1% to 9% of the table, and each kept example counts 10 times.
Tip
Resample the training rows only. The test set should keep the real mix of answers.
The final checklist
Run through these before you hand the table to a model.
- 1
No missing values left
Every cell in X_train and X_test holds a number. scikit-learn's estimators assume features are not missing.
- 2
No text left
Every category is encoded. If a cell still holds a word, the model cannot read it.
- 3
Same columns in train and test
The same columns, in the same order, with the same meaning. A column that means size in one table and Leeds in another will give nonsense.
- 4
The label is not inside the features
y must never be a column of X. By definition y is unavailable at prediction time, so a model that sees it while learning would be using information it will not have when it goes live.
- 5
Rows match labels
X and y have the same number of rows, in the same order.
- 6
Every learned number came from training rows
The medians, category lists and scaling limits all came from the training set only.
- 7
The recipe is saved
New data will get exactly the same treatment, later, from the same recipe.
Tip
Ask of every column: would I know this when a new example arrives? If not, it should not be in X.
What to remember
Shape
X is examples by features, all numbers. y is one answer per row, kept apart from X.
Order
Split first. Fit each step on the training rows only. Transform train and test with the same numbers.
Care
Count the labels for imbalance, run the checklist, and save the recipe for new data.
Next: scikit-learn, the toolbox most of this course points to.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.