esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.14

Data preparation: the final training table

The last step before training: join your cleaned, encoded and scaled columns into one numeric table, X, and one column of answers, y, built the safe way.

13 min readFree

The last step before training

Picture a chef before a busy dinner. The vegetables are washed and cut, the sauces are made. The last job is to lay everything on one tray, in the order the cook will reach for it. Data preparation is that tray. It is the final step before a model starts learning.

Wikipedia describes data preparation as the act of manipulating (or pre-processing) raw data into a form that can be readily and accurately analysed. You have already met the steps one by one: cleaning, feature engineering, scaling, selection and the train and test split. Here we join them into the shape a model wants, in the right order.

Key words

Example (row)
One thing you learn from, such as one house. scikit-learn calls it a sample.
Feature (column)
One fact you know about every example, such as its size in square metres.
X
The table of features. One row per example, one column per feature.
y
The answers you want the model to predict, one per row. Also called the label or target.
Fit
To learn a number from data, such as the median or the smallest value.
Transform
To apply a number you already learned, without learning anything new.
Class imbalance
When one answer is far more common than another in y.

The shape of X and y

scikit-learn, a popular Python toolkit for machine learning, names the two parts in its glossary. X is data that is observed at training and prediction time, used as independent variables in learning. y is data that may be observed at training time as the dependent variable in learning, but which is unavailable at prediction time, and is usually the target of prediction.

In plain words, X is what you know and y is what you want to guess. Our model will read a house's size and city (X) and guess whether it sells fast (y). For a new house you will have X but not y. That is the whole point.

Think of a spreadsheet. The glossary says it the same way: n_samples is the number of rows in a data array X, and in a data matrix, features are represented as columns.

  • X has rows and columns

    Rows are examples. Columns are features. Ten houses with three features give ten rows and three columns.

  • y has one value per row

    Row 4 of X goes with value 4 of y. Shuffle one and not the other and the answers no longer match.

  • Every cell is a number

    scikit-learn says that nearly all its estimators assume that features are numeric, finite and not missing. No words, no gaps.

Building one numeric table

Raw data rarely arrives as a neat grid. It has gaps, words and columns of very different sizes. Each earlier lesson fixed one kind of mess. Now the fixes meet in one place.

  • Gaps become numbers

    Missing cells are filled or flagged, as in data cleaning. No cell may stay empty.

  • Words become numbers

    Categories such as a city name are encoded, for example one column per city with a 1 or a 0 (one-hot encoding).

  • Sizes become comparable

    Numeric columns are scaled to similar ranges, as in feature scaling and normalization.

Feature selection and dimensionality reduction then decide which columns stay. The result is one table where every column is a number, with the label in its own variable, y.

One idea ties the steps together. Many learn something from the data: a median to fill gaps, a list of cities, a smallest and largest value. Each must come from the training rows only.

The order of steps

Order matters, because some steps learn from the data. The safe order is simple.

  1. 1

    Split first

    Set the test rows aside before you learn anything from the data. scikit-learn's advice: always split the data into train and test subsets first, particularly before any preprocessing steps.

  2. 2

    Fit each step on the training rows

    Learn the median, the category list, the smallest and largest values. Use training rows only.

  3. 3

    Transform the training rows

    Apply what you learned to the training table. This makes X_train.

  4. 4

    Transform the test rows with the same numbers

    Apply the very same numbers to the test table. Never learn new ones from it. This makes X_test.

  5. 5

    Check, then train

    Run the final checklist below. Only then give X_train and y_train to a model.

Why must the test rows teach nothing? scikit-learn defines data leakage as information that would not be available at prediction time being used when building the model. Its example is a normalization step: the average should be the average of the train subset, not the average of all the data, because if the test subset is included in the average calculation, information from the test subset is influencing the model. The scaling lesson showed the same rule.

In the library's words: never include test data when using the fit and fit_transform methods, and use transform on both train and test subsets, as the same preprocessing should be applied to all the data.

Quick check

You fill missing sizes with the median. Which median should you use?

A worked example in plain Python

Here is all of this on ten houses. Each row holds a size in square metres, a city and a label: 1 if the house sold within a month, 0 if not. None means the size is missing. Only Python's standard library is used.

We pick three test rows by hand, so you can follow them. Then we fit three things on the training rows only: the median size to fill gaps, the list of cities to one-hot, and the smallest and largest size for min-max scaling (which squeezes values to between 0 and 1). We keep them in a small dictionary called a recipe.

Python
from statistics import median

# Raw table: size_m2, city, label (1 = sold within a month). None = missing.
raw = [
    (50, "Leeds", 1), (62, "York", 0), (None, "Leeds", 1), (80, "Hull", 0),
    (95, "York", 0), (70, "Leeds", 1), (120, "York", 0), (None, "Leeds", 1),
    (85, "Hull", 0), (60, "Leeds", 1),
]
test_ids = [3, 7, 8]                      # chosen by hand so the output is easy to follow
train = [r for i, r in enumerate(raw) if i not in test_ids]
test = [r for i, r in enumerate(raw) if i in test_ids]

# Fit every step on the TRAINING rows only
sizes = [r[0] for r in train if r[0] is not None]
fill = median(sizes)
cities = sorted({r[1] for r in train})
filled = [fill if r[0] is None else r[0] for r in train]
lo, hi = min(filled), max(filled)
recipe = {"fill": fill, "cities": cities, "lo": lo, "hi": hi}
print("recipe:", recipe)

def transform(rows, recipe):
    X, y = [], []
    for size, city, label in rows:
        size = recipe["fill"] if size is None else size
        size = (size - recipe["lo"]) / (recipe["hi"] - recipe["lo"])
        onehot = [1 if city == c else 0 for c in recipe["cities"]]
        X.append([round(size, 2)] + onehot)
        y.append(label)
    return X, y

X_train, y_train = transform(train, recipe)
X_test, y_test = transform(test, recipe)
print("columns: size, " + ", ".join(cities))
print("X_train:", X_train)
print("y_train:", y_train)
print("X_test: ", X_test)
print("y_test: ", y_test)

Output

recipe: {'fill': 66.0, 'cities': ['Leeds', 'York'], 'lo': 50, 'hi': 120}
columns: size, Leeds, York
X_train: [[0.0, 1, 0], [0.17, 0, 1], [0.23, 1, 0], [0.64, 0, 1], [0.29, 1, 0], [1.0, 0, 1], [0.14, 1, 0]]
y_train: [1, 0, 1, 0, 1, 0, 1]
X_test:  [[0.43, 0, 0], [0.23, 1, 0], [0.5, 0, 0]]
y_test:  [0, 1, 0]

The recipe says: fill gaps with 66.0, the cities are Leeds and York, and sizes run from 50 to 120. All of it comes from the seven training rows.

Each row of X has three numbers: the scaled size, then a 1 or 0 for Leeds and for York. The label lives in y.

Reading the result

  • A gap was filled

    The third training row had no size. It got the training median, 66.0, which scales to 0.23. The missing test size got the same 66.0.

  • Scaling used training numbers

    The largest training size, 120, became 1.0. A test house of 85 square metres became 0.5, using the same 50 and 120. A test value above 120 would land above 1.0, which is honest.

  • An unseen city is all zeros

    Hull never appears in the training rows, so it has no column. Both Hull test houses became 0, 0. The recipe knows only what the training rows showed it.

Saving the recipe

scikit-learn warns about this: if data transforms are used when training a model, they also must be used on subsequent datasets, whether it is test data or data in a production system. Otherwise, the feature space will change, and the model will not be able to perform effectively.

So the prepared table is not the only thing to keep. Keep the recipe too: the fill value, the category list, the smallest and largest values, and the order of the columns. Wikipedia makes the same point about data preparation tools: once the preparation work is complete, the underlying steps can be run on other datasets to perform the same operations.

In real projects you rarely write this by hand. scikit-learn offers Pipeline, which chains steps into a fixed sequence. Its documentation lists a Safety benefit: pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors. Its ColumnTransformer applies different transformations to different columns, such as filling and scaling the numbers while one-hot encoding the categories, inside a Pipeline that is safe from data leakage.

Class imbalance

There is one more thing to look at in y before you train. Count how often each answer appears. Google's Machine Learning Crash Course explains that in a class-balanced dataset, the number of positive classes and negative classes is about equal, for example 235 and 247. In a class-imbalanced dataset, one label is considerably more common than the other.

The course says class-imbalanced datasets are far more common than class-balanced datasets in the real world. In a dataset of credit card transactions, fraudulent purchases might make up less than 0.1% of the examples. In a medical diagnosis dataset, patients with a rare virus might be less than 0.01%. The more common label is the majority class, the less common one the minority class.

Imagine a bag of 990 red marbles and 10 blue ones. A model could say 'red' every time and be right 99 times out of 100. Google notes that accuracy is usually a poor metric for assessing a model trained on a class-imbalanced dataset.

Why it makes training hard

Models learn from small groups of examples called batches. Google says batches need a sufficient number of both positive classes and negative classes. A mild imbalance is no problem, but a severely class-imbalanced dataset might not contain enough minority class examples for proper training.

Its example has 200 labels in the majority class and 2 in the minority. If the batch size is 20, most batches won't contain any examples of the minority class. If the batch size is 100, each batch will contain an average of only one minority class example, which is insufficient for proper training.

Two ways to help

Google teaches a two-step technique: downsample the majority class, then upweight it.

  1. 1

    Downsample the majority class

    Train on a disproportionately low percentage of majority class examples. Google's example has 99% majority and 1% minority. Downsampling the majority class by a factor of 25 gives 80% majority to 20% minority, and each batch is much more likely to hold enough minority examples.

  2. 2

    Upweight the majority class

    Downsampling shows the model an artificial world where the classes are more balanced than in the real world. To correct this prediction bias, upweight by the same factor. Upweighting means treating the loss on a majority class example more harshly than the loss on a minority class example. For a factor of 25, a mistaken majority prediction counts as 25 errors.

How much should you downsample and upweight? Google says to experiment with different factors, just as you would with other hyperparameters (settings you choose before training). Its benefits: a better model that knows both the link between features and labels and the true distribution of the classes, and faster convergence.

scikit-learn offers a related tool called class weights. Its glossary says that class_weight='balanced' can be used to give all classes equal weight by giving each sample a weight inversely related to its class's prevalence in the training data: n_samples / (n_classes * np.bincount(y)). Here is that arithmetic, and the downsampling idea, in plain Python with the marble bag's 990 and 10.

Python
# 990 "not fraud" (0) and 10 "fraud" (1)
y = [0] * 990 + [1] * 10
n, counts = len(y), {0: 990, 1: 10}

# Class weights: n_samples / (n_classes * count of that class)
weights = {c: round(n / (2 * k), 2) for c, k in counts.items()}
print(weights)

# Downsample the majority class by 10, then upweight it by 10
kept = 990 // 10
print("kept:", {0: kept, 1: 10}, "-> share of fraud:", round(10 / (kept + 10), 2))
print("weight of each kept majority example:", 10)

Output

{0: 0.51, 1: 50.0}
kept: {0: 99, 1: 10} -> share of fraud: 0.09
weight of each kept majority example: 10

Each common example weighs 0.51 and each rare one 50.0, so both classes pull with about the same total strength (990 times 0.51 and 10 times 50 are both about 500). In the second part, keeping 99 of the 990 common examples lifts the rare class from 1% to 9% of the table, and each kept example counts 10 times.

Tip

Resample the training rows only. The test set should keep the real mix of answers.

The final checklist

Run through these before you hand the table to a model.

  1. 1

    No missing values left

    Every cell in X_train and X_test holds a number. scikit-learn's estimators assume features are not missing.

  2. 2

    No text left

    Every category is encoded. If a cell still holds a word, the model cannot read it.

  3. 3

    Same columns in train and test

    The same columns, in the same order, with the same meaning. A column that means size in one table and Leeds in another will give nonsense.

  4. 4

    The label is not inside the features

    y must never be a column of X. By definition y is unavailable at prediction time, so a model that sees it while learning would be using information it will not have when it goes live.

  5. 5

    Rows match labels

    X and y have the same number of rows, in the same order.

  6. 6

    Every learned number came from training rows

    The medians, category lists and scaling limits all came from the training set only.

  7. 7

    The recipe is saved

    New data will get exactly the same treatment, later, from the same recipe.

Tip

Ask of every column: would I know this when a new example arrives? If not, it should not be in X.

What to remember

  • Shape

    X is examples by features, all numbers. y is one answer per row, kept apart from X.

  • Order

    Split first. Fit each step on the training rows only. Transform train and test with the same numbers.

  • Care

    Count the labels for imbalance, run the checklist, and save the recipe for new data.

Next: scikit-learn, the toolbox most of this course points to.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.