Machine Learning Foundations / Beginner / 01.52
Preprocessing: getting data ready for a model
Models only do arithmetic. Preprocessing is how raw, messy data becomes a clean table of numbers, and the one rule that keeps your results honest.
14 min readFree
What preprocessing is
Think about cooking. You do not toss a whole muddy carrot into a pot. You wash it, peel it, cut it and measure it first. Data is the same. Raw data arrives muddy: missing cells, words where numbers should be, columns that use different units. A model cannot cook with that. Getting it ready is called preprocessing.
The scikit-learn user guide, a well-known Python toolkit for machine learning, describes its preprocessing tools like this: they 'change raw feature vectors into a representation that is more suitable for the downstream estimators.' In plain words, preprocessing turns raw data into a clean table of numbers that a model can learn from.
The last lesson covered file formats. This one is the map of what happens next. Later lessons zoom into cleaning, feature engineering, scaling, selection and dimensionality reduction, so here each gets a short introduction and no more.
Key words
- Preprocessing
- Everything you do to raw data before a model sees it, so that it becomes a clean table of numbers.
- Feature
- One column of the table: a single thing you know about each row, such as the number of rooms in a house.
- Categorical
- A column whose values are labels from a list, such as a city name or a size of small, medium or large.
- Encoding
- Turning labels or other non-number values into numbers.
- Training data / test data
- Training data is what the model learns from. Test data is kept aside and used afterwards to check the model on rows it has never seen.
- Data leakage
- When information from the test data sneaks into the model's learning, which makes the results look better than they really are.
- Pipeline
- A fixed chain of steps that always runs in the same order, so you can repeat it exactly.
Why models need it
A model is a machine that does arithmetic. Words like 'Leeds' or 'large' mean nothing to it, and a gap where a number should be breaks the sum. Google's Machine Learning Crash Course says it plainly: 'Machine learning models can only manipulate floating-point numbers.' A floating-point number is just a number with a decimal point, like 3.0 or 0.25.
There is a second reason: scale. A house has 3 rooms but costs 250,000. The scikit-learn guide warns that if a feature 'has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.' A column of huge numbers can drown out a column of small ones.
So preprocessing has two jobs: make everything a number, and make those numbers behave well together.
Tip
Whenever a step 'learns' something from the data, ask: learned from which rows? The golden rule below is the answer.
The usual steps
Every project differs, but the journey has a common shape. Here it is as a walk-through.
- 1
Inspect
Look before you touch. How many rows and columns? What type is each column? Which cells are empty? Which values look odd?
- 2
Split
Set some rows aside as test data before you learn anything from the data. More on why in a moment.
- 3
Clean
Fix or fill the empty cells, remove duplicates, correct obvious mistakes. The next lesson, data cleaning, goes deep here.
- 4
Transform and encode
Turn labels, text and dates into numbers. Create new useful columns from old ones. This is the heart of feature engineering.
- 5
Scale
Bring numeric columns onto similar ranges so that none shouts over the others.
- 6
Select or reduce
Keep the columns that help and drop the ones that do not, or squeeze many columns into fewer.
Notice that Split comes early, before any step learns from the data. The golden rule below explains why.
Here is the first step on a tiny made-up table of three houses. Each row is a dictionary: a name for each value. The program counts the rows and finds the missing number. In Python, a missing value is often written None.
from statistics import correlation, pvariance
table = {
"size_m2": [50, 62, 75, 80, 95, 110, 120, 140],
"size_ft2": [538, 667, 807, 861, 1023, 1184, 1292, 1507],
"rooms": [2, 3, 3, 4, 4, 5, 4, 6],
"country": [1, 1, 1, 1, 1, 1, 1, 1],
"door_code": [7, 2, 9, 4, 1, 8, 3, 5],
}
price = [150, 190, 220, 250, 280, 340, 350, 420]
print("variance of each feature")
for name, col in table.items():
print(f"{name:10} {pvariance(col):10.2f}")
print()
print("correlation with price")
for name, col in table.items():
if pvariance(col) > 0:
print(f"{name:10} {correlation(col, price):6.3f}")
print()
print("size_m2 vs size_ft2:", round(correlation(table["size_m2"], table["size_ft2"]), 3))
print("size_m2 vs rooms:", round(correlation(table["size_m2"], table["rooms"]), 3))Output
variance of each feature size_m2 814.50 size_ft2 94471.36 rooms 1.36 country 0.00 door_code 7.36 correlation with price size_m2 0.997 size_ft2 0.997 rooms 0.943 door_code -0.090 size_m2 vs size_ft2: 1.0 size_m2 vs rooms: 0.918
Row number 1 has no value for rooms. Counting starts at 0, so the second row is index 1. Spotting this is inspection. What to do about it is cleaning, and that is the next lesson.
Quick check
A column in your table says 'small', 'medium' or 'large'. What kind of column is this?
Encoding categories
Categorical columns are the most common reason a table is not ready. The scikit-learn guide notes that such features 'can be efficiently coded as integers', and gives an example of a person described as 'male', 'from US', 'uses Internet Explorer' becoming [0, 1, 3]. There are two main ways to encode them, and the difference matters.
Ordinal encoding
Give each label a whole number, such as 0, 1 and 2. This is right only when the labels really have an order, like small, medium, large. The scikit-learn guide warns that integer codes make estimators 'interpret the categories as being ordered, which is often not desired' when the order is arbitrary.
One-hot encoding
Make one new column per label. A row gets a 1 in its own label's column and 0 in all the others. No order is implied. Google's course describes it as turning categorical values into vectors where each category is represented by a unique element with a value of 1.
Why not just numbers for everything?
Google's course gives the risk: if you leave the labels as indexed integers, 'the model would treat the indexed values as continuous floating-point numbers. The model would then consider "purple" six times more likely than "orange."' A code is not a size.
Here is both ideas on tiny lists. The first loop one-hot encodes four house cities. The second part gives sizes an ordered code, because small, medium and large do have an order.
from statistics import correlation
price = [150, 190, 220, 250, 280, 340, 350, 420]
price_with_tax = [round(p * 1.2) for p in price]
size_m2 = [50, 62, 75, 80, 95, 110, 120, 140]
print("size_m2 vs price:", round(correlation(size_m2, price), 3))
print("price_with_tax vs price:", round(correlation(price_with_tax, price), 3))Output
size_m2 vs price: 0.997 price_with_tax vs price: 1.0
The first line is the column order: Hull, Leeds, York. Leeds is [0, 1, 0] because the 1 sits in its column. The sizes come out as [0, 2, 1, 0]: small is 0, medium 1, large 2.
One-hot encoding has a cost: a column with thousands of different labels would become thousands of new columns. Google's course says that for features with many categories 'embeddings or hashing are recommended', and that a one-hot table is mostly zeros, so it can be stored by recording only the position of the 1. You will meet those ideas in the feature engineering lesson.
Quick check
A column holds three cities with no natural order. Which encoding avoids telling the model that one city is 'bigger' than another?
Text and dates, in short
Text and dates are the two other kinds of column you meet often. Both need turning into numbers too. This is only the first look; feature engineering comes later.
Text
The scikit-learn guide says that 'the raw data, a sequence of symbols, cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size.' Its basic fix is called a bag of words: split text into words, then count how often each word occurs, 'while completely ignoring the relative position information of the words'.
Dates
A date as one chunk of text is not useful to a model. A common idea is to pull out the parts that matter, such as the year, the month and the day of the week. The scikit-learn guide notes that datasets often have 'features of different types, say text, floats, and dates, where each type of feature requires separate preprocessing'.
Same goal
Different tools, same goal: a plain number in every cell.
This sample does both by hand with the standard library. Python's datetime module gives the date parts; weekday() counts Monday as 0. The word counts line up with a sorted word list.
from statistics import correlation
rooms = [2, 3, 3, 4, 4, 5, 4, 6]
door_code = [7, 2, 9, 4, 1, 8, 3, 5]
price = [150, 190, 220, 250, 280, 340, 350, 420]
train = slice(0, 6) # the first 6 houses are the training rows
for name, col in [("rooms", rooms), ("door_code", door_code)]:
r = correlation(col[train], price[train])
print(f"{name:10} r on training rows = {r:6.3f}")Output
rooms r on training rows = 0.977 door_code r on training rows = 0.019
So 16 November 2026 becomes the numbers 2026, 11, 16 and 0, and it is a Monday. The sentence becomes a list of counts, one slot per word: 'the' appears twice and every other word once.
The golden rule: learn from training data only
Now the most important idea in this lesson. Some preprocessing steps learn something from the data. A scaler learns the smallest and largest value. A filler of missing cells learns the average. An encoder learns which labels exist. The word for learning this is fit. Using what was learned is transform.
The scikit-learn guide explains the danger, called data leakage: 'Data leakage occurs when information that would not be available at prediction time is used when building the model. This results in overly optimistic performance estimates.' Overly optimistic means it looks good in your notebook and then disappoints in real life.
It then states the rule: 'The general rule is to never call fit on the test data.' And it gives the reason in terms of an average: 'if you have a normalization step where you divide by the average value, the average should be the average of the train subset, not the average of all the data. If the test subset is included in the average calculation, information from the test subset is influencing the model.'
An everyday picture: a student who sees the exam questions while studying will score well and prove nothing. The same guide lists how to avoid leakage. First, 'always split the data into train and test subsets first, particularly before any preprocessing steps.' Second, use fit_transform on the training part and transform on the test part.
The sample below shows a tiny scaler that squeezes numbers to the range 0 to 1. The ten values are split into 8 for training and 2 for testing. First we learn the smallest and largest from the training rows only. Then, for contrast, the leaky version learns them from all ten.
def fit_scaler(values):
lo, hi = min(values), max(values)
return lo, hi
def apply_scaler(values, lo, hi):
return [round((v - lo) / (hi - lo), 2) for v in values]
data = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
train, test = data[:8], data[8:]
lo, hi = fit_scaler(train)
print("fit on train only:", lo, hi)
print("train:", apply_scaler(train, lo, hi))
print("test: ", apply_scaler(test, lo, hi))
lo2, hi2 = fit_scaler(data)
print("leaky, fit on all:", lo2, hi2)
print("test: ", apply_scaler(test, lo2, hi2))Output
fit on train only: 10 80 train: [0.0, 0.14, 0.29, 0.43, 0.57, 0.71, 0.86, 1.0] test: [1.14, 1.29] leaky, fit on all: 10 100 test: [0.89, 1.0]
Tip
Order of work: split first, fit on the training part, then apply the same fitted steps to the test part.
With the correct approach, the scaler learned 10 and 80 from training data. The two test values, 90 and 100, scale to 1.14 and 1.29, above 1, and that is fine and honest: the model really is meeting numbers it has not seen. The leaky version learned 100 from the test set, so the test values land neatly inside 0 to 1 at 0.89 and 1.0. It hid the surprise, which is the leak.
Quick check
You want to fill missing cells with the average. Which average do you use for the test rows?
Keep it repeatable with pipelines
Preprocessing is a recipe, and recipes must be repeatable. By hand, you can forget a step, run steps out of order or fit on the wrong rows. And new data later needs exactly the same treatment.
The answer is a pipeline: a chain of steps run in a fixed order. The scikit-learn documentation says a Pipeline 'can be used to chain multiple estimators into one', and lists three benefits. Convenience: 'You only have to call fit and predict once on your data to fit a whole sequence of estimators.' Safety: 'Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors.' Cross-validation is a way of testing a model several times on different pieces of the data.
Every step except the last must be a transformer, meaning it can transform data; the last can be anything. A companion tool, ColumnTransformer, gives different columns different treatment.
scikit-learn is not installed here, so the next block is not run. This is how the library's own documentation shows leakage-safe preprocessing. It is the documentation's example, shown without output.
# from the scikit-learn user guide (Common pitfalls), not run here
from sklearn.pipeline import make_pipeline
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
pipeline = make_pipeline(SelectKBest(k=25),
HistGradientBoostingClassifier(random_state=1))
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)The idea is simple enough to build yourself. This toy pipeline in plain Python has two steps: fill a missing number with the mean, then halve every value. It learns the mean from [2, None, 6], which is 4, then is applied to new data [None, 10] without learning again.
class Pipe:
def __init__(self, steps):
self.steps = steps
def fit_transform(self, rows):
for step in self.steps:
rows = step.fit(rows).transform(rows)
return rows
def transform(self, rows):
for step in self.steps:
rows = step.transform(rows)
return rows
class MeanFill:
def fit(self, rows):
known = [r for r in rows if r is not None]
self.mean = sum(known) / len(known)
return self
def transform(self, rows):
return [self.mean if r is None else r for r in rows]
class Halve:
def fit(self, rows):
return self
def transform(self, rows):
return [r / 2 for r in rows]
pipe = Pipe([MeanFill(), Halve()])
print(pipe.fit_transform([2, None, 6]))
print(pipe.transform([None, 10]))Output
[1.0, 2.0, 3.0] [2.0, 5.0]
The training run gives [1.0, 2.0, 3.0]. The new data becomes [2.0, 5.0], because its gap was filled with the training mean of 4. Same recipe each time: that is what a pipeline guarantees.
What comes next
Each later lesson widens one part of the path.
Data cleaning
Missing values, duplicates, outliers (very odd values) and typos. This is the next lesson.
Feature engineering, scaling, and selection
Making useful new columns, bringing numbers to a common range, and choosing which columns to keep.
Dimensionality reduction
Squeezing many columns into fewer while keeping what matters. Useful when one-hot encoding has made the table very wide.
Quick check
Why does the guide recommend a pipeline for preprocessing?
Next: data cleaning.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.