Machine Learning Foundations / Core / 02.02
Feature scaling and normalization
Put every feature on a fair scale, so a number that is big only because of its units does not shout over the rest.
14 min readFree
Why scale at all?
Imagine a talent show with two judges. One scores out of 10. The other scores out of 10,000. Add their scores together and the second judge decides every winner. Nobody planned that. It is only the scale.
Models meet the same problem. A feature is one column of your data, such as age or income. Features use different units, so their numbers live on very different ranges. Google's Machine Learning Crash Course gives an example: one feature spans 154 to 24,917,482, another spans 5 to 22. Feature scaling moves features onto a similar scale, so each gets a fair say.
Key words
- Feature
- One column of your data: one thing you measure about each example, such as age or income.
- Min-max scaling
- Squeezing a feature so its smallest value becomes 0 and its largest becomes 1.
- Standardization
- Shifting and stretching a feature so it has a mean of 0 and a standard deviation of 1. Its values are called z-scores.
- Robust scaling
- A scaling that uses the median and the IQR instead of the mean, so outliers cannot bend it.
- Clipping
- Capping extreme values at a chosen limit.
- Training set and test set
- The part of the data a model learns from, and the part kept back to check it later.
Mean, standard deviation, median, IQR and outlier come from the descriptive statistics lesson.
Trouble one: distance
Many models decide by distance. Wikipedia explains the risk: many classifiers use the Euclidean distance, the straight-line distance between two points. If one feature has a broad range of values, the distance is governed by that feature. So each feature's range should be normalized, so each contributes about equally.
A small test: three people, described by age and yearly income. A is 25 and earns 40,000. B is 60 and earns 41,000. C is 26 and earns 52,000. Who is most like A? Most of us say C. B is 35 years older. Watch the computer.
import math
# (age in years, yearly income in dollars)
a = (25, 40000)
b = (60, 41000)
c = (26, 52000)
def dist(p, q):
return math.sqrt(sum((x - y) ** 2 for x, y in zip(p, q)))
print("raw A-B", round(dist(a, b), 1))
print("raw A-C", round(dist(a, c), 1))
ages = [25, 60, 26]
incomes = [40000, 41000, 52000]
def minmax(values):
lo, hi = min(values), max(values)
return [(v - lo) / (hi - lo) for v in values]
people = list(zip(minmax(ages), minmax(incomes)))
sa, sb, sc = people
print("scaled A-B", round(dist(sa, sb), 3))
print("scaled A-C", round(dist(sa, sc), 3))Output
raw A-B 1000.6 raw A-C 12000.0 scaled A-B 1.003 scaled A-C 1.0
With raw numbers, B looks twelve times closer to A than C does. The 35-year age gap barely registers next to the income numbers. After min-max scaling (explained below), age and income count about equally and the two distances are almost the same.
Trouble two: gradient descent
Many models learn by gradient descent. If it is hazy, revisit the lessons on derivatives and the gradient. In short: the loss scores how wrong the model is, the gradient points uphill, and the model takes small steps downhill until it reaches the bottom.
Google says that when features have different ranges, gradient descent can "bounce" and slow convergence, meaning it takes longer to settle. More advanced optimizers, such as Adagrad and Adam, protect against this by changing the effective learning rate over time, so scaling helps but is not magic.
Picture the loss as a bowl. With one wide-range feature it becomes a long narrow valley: a big step bounces wall to wall, a small step crawls.
This is a made-up bowl, not a real model. One direction is 100 times steeper. The step size must stay below 2 divided by 100, so we use 0.019. On the round bowl, 0.9 is safe. The code counts steps to get near the bottom.
# A bowl-shaped loss: 0.5 * (a * w1**2 + b * w2**2). Its lowest point is (0, 0).
# a and b say how steep the bowl is along each weight.
def steps_to_settle(a, b, lr):
w1, w2 = 1.0, 1.0
for step in range(1, 100_001):
w1 -= lr * a * w1 # gradient along w1 is a * w1
w2 -= lr * b * w2 # gradient along w2 is b * w2
if 0.5 * (a * w1**2 + b * w2**2) < 1e-6:
return step
return None
# Unscaled: one feature has a range 100 times bigger, so the bowl is steep one way.
# The biggest safe step size is limited by the steep direction (must stay below 2 / 100).
print("lopsided bowl:", steps_to_settle(1, 100, 0.019))
# Scaled: the bowl is round, so a big step is safe.
print("round bowl: ", steps_to_settle(1, 1, 0.9))Output
lopsided bowl: 343 round bowl: 3
343 steps against 3, on a toy bowl. The idea carries over to real data.
Google adds two more: without scaling, the model pays too much attention to wide-range features, and very large values can cause the "NaN trap", where a value over the floating-point limit becomes NaN (not a number) and other numbers eventually follow.
Min-max scaling
The simplest method is min-max scaling, also called rescaling. Subtract the feature's smallest value, then divide by the gap between the largest and smallest. In words: how far along the range is this value, as a fraction?
Here x is the original value, x' the scaled value, and x min and x max the smallest and largest values. The smallest lands on 0, the largest on 1. Wikipedia notes you can aim for another range, such as [-1, 1].
- 1
1. Find the smallest and largest
Google's feature quantity spans 100 to 900.
- 2
2. Subtract the minimum
For a value of 300: 300 minus 100 is 200.
- 3
3. Divide by the range
The range is 800. 200 divided by 800 is 0.25, one quarter of the way along.
Google calls this linear scaling and says it suits a feature when the bounds do not change much over time, outliers are few and not extreme, and values are roughly evenly spread. Its example is human age: bounds of about 0 to 100, and only about 0.3 percent of people are over 100.
The outlier condition matters: min-max uses the smallest and largest values, exactly what an outlier changes. One wild value stretches the range and squashes everything else.
Standardization (z-score)
Standardization, or z-score scaling, uses the mean and standard deviation: subtract the mean, then divide by the standard deviation. In words: how many standard deviations from average is this value?
Here mu is the mean and sigma the standard deviation. Wikipedia says each feature ends with zero mean and unit variance, widely used in algorithms such as support vector machines, logistic regression and neural networks. Google's example: mean 100, standard deviation 20, value 130 gives (130 minus 100) divided by 20, which is +1.5.
# min-max, worked on the quantity example from the text
lo, hi, x = 100, 900, 300
print((x - lo) / (hi - lo))
# z-score, on 5 house sizes (square metres)
import statistics
sizes = [50, 60, 70, 80, 90]
mu = statistics.mean(sizes)
sd = statistics.pstdev(sizes)
print("mean", mu, "sd", round(sd, 3))
print([round((s - mu) / sd, 3) for s in sizes])Output
0.25 mean 70 sd 14.142 [-1.414, -0.707, 0.0, 0.707, 1.414]
The sizes become -1.414 to +1.414, centred on zero. The standard deviation here divides by n, the population version.
Google says z-score scaling suits data that is normal, the bell shape, or close. If a bell-shaped feature still has extreme outliers, such as net worth, Google suggests combining z-scores with clipping.
Robust scaling for outliers
The mean and standard deviation bend toward outliers. The median and IQR stay put. Wikipedia describes robust scaling as standardization using the median and IQR, designed to be robust to outliers. Subtract the median, then divide by the IQR.
Q1, Q2 and Q3 are the 25th, 50th and 75th percentiles, so Q2 is the median and Q3 minus Q1 is the IQR. Scikit-learn agrees: with many outliers, scaling with the mean and variance is likely to not work very well, and RobustScaler uses more robust estimates for the centre and range.
Ten made-up house prices, in thousands: nine ordinary (30 to 45) and one at 900, a mansion or a typo. We try both scalings on a normal house (40) and the odd one (900).
import statistics
prices = [30, 32, 35, 36, 38, 40, 41, 43, 45, 900] # one extreme value (thousands)
mean = statistics.mean(prices)
sd = statistics.pstdev(prices)
print("mean", round(mean, 1), "sd", round(sd, 1))
print("z of 40:", round((40 - mean) / sd, 3))
print("z of 900:", round((900 - mean) / sd, 3))
median = statistics.median(prices)
q1, _, q3 = statistics.quantiles(prices, n=4, method="inclusive")
iqr = q3 - q1
print("median", median, "Q1", q1, "Q3", q3, "IQR", iqr)
print("robust of 40:", round((40 - median) / iqr, 3))
print("robust of 900:", round((900 - median) / iqr, 3))
others = [round((p - mean) / sd, 3) for p in prices[:-1]]
print("z of the other nine:", min(others), "to", max(others))Output
mean 124 sd 258.7 z of 40: -0.325 z of 900: 3.0 median 39.0 Q1 35.25 Q3 42.5 IQR 7.25 robust of 40: 0.138 robust of 900: 118.759 z of the other nine: -0.363 to -0.305
The mansion drags the mean up to 124 and the standard deviation up to 258.7. All nine ordinary houses get z-scores between -0.363 and -0.305, a sliver the model can barely tell apart. With robust scaling the house at 40 sits near zero and the mansion stands out at about 119. The outlier stays, but it no longer distorts the scale for everyone else. Min-max would squash them too, between 0 and about 0.02.
Long tails: log scaling and clipping
Some features have a long tail: most values are small and a few are enormous. Google calls this a power law distribution. Its examples are movie ratings (a few movies have lots of ratings, most have very few) and book sales (most books sell a couple of hundred copies, only a few bestsellers sell over a million).
For heavy tails Google suggests log scaling: use each value's natural logarithm, which shrinks huge gaps. Its book example: ln(100) is about 4.6 and ln(1,000,000) is about 13.8. A million copies is 10,000 times 100 copies, yet the log is only about three times bigger.
You cannot take the log of zero or a negative number, so check your feature first.
Clipping is the blunt tool for extreme outliers: cap values above an upper limit, and below a lower limit. Google's roomsPerPerson feature has over 99 percent of values near a mean of 1.8 (standard deviation 0.7), but a few reach 17. Cap it at 4.0 and every value above 4.0 becomes 4.0. After z-scores, you can set anything above 3 to 3 and below -3 to -3.
import math
sales = [100, 250, 400, 1_000, 5_000, 20_000, 1_000_000]
print([round(math.log(s), 2) for s in sales])
def clip(value, low, high):
return max(low, min(high, value))
zs = [-0.4, 0.1, 0.8, -1.2, 7.5, -9.0]
print([clip(z, -3, 3) for z in zs])Output
[4.61, 5.52, 5.99, 6.91, 8.52, 9.9, 13.82] [-0.4, 0.1, 0.8, -1.2, 3, -3]
Sales from 100 to a million become 4.61 to 13.82. Clipping left normal z-scores alone and pulled 7.5 down to 3 and -9.0 up to -3.
Tip
Clip with care. Google warns that some outliers are actually important.
Quick check
Almost all of a data centre's temperature readings fall between 15 and 30 degrees Celsius. A few hot days give values between 31 and 45. Every 1,000th reading is wrongly stored as 1,000. What does Google's exercise suggest?
Which models care?
Not every model needs scaling. Scikit-learn's example says even if tree based models are (almost) not affected by scaling, many other algorithms require features to be normalized: to ease the convergence (a non-penalized logistic regression) or to create a completely different model fit (KNeighbors models).
Distance-based models
K-nearest neighbours and clustering judge by distance. Wikipedia says K-means is sensitive to feature scales. Our people test showed scaling can change the answer.
Models trained by gradient descent
Scaling helps these converge faster. Wikipedia adds it matters when regularization is in the loss, so coefficients are penalized appropriately.
Tree-based models
Scikit-learn says these are (almost) not affected by scaling.
The same example shows that principal component analysis (PCA) is impacted by scaling too. Google recommends normalizing numeric features covering distinctly different ranges, such as age and income, and a single feature with a wide range, such as city population. Its advice: the best technique is the one that works well in practice, so try new ideas.
Learn the numbers from training data only
Scaling uses numbers learned from data: the min and max, the mean, the median. Whose data? Think of an exam: you must not see the paper while you revise. If numbers from the test set shape the scaling, the test is no longer a fair exam.
Scikit-learn's StandardScaler computes the mean and standard deviation on a training set so the same transformation can be re-applied to the testing set later. Google adds: if you normalize a feature during training, you must also normalize it when making predictions.
- 1
1. Split first
Set the test data aside before you compute anything.
- 2
2. Learn from training only
Find the min and max, or mean and standard deviation, or median and IQR, from training rows only.
- 3
3. Apply everywhere
Scale training, test and future data with those same numbers. Never recompute them on the test set.
- 4
4. Keep the numbers
Save them with the model for every future prediction.
train = [10, 20, 30, 40, 50]
test = [60]
lo, hi = min(train), max(train) # learned from the training data only
scale = lambda v: (v - lo) / (hi - lo)
print("train:", [scale(v) for v in train])
print("test: ", [scale(v) for v in test])
# The mistake: learning min and max from train + test together
all_lo, all_hi = min(train + test), max(train + test)
print("leaky train:", [round((v - all_lo) / (all_hi - all_lo), 2) for v in train])Output
train: [0.0, 0.25, 0.5, 0.75, 1.0] test: [1.25] leaky train: [0.0, 0.2, 0.4, 0.6, 0.8]
The test value 60 lands at 1.25, outside 0 to 1. That is honest: it is beyond anything the training data had. The leaky version quietly changes the training numbers using data the model should not see.
Quick check
You scale a feature with min-max, learning the min and max from the training set. A new value arrives at prediction time. What should you do?
One word, many meanings
The word normalization is used for several different things. Our sources use it three ways.
Google's Crash Course
Normalization means transforming features to a similar scale, covering linear scaling, Z-score scaling, log scaling and clipping.
Wikipedia and scikit-learn's example
Wikipedia says "min-max normalization" for min-max scaling and "Z-score normalization" for standardization. Scikit-learn's example agrees on the second.
Scikit-learn's preprocessing guide
There, Normalization is the process of scaling individual samples to have unit norm. It works on each row, not each column. Wikipedia calls the same idea unit vector normalization.
That third meaning is different. Every method so far works on a column (one feature). Unit norm works on a row (one example): it divides the row by its length, so the row has length 1. Scikit-learn says this helps if you use a quadratic form such as the dot-product to quantify the similarity of two samples.
import math
row = [1.0, -1.0, 2.0] # one sample (one row) with three feature values
length = math.sqrt(sum(v * v for v in row))
print(round(length, 3))
print([round(v / length, 3) for v in row])Output
2.449 [0.408, -0.408, 0.816]
Tip
When you read a tutorial, check which normalization it means. In your own notes, name the method: 'min-max scaled' or 'standardized'.
What to remember
Why
Wide-range features dominate distances, slow gradient descent and win too much attention.
How
Min-max for tidy data, z-score for bell shapes, robust for outliers, log for long tails, clipping for extremes.
Rules
Learn the numbers from training data only, reuse them everywhere, and name the method you mean.
Next: feature selection.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.