esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.02

Feature scaling and normalization

Put every feature on a fair scale, so a number that is big only because of its units does not shout over the rest.

14 min readFree

Why scale at all?

Imagine a talent show with two judges. One scores out of 10. The other scores out of 10,000. Add their scores together and the second judge decides every winner. Nobody planned that. It is only the scale.

Models meet the same problem. A feature is one column of your data, such as age or income. Features use different units, so their numbers live on very different ranges. Google's Machine Learning Crash Course gives an example: one feature spans 154 to 24,917,482, another spans 5 to 22. Feature scaling moves features onto a similar scale, so each gets a fair say.

Key words

Feature
One column of your data: one thing you measure about each example, such as age or income.
Min-max scaling
Squeezing a feature so its smallest value becomes 0 and its largest becomes 1.
Standardization
Shifting and stretching a feature so it has a mean of 0 and a standard deviation of 1. Its values are called z-scores.
Robust scaling
A scaling that uses the median and the IQR instead of the mean, so outliers cannot bend it.
Clipping
Capping extreme values at a chosen limit.
Training set and test set
The part of the data a model learns from, and the part kept back to check it later.

Mean, standard deviation, median, IQR and outlier come from the descriptive statistics lesson.

Trouble one: distance

Many models decide by distance. Wikipedia explains the risk: many classifiers use the Euclidean distance, the straight-line distance between two points. If one feature has a broad range of values, the distance is governed by that feature. So each feature's range should be normalized, so each contributes about equally.

A small test: three people, described by age and yearly income. A is 25 and earns 40,000. B is 60 and earns 41,000. C is 26 and earns 52,000. Who is most like A? Most of us say C. B is 35 years older. Watch the computer.

Python
import math

# (age in years, yearly income in dollars)
a = (25, 40000)
b = (60, 41000)
c = (26, 52000)

def dist(p, q):
    return math.sqrt(sum((x - y) ** 2 for x, y in zip(p, q)))

print("raw  A-B", round(dist(a, b), 1))
print("raw  A-C", round(dist(a, c), 1))

ages = [25, 60, 26]
incomes = [40000, 41000, 52000]

def minmax(values):
    lo, hi = min(values), max(values)
    return [(v - lo) / (hi - lo) for v in values]

people = list(zip(minmax(ages), minmax(incomes)))
sa, sb, sc = people
print("scaled A-B", round(dist(sa, sb), 3))
print("scaled A-C", round(dist(sa, sc), 3))

Output

raw  A-B 1000.6
raw  A-C 12000.0
scaled A-B 1.003
scaled A-C 1.0

With raw numbers, B looks twelve times closer to A than C does. The 35-year age gap barely registers next to the income numbers. After min-max scaling (explained below), age and income count about equally and the two distances are almost the same.

Trouble two: gradient descent

Many models learn by gradient descent. If it is hazy, revisit the lessons on derivatives and the gradient. In short: the loss scores how wrong the model is, the gradient points uphill, and the model takes small steps downhill until it reaches the bottom.

Google says that when features have different ranges, gradient descent can "bounce" and slow convergence, meaning it takes longer to settle. More advanced optimizers, such as Adagrad and Adam, protect against this by changing the effective learning rate over time, so scaling helps but is not magic.

Picture the loss as a bowl. With one wide-range feature it becomes a long narrow valley: a big step bounces wall to wall, a small step crawls.

This is a made-up bowl, not a real model. One direction is 100 times steeper. The step size must stay below 2 divided by 100, so we use 0.019. On the round bowl, 0.9 is safe. The code counts steps to get near the bottom.

Python
# A bowl-shaped loss: 0.5 * (a * w1**2 + b * w2**2). Its lowest point is (0, 0).
# a and b say how steep the bowl is along each weight.
def steps_to_settle(a, b, lr):
    w1, w2 = 1.0, 1.0
    for step in range(1, 100_001):
        w1 -= lr * a * w1          # gradient along w1 is a * w1
        w2 -= lr * b * w2          # gradient along w2 is b * w2
        if 0.5 * (a * w1**2 + b * w2**2) < 1e-6:
            return step
    return None

# Unscaled: one feature has a range 100 times bigger, so the bowl is steep one way.
# The biggest safe step size is limited by the steep direction (must stay below 2 / 100).
print("lopsided bowl:", steps_to_settle(1, 100, 0.019))
# Scaled: the bowl is round, so a big step is safe.
print("round bowl:   ", steps_to_settle(1, 1, 0.9))

Output

lopsided bowl: 343
round bowl:    3

343 steps against 3, on a toy bowl. The idea carries over to real data.

Google adds two more: without scaling, the model pays too much attention to wide-range features, and very large values can cause the "NaN trap", where a value over the floating-point limit becomes NaN (not a number) and other numbers eventually follow.

Min-max scaling

The simplest method is min-max scaling, also called rescaling. Subtract the feature's smallest value, then divide by the gap between the largest and smallest. In words: how far along the range is this value, as a fraction?

x′=x−xmin⁡xmax⁡−xmin⁡x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}

Here x is the original value, x' the scaled value, and x min and x max the smallest and largest values. The smallest lands on 0, the largest on 1. Wikipedia notes you can aim for another range, such as [-1, 1].

  1. 1

    1. Find the smallest and largest

    Google's feature quantity spans 100 to 900.

  2. 2

    2. Subtract the minimum

    For a value of 300: 300 minus 100 is 200.

  3. 3

    3. Divide by the range

    The range is 800. 200 divided by 800 is 0.25, one quarter of the way along.

Google calls this linear scaling and says it suits a feature when the bounds do not change much over time, outliers are few and not extreme, and values are roughly evenly spread. Its example is human age: bounds of about 0 to 100, and only about 0.3 percent of people are over 100.

The outlier condition matters: min-max uses the smallest and largest values, exactly what an outlier changes. One wild value stretches the range and squashes everything else.

Standardization (z-score)

Standardization, or z-score scaling, uses the mean and standard deviation: subtract the mean, then divide by the standard deviation. In words: how many standard deviations from average is this value?

x′=x−μσx' = \frac{x - \mu}{\sigma}

Here mu is the mean and sigma the standard deviation. Wikipedia says each feature ends with zero mean and unit variance, widely used in algorithms such as support vector machines, logistic regression and neural networks. Google's example: mean 100, standard deviation 20, value 130 gives (130 minus 100) divided by 20, which is +1.5.

Python
# min-max, worked on the quantity example from the text
lo, hi, x = 100, 900, 300
print((x - lo) / (hi - lo))
# z-score, on 5 house sizes (square metres)
import statistics
sizes = [50, 60, 70, 80, 90]
mu = statistics.mean(sizes)
sd = statistics.pstdev(sizes)
print("mean", mu, "sd", round(sd, 3))
print([round((s - mu) / sd, 3) for s in sizes])

Output

0.25
mean 70 sd 14.142
[-1.414, -0.707, 0.0, 0.707, 1.414]

The sizes become -1.414 to +1.414, centred on zero. The standard deviation here divides by n, the population version.

Google says z-score scaling suits data that is normal, the bell shape, or close. If a bell-shaped feature still has extreme outliers, such as net worth, Google suggests combining z-scores with clipping.

Robust scaling for outliers

The mean and standard deviation bend toward outliers. The median and IQR stay put. Wikipedia describes robust scaling as standardization using the median and IQR, designed to be robust to outliers. Subtract the median, then divide by the IQR.

x′=x−Q2Q3−Q1x' = \frac{x - Q_2}{Q_3 - Q_1}

Q1, Q2 and Q3 are the 25th, 50th and 75th percentiles, so Q2 is the median and Q3 minus Q1 is the IQR. Scikit-learn agrees: with many outliers, scaling with the mean and variance is likely to not work very well, and RobustScaler uses more robust estimates for the centre and range.

Ten made-up house prices, in thousands: nine ordinary (30 to 45) and one at 900, a mansion or a typo. We try both scalings on a normal house (40) and the odd one (900).

Python
import statistics

prices = [30, 32, 35, 36, 38, 40, 41, 43, 45, 900]  # one extreme value (thousands)

mean = statistics.mean(prices)
sd = statistics.pstdev(prices)
print("mean", round(mean, 1), "sd", round(sd, 1))
print("z of 40:", round((40 - mean) / sd, 3))
print("z of 900:", round((900 - mean) / sd, 3))

median = statistics.median(prices)
q1, _, q3 = statistics.quantiles(prices, n=4, method="inclusive")
iqr = q3 - q1
print("median", median, "Q1", q1, "Q3", q3, "IQR", iqr)
print("robust of 40:", round((40 - median) / iqr, 3))
print("robust of 900:", round((900 - median) / iqr, 3))
others = [round((p - mean) / sd, 3) for p in prices[:-1]]
print("z of the other nine:", min(others), "to", max(others))

Output

mean 124 sd 258.7
z of 40: -0.325
z of 900: 3.0
median 39.0 Q1 35.25 Q3 42.5 IQR 7.25
robust of 40: 0.138
robust of 900: 118.759
z of the other nine: -0.363 to -0.305

The mansion drags the mean up to 124 and the standard deviation up to 258.7. All nine ordinary houses get z-scores between -0.363 and -0.305, a sliver the model can barely tell apart. With robust scaling the house at 40 sits near zero and the mansion stands out at about 119. The outlier stays, but it no longer distorts the scale for everyone else. Min-max would squash them too, between 0 and about 0.02.

Long tails: log scaling and clipping

Some features have a long tail: most values are small and a few are enormous. Google calls this a power law distribution. Its examples are movie ratings (a few movies have lots of ratings, most have very few) and book sales (most books sell a couple of hundred copies, only a few bestsellers sell over a million).

For heavy tails Google suggests log scaling: use each value's natural logarithm, which shrinks huge gaps. Its book example: ln(100) is about 4.6 and ln(1,000,000) is about 13.8. A million copies is 10,000 times 100 copies, yet the log is only about three times bigger.

x′=ln⁡(x)x' = \ln(x)

You cannot take the log of zero or a negative number, so check your feature first.

Clipping is the blunt tool for extreme outliers: cap values above an upper limit, and below a lower limit. Google's roomsPerPerson feature has over 99 percent of values near a mean of 1.8 (standard deviation 0.7), but a few reach 17. Cap it at 4.0 and every value above 4.0 becomes 4.0. After z-scores, you can set anything above 3 to 3 and below -3 to -3.

Python
import math

sales = [100, 250, 400, 1_000, 5_000, 20_000, 1_000_000]
print([round(math.log(s), 2) for s in sales])

def clip(value, low, high):
    return max(low, min(high, value))

zs = [-0.4, 0.1, 0.8, -1.2, 7.5, -9.0]
print([clip(z, -3, 3) for z in zs])

Output

[4.61, 5.52, 5.99, 6.91, 8.52, 9.9, 13.82]
[-0.4, 0.1, 0.8, -1.2, 3, -3]

Sales from 100 to a million become 4.61 to 13.82. Clipping left normal z-scores alone and pulled 7.5 down to 3 and -9.0 up to -3.

Tip

Clip with care. Google warns that some outliers are actually important.

Quick check

Almost all of a data centre's temperature readings fall between 15 and 30 degrees Celsius. A few hot days give values between 31 and 45. Every 1,000th reading is wrongly stored as 1,000. What does Google's exercise suggest?

Which models care?

Not every model needs scaling. Scikit-learn's example says even if tree based models are (almost) not affected by scaling, many other algorithms require features to be normalized: to ease the convergence (a non-penalized logistic regression) or to create a completely different model fit (KNeighbors models).

  • Distance-based models

    K-nearest neighbours and clustering judge by distance. Wikipedia says K-means is sensitive to feature scales. Our people test showed scaling can change the answer.

  • Models trained by gradient descent

    Scaling helps these converge faster. Wikipedia adds it matters when regularization is in the loss, so coefficients are penalized appropriately.

  • Tree-based models

    Scikit-learn says these are (almost) not affected by scaling.

The same example shows that principal component analysis (PCA) is impacted by scaling too. Google recommends normalizing numeric features covering distinctly different ranges, such as age and income, and a single feature with a wide range, such as city population. Its advice: the best technique is the one that works well in practice, so try new ideas.

Learn the numbers from training data only

Scaling uses numbers learned from data: the min and max, the mean, the median. Whose data? Think of an exam: you must not see the paper while you revise. If numbers from the test set shape the scaling, the test is no longer a fair exam.

Scikit-learn's StandardScaler computes the mean and standard deviation on a training set so the same transformation can be re-applied to the testing set later. Google adds: if you normalize a feature during training, you must also normalize it when making predictions.

  1. 1

    1. Split first

    Set the test data aside before you compute anything.

  2. 2

    2. Learn from training only

    Find the min and max, or mean and standard deviation, or median and IQR, from training rows only.

  3. 3

    3. Apply everywhere

    Scale training, test and future data with those same numbers. Never recompute them on the test set.

  4. 4

    4. Keep the numbers

    Save them with the model for every future prediction.

Python
train = [10, 20, 30, 40, 50]
test = [60]

lo, hi = min(train), max(train)       # learned from the training data only
scale = lambda v: (v - lo) / (hi - lo)
print("train:", [scale(v) for v in train])
print("test: ", [scale(v) for v in test])

# The mistake: learning min and max from train + test together
all_lo, all_hi = min(train + test), max(train + test)
print("leaky train:", [round((v - all_lo) / (all_hi - all_lo), 2) for v in train])

Output

train: [0.0, 0.25, 0.5, 0.75, 1.0]
test:  [1.25]
leaky train: [0.0, 0.2, 0.4, 0.6, 0.8]

The test value 60 lands at 1.25, outside 0 to 1. That is honest: it is beyond anything the training data had. The leaky version quietly changes the training numbers using data the model should not see.

Quick check

You scale a feature with min-max, learning the min and max from the training set. A new value arrives at prediction time. What should you do?

One word, many meanings

The word normalization is used for several different things. Our sources use it three ways.

  • Google's Crash Course

    Normalization means transforming features to a similar scale, covering linear scaling, Z-score scaling, log scaling and clipping.

  • Wikipedia and scikit-learn's example

    Wikipedia says "min-max normalization" for min-max scaling and "Z-score normalization" for standardization. Scikit-learn's example agrees on the second.

  • Scikit-learn's preprocessing guide

    There, Normalization is the process of scaling individual samples to have unit norm. It works on each row, not each column. Wikipedia calls the same idea unit vector normalization.

That third meaning is different. Every method so far works on a column (one feature). Unit norm works on a row (one example): it divides the row by its length, so the row has length 1. Scikit-learn says this helps if you use a quadratic form such as the dot-product to quantify the similarity of two samples.

Python
import math

row = [1.0, -1.0, 2.0]               # one sample (one row) with three feature values
length = math.sqrt(sum(v * v for v in row))
print(round(length, 3))
print([round(v / length, 3) for v in row])

Output

2.449
[0.408, -0.408, 0.816]

Tip

When you read a tutorial, check which normalization it means. In your own notes, name the method: 'min-max scaled' or 'standardized'.

What to remember

  • Why

    Wide-range features dominate distances, slow gradient descent and win too much attention.

  • How

    Min-max for tidy data, z-score for bell shapes, robust for outliers, log for long tails, clipping for extremes.

  • Rules

    Learn the numbers from training data only, reuse them everywhere, and name the method you mean.

Next: feature selection.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.