esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.26

Ridge regression: keeping weights small

A model that memorises its data usually has huge weights. Ridge regression adds a price for big weights, so the model stays calm.

13 min readFree

From margins to small weights

The last lesson, on support vector machines, ended with a promise: ridge regression, keeping a model's weights small. Here it is.

Picture a student who learns for an exam by memorising last year's paper, word for word. Ask the same questions and the marks are perfect. Change one question and the student is lost. A model can do the same thing with its training data. It scores very well on data it has seen, and badly on anything new. That is overfitting.

Ridge regression is a fix. It is ordinary linear regression with one extra rule: big weights cost extra. The model can still fit the data, but it has to pay for every large number it uses. So it learns to get by with small, calm ones.

Key words

Weight
A number the model multiplies an input by. A bigger weight makes that input matter more.
Loss
A score for how wrong the model is. Training tries to make it small.
Overfitting
Fitting the training data so closely, noise included, that new data goes badly.
Regularisation
Anything that holds a model back from overfitting. Ridge adds a penalty to the loss.
Penalty
An extra cost added to the loss, like a fine.
Alpha
How strong the penalty is. You choose it before training. Many books write lambda instead.
Multicollinearity
A long word for input columns that copy each other, fully or nearly.

Why big weights are a warning sign

In the lesson on polynomial regression you gave a model extra columns, x squared, x cubed and so on, so its line could bend. A high degree lets the curve wiggle through every training point. It looks perfect on the points it saw, then swings wildly in between them.

To swing like that, the model needs huge weights. Some push the curve up hard, others pull it back down. Big weights that cancel each other out are how a model bends to chase noise.

That gives us a handy idea: if big weights go with overfitting, then discourage big weights. Google's Machine Learning Crash Course describes L2 regularisation in these terms, as a technique that reduces model complexity by penalising large weights. Ridge regression is that idea applied to linear regression.

Regularisation: a penalty on the loss

Normal training minimises the loss and nothing else. Wikipedia describes explicit regularisation as adding a term to the optimisation problem. In plain words: the model now has two things to keep small, its mistakes and its weights.

Google writes the goal as minimise(loss + complexity). The loss says how wrong the model is. The complexity says how big its weights are. A model that wants a low total cannot ignore either one.

The L2 penalty

Ridge measures the size of the weights with the L2 penalty. The recipe is short: square each weight, then add them all up. Google's formula for it, for weights w1 to wn, is the sum of the squares:

L2=w12+w22+⋯+wn2L_2 = w_1^2 + w_2^2 + \dots + w_n^2

Squaring does two jobs. A negative weight and a positive weight of the same size cost the same, because the square is never negative. And large weights cost far more than small ones, because squaring blows them up.

Google shows a model with six weights: 0.2, -0.5, 5.0, -1.2, 0.3 and -0.1. Their squares add up to 26.83. The single weight of 5.0 contributes about 93% of that total, and the other five together about 7%. Google's summary: weights close to zero do not affect L2 regularisation much, but large weights can have a huge impact.

Ridge: least squares plus the penalty

Ordinary linear regression finds the weights with the smallest squared mistakes. Scikit-learn, a widely used Python library, says ridge regression addresses some of the problems of ordinary least squares by imposing a penalty on the size of the coefficients. (Coefficient is the statistician's word for weight.) It writes the thing being minimised like this, where X holds the inputs, y the answers, w the weights, and alpha the penalty's strength:

min⁡w;∥Xw−y∥22+α ∥w∥22\min_{w} ; \lVert Xw - y \rVert_2^2 + \alpha \, \lVert w \rVert_2^2

Read it in two halves. The first half adds up the squared mistakes. The second half is alpha times the sum of squared weights. The model picks the weights that make the total as small as possible.

Alpha: how strong the penalty is

Alpha is a dial. Scikit-learn says the complexity parameter alpha, which must be zero or more, controls the amount of shrinkage: the larger the value, the greater the shrinkage. Google says the same about its regularisation rate: a high rate strengthens regularisation and lowers the chance of overfitting, while a low rate raises that chance.

  • Alpha = 0

    No penalty at all. Google notes that a rate of zero removes regularisation completely, which carries the highest possible risk of overfitting. You get plain linear regression.

  • Small alpha

    A gentle fine. Big weights are discouraged, but the model keeps most of its freedom.

  • Large alpha

    A heavy fine. Weights are squeezed hard. Too heavy, and the model becomes too stiff to learn the real pattern.

Alpha is a hyperparameter: a setting you pick before training, not something the model works out. Google is direct that the ideal value is data-dependent, so you must do some tuning. That means trying several alphas and comparing them on validation data, as in the lesson on tuning. Scikit-learn even has RidgeCV, ridge with built-in cross-validation of the alpha parameter.

Weights shrink, but never reach zero

As alpha grows, every weight is pulled toward zero. Google says L2 regularisation encourages weights toward 0, but never pushes weights all the way to zero.

That matters if you hoped to find out which features are useless. Ridge will not tell you by dropping them. Google's own quiz says it directly: with L2 regularisation, all features will still contribute something to the model. The next lesson, on lasso, uses a different penalty that can switch weights off.

When features copy each other

Sometimes two columns carry nearly the same information. Height in centimetres and height in inches is the extreme case. Statisticians call this multicollinearity. Scikit-learn explains the trouble: when columns are close to linearly dependent, the design matrix becomes close to singular, and the least-squares estimate becomes highly sensitive to random errors in the observed target, producing a large variance.

In plain words: if two columns say the same thing, the model cannot tell which one deserves the credit. It might give one a huge positive weight and the other a huge negative weight, and the two cancel out. The answer is shaky, and a tiny change in the data can flip it.

Ridge calms this down. Scikit-learn says that the larger the value of alpha, the more the coefficients become robust to collinearity. Wikipedia calls ridge particularly useful to mitigate the problem of multicollinearity in linear regression. The penalty makes giant opposite weights too expensive, so the model shares the credit out instead.

Here is the extreme case. Two columns are exact copies, and the true rule is y = 3 times x. Without a penalty, the maths has no single answer and divides by zero. With alpha = 1, it splits the credit evenly.

Python
def solve(A, b):
    n = len(b)
    M = [row[:] + [b[i]] for i, row in enumerate(A)]
    for c in range(n):
        p = max(range(c, n), key=lambda r: abs(M[r][c]))
        M[c], M[p] = M[p], M[c]
        for r in range(c + 1, n):
            f = M[r][c] / M[c][c]
            for k in range(c, n + 1):
                M[r][k] -= f * M[c][k]
    w = [0.0] * n
    for r in range(n - 1, -1, -1):
        w[r] = (M[r][n] - sum(M[r][k] * w[k] for k in range(r + 1, n))) / M[r][r]
    return w

# Two columns that are exact copies of each other. The true answer: y = 3 * x.
x = [-2.0, -1.0, 0.0, 1.0, 2.0]
y = [3 * v for v in x]
rows = [[v, v] for v in x]

for alpha in [0, 1]:
    A = [[sum(r[i] * r[j] for r in rows) + (alpha if i == j else 0) for j in range(2)] for i in range(2)]
    b = [sum(r[i] * t for r, t in zip(rows, y)) for i in range(2)]
    try:
        print("alpha", alpha, "weights:", [round(v, 3) for v in solve(A, b)])
    except ZeroDivisionError:
        print("alpha", alpha, "no answer: the maths divides by zero")

Output

alpha 0 no answer: the maths divides by zero
alpha 1 weights: [1.429, 1.429]

Each copy gets 1.429, so together they carry 2.858, a little under the true 3. That small gap is the shrinkage at work. Ridge accepts a slightly biased answer to get a stable one. Wikipedia describes the trade as improved efficiency in parameter estimation in exchange for a tolerable amount of bias.

Two rules before you fit

Ridge adds up the weights, so the size of each weight has to mean something. Two habits keep that fair.

  1. 1

    1. Scale the features first

    Say one column is a house's area in square feet (thousands) and another is its number of rooms (a handful). To match the answer, the area column needs a tiny weight and the rooms column a large one. The penalty then punishes the rooms column for nothing but its units. Scale every column to a similar size first, as in the lesson on feature scaling. Scikit-learn, on its penalised models, says the feature matrix should be standardised before fitting, which ensures the penalty treats features equally.

  2. 2

    2. Leave the bias out of the penalty

    The bias (also called the intercept) is the number added at the end, the model's starting point. It says where the line sits, not how much any feature matters. Google's formula sums only the weights w1 to wn, and the bias is not in it. Penalising it would drag the whole line toward zero for no good reason. In the code below, the bias is just the average answer, kept out of the penalty.

Tip

Fit the scaler on the training data only, then reuse its numbers on the test data. The code below does exactly that. Letting test data leak into the scaler is a quiet way to cheat.

Worked example: ridge by hand

Now the full picture, with nothing but standard Python. The data are made up: 12 training points on a curve (a sine wave plus noise), and 200 fresh test points from the same source. A seed is set, so you will get the same numbers.

We fit a degree-9 polynomial, nine weights on 12 points. That is far too flexible. We try alpha = 0, 0.1, 1 and 10, and print the weights, the biggest weight, and the training and test MSE (mean squared error: the average of the squared mistakes, so lower is better).

Python
import math, random

random.seed(4)

def make(n):
    xs = [random.uniform(-1, 1) for _ in range(n)]
    ys = [math.sin(3 * xs[i]) + random.gauss(0, 0.25) for i in range(n)]
    return xs, ys

def features(x, degree=9):
    return [x ** k for k in range(1, degree + 1)]

def solve(A, b):
    # Gaussian elimination with partial pivoting
    n = len(b)
    M = [row[:] + [b[i]] for i, row in enumerate(A)]
    for c in range(n):
        p = max(range(c, n), key=lambda r: abs(M[r][c]))
        M[c], M[p] = M[p], M[c]
        for r in range(c + 1, n):
            f = M[r][c] / M[c][c]
            for k in range(c, n + 1):
                M[r][k] -= f * M[c][k]
    w = [0.0] * n
    for r in range(n - 1, -1, -1):
        w[r] = (M[r][n] - sum(M[r][k] * w[k] for k in range(r + 1, n))) / M[r][r]
    return w

def mse(a, b):
    return sum((p - q) ** 2 for p, q in zip(a, b)) / len(a)

xtr, ytr = make(12)
xte, yte = make(200)
Ftr = [features(x) for x in xtr]
Fte = [features(x) for x in xte]
d = len(Ftr[0])

# Scale first: every column gets mean 0 and spread 1, using training data only
mean = [sum(r[j] for r in Ftr) / len(Ftr) for j in range(d)]
sd = [math.sqrt(sum((r[j] - mean[j]) ** 2 for r in Ftr) / len(Ftr)) for j in range(d)]
Ztr = [[(r[j] - mean[j]) / sd[j] for j in range(d)] for r in Ftr]
Zte = [[(r[j] - mean[j]) / sd[j] for j in range(d)] for r in Fte]
ybar = sum(ytr) / len(ytr)  # the bias: left out of the penalty

def ridge(alpha):
    A = [[sum(r[i] * r[j] for r in Ztr) + (alpha if i == j else 0.0) for j in range(d)] for i in range(d)]
    b = [sum(r[i] * (y - ybar) for r, y in zip(Ztr, ytr)) for i in range(d)]
    return solve(A, b)

for alpha in [0, 0.1, 1, 10]:
    w = ridge(alpha)
    ptr = [ybar + sum(a * b for a, b in zip(r, w)) for r in Ztr]
    pte = [ybar + sum(a * b for a, b in zip(r, w)) for r in Zte]
    print("alpha", alpha)
    print("  weights:", [round(v, 2) for v in w])
    print("  biggest |weight|:", round(max(abs(v) for v in w), 2))
    print("  train MSE:", round(mse(ptr, ytr), 4), " test MSE:", round(mse(pte, yte), 4))

Output

alpha 0
  weights: [-0.04, -7.51, 4.62, 40.76, -1.92, -68.78, -16.65, 35.89, 14.83]
  biggest |weight|: 68.78
  train MSE: 0.0151  test MSE: 114.547
alpha 0.1
  weights: [1.22, 0.05, -0.52, 0.37, -0.36, -0.02, -0.02, -0.28, 0.18]
  biggest |weight|: 1.22
  train MSE: 0.0549  test MSE: 0.1305
alpha 1
  weights: [0.71, 0.17, 0.06, 0.12, -0.11, -0.01, -0.12, -0.11, -0.1]
  biggest |weight|: 0.71
  train MSE: 0.0939  test MSE: 0.1993
alpha 10
  weights: [0.26, 0.07, 0.1, 0.05, 0.02, 0.03, -0.01, 0.02, -0.02]
  biggest |weight|: 0.26
  train MSE: 0.2254  test MSE: 0.3691

Read the table row by row.

  • Alpha = 0: memorised

    Train MSE is tiny, 0.0151, because the curve passes almost through every training point. But the weights are huge (the biggest is 68.78) and the test MSE is 114.547. The model is useless on new data. Exactly the wild swinging from earlier.

  • Alpha = 0.1 and 1: calm and useful

    The biggest weight drops to 1.22, then 0.71. Train MSE rises a little (0.0549, then 0.0939), because the model no longer chases noise. But test MSE falls to 0.1305, then 0.1993, a drop from 114.547 to something sensible.

  • Alpha = 10: too stiff

    Weights are squeezed to at most 0.26, but now the model is too simple. Train MSE is 0.2254 and test MSE climbs to 0.3691. A penalty that is too strong gives underfitting.

Train MSE only goes up as alpha grows: the penalty always costs some fit. The best test score here came at alpha = 0.1, in the middle of the range tried, the U shape from the lesson on tuning. No weight is exactly zero: they get small, but they stay.

Quick check

Quick check

You raise alpha from 0.1 to 10 on a ridge model. Which result do you expect?

What to remember

  • The idea

    Overfitted models tend to have huge weights. Ridge adds a penalty, alpha times the sum of squared weights, to the loss, so big weights have to earn their place.

  • The dial

    Alpha is a hyperparameter. Zero is plain linear regression, larger shrinks the weights more, and too large underfits. Pick it on validation data.

  • The habits

    Scale the features first. Leave the bias out of the penalty. Expect small weights, never zero ones, and expect help when features copy each other.

Next: lasso, the penalty that can switch features off.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.