esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.28

ElasticNet: ridge and lasso in one

ElasticNet mixes the ridge and lasso penalties. It drops useless features and still keeps groups of look-alike features together.

13 min readFree

Why mix two penalties?

The last lesson ended with a promise: ElasticNet, mixing ridge and lasso. Here it is.

Think of two coaches. One (ridge) tells every player on the team to run a little slower, so nobody burns out. The other (lasso) benches some players completely. Both ideas are useful. ElasticNet hires both coaches and lets them share the job.

Both ideas are fixes for the same problem: a model that memorises its training data (overfitting). The lessons on Ridge and Lasso cover them properly. This lesson only needs the short version below.

Key words

Regularization
A penalty added to training that keeps the model's weights small, so it cannot chase noise.
Weight
The number a model multiplies a feature by. A weight of zero means the feature is ignored.
L2 penalty
Adds up the squares of the weights. Ridge uses it. It shrinks every weight but rarely makes one exactly zero.
L1 penalty
Adds up the sizes of the weights, ignoring the sign. Lasso uses it. It can push weights to exactly zero.
Alpha
The dial for overall strength. More alpha, more shrinking.
l1_ratio
The dial for the mix: how much of the penalty is L1 and how much is L2.
Correlated features
Two features that move together, such as height in centimetres and height in inches.

If a word is still fuzzy, the Linear Regression lesson explains weights and error, and the Tuning lesson explains why settings like alpha are chosen by you, not learned.

Ridge and lasso in one minute

Ridge and lasso both start from ordinary linear regression and add a price tag on big weights. The model now has two goals: fit the data, and keep the price low.

  • Ridge (L2)

    Shrinks all the weights a little. In scikit-learn's words, a larger alpha means more shrinkage, so the coefficients become more robust to collinearity (features that overlap).

  • Lasso (L1)

    Scikit-learn says it estimates sparse coefficients, which means it is able to set coefficients exactly to zero. That quietly removes features.

  • The catch with lasso

    Give it a group of look-alike features and it tends to keep one and drop the rest. Which one it keeps can be close to a coin flip.

The ElasticNet penalty

ElasticNet adds both penalties together. Wikipedia describes it as a regularized regression method that linearly combines the L1 and L2 penalties of the lasso and ridge methods. "Linearly combines" just means a blend with two amounts, like mixing orange squash: some concentrate (L1), some water (L2).

Here is the recipe scikit-learn minimizes. The first part is the usual error. The rest is the price tag.

min⁡w  12n∥Xw−y∥22  +  αρ ∥w∥1  +  α(1−ρ)2∥w∥22\min_{w} \; \frac{1}{2n} \lVert Xw - y \rVert_2^2 \; + \; \alpha \rho \, \lVert w \rVert_1 \; + \; \frac{\alpha (1 - \rho)}{2} \lVert w \rVert_2^2

In words: pick the weights w that make the error small, plus alpha times rho times the sum of the weights' sizes (the L1 part), plus alpha times one minus rho, halved, times the sum of the squared weights (the L2 part). Here n is the number of rows, X holds the features, y holds the answers, and rho is the l1_ratio.

The user guide calls the mix a convex combination of the L1 and L2 penalties. You do not need that term. Just read rho as a slider between two ends.

The two dials

ElasticNet gives you two settings. Both are hyperparameters, so you choose them before training.

  1. 1

    1. alpha: how hard to press

    Scikit-learn calls it the constant that multiplies the penalty terms, with a default of 1.0. Alpha of 0 is the same as plain least squares with no penalty. Bigger alpha means smaller weights and a simpler model.

  2. 2

    2. l1_ratio: which coach leads

    A number from 0 to 1. At 0 the penalty is pure L2, which is ridge. At 1 it is pure L1, which is lasso. Anything in between is a blend. The default is 0.5, an even split.

  3. 3

    3. Read them together

    Alpha sets how much total penalty. The ratio sets what it is made of. Moving one does not undo the other.

Wikipedia notes that lasso and ridge are both special cases of the elastic net, so you lose nothing by trying it. The user guide also warns about naming: the l1_ratio parameter corresponds to alpha in the glmnet R package, while alpha corresponds to the lambda parameter in glmnet. If you read papers or R code, watch for the swap.

Tip

Scikit-learn says l1_ratio of 0.01 or lower is not reliable unless you supply your own sequence of alpha values. For a nearly pure ridge, use the Ridge model instead.

When ElasticNet is the right pick

Reach for ElasticNet when you have many features and some of them move together. The user guide gives the reason in two lines: elastic-net is useful when there are multiple features that are correlated with one another, and lasso is likely to pick one of these at random, while elastic-net is likely to pick both.

Wikipedia names two limits of lasso that the elastic net was built to overcome. If there is a group of highly correlated variables, lasso tends to select one variable from a group and ignore the others. And when you have far more features than rows (what statisticians call large p, small n), lasso selects at most n variables before it saturates.

A daily-life picture: you are packing a bag for a trip and the forecast says rain. Three rain jackets would be wasteful, so lasso packs one. But if all your jackets are almost the same and you cannot tell which one is best, ElasticNet packs a little of each. Splitting the weight between the twins also makes the answer steadier: the user guide says the mix lets ElasticNet inherit some of ridge's stability under rotation.

Tip

Wikipedia says elastic net regularization is typically more accurate than both methods with regard to reconstruction. "Typically" is not "always". On your data, check with held-out scores, as the Tuning lesson explains.

Ridge, lasso and ElasticNet side by side

  • Ridge: l1_ratio = 0

    Shrinks everything and zeroes nothing. Shares weight across look-alike features. Keeps every feature in the model.

  • Lasso: l1_ratio = 1

    Shrinks and zeroes. Picks one feature from a look-alike group. Gives a short, readable model.

  • ElasticNet: 0 < l1_ratio < 1

    Shrinks and zeroes. Tends to keep look-alike features together. Costs you one extra dial to tune.

Quick check

You have 200 features, and many of them are near-copies of each other. You want a model that drops useless features but does not randomly pick one copy from each group. Which fits best?

Watch it happen

Words are easy to say, so let us test them. This script uses only Python's standard library, so there is nothing to install. It makes up 60 rows of data with a fixed random seed, so you get the same numbers every time.

The made-up world has six features. Two of them, twin_a and twin_b, are near-copies of one hidden signal (they differ by a little noise). The other four are pure noise that has nothing to do with the answer. The answer itself is three times the hidden signal, plus noise.

We fit three models with the same alpha of 2.0 and compare the weights. The fitting method is coordinate descent, which scikit-learn also uses. It is simple: update one weight at a time, holding the others still, and loop until nothing changes.

  1. 1

    1. Make the data

    One hidden signal, two noisy copies of it, four noise columns, and an answer built from the signal.

  2. 2

    2. Put features on one scale

    Each column is shifted to an average of 0 and stretched to a spread of 1. More on why below.

  3. 3

    3. Update one weight at a time

    For each feature, look at what the other features have not explained yet, then nudge this weight. The soft function does the L1 shrinking and snaps small values to exactly zero. The division does the L2 shrinking.

  4. 4

    4. Repeat 500 times, for three l1_ratio values

    1.0 is lasso, 0.0 is ridge, and 0.5 is ElasticNet.

Python
import random

random.seed(7)
n = 60

# One hidden signal z. Two features are near-copies of it (twins); four are pure noise.
z = [random.gauss(0, 1) for _ in range(n)]
cols = {
    "twin_a": [v + random.gauss(0, 0.1) for v in z],
    "twin_b": [v + random.gauss(0, 0.1) for v in z],
}
for k in range(1, 5):
    cols[f"noise_{k}"] = [random.gauss(0, 1) for _ in range(n)]
y = [3 * v + random.gauss(0, 1) for v in z]

def standardize(c):
    m = sum(c) / len(c)
    s = (sum((v - m) ** 2 for v in c) / len(c)) ** 0.5
    return [(v - m) / s for v in c]

names = list(cols)
X = [standardize(cols[k]) for k in names]   # X[j][i]: feature j, row i
ym = sum(y) / n
yc = [v - ym for v in y]

def soft(r, t):
    # shrink r toward zero by t; stop at exactly zero
    return 0.0 if abs(r) <= t else (abs(r) - t) * (1 if r > 0 else -1)

def fit(alpha, l1_ratio, sweeps=500):
    w = [0.0] * len(names)
    for _ in range(sweeps):
        for j in range(len(names)):
            # residual with feature j's own part left out
            r = [yc[i] - sum(w[k] * X[k][i] for k in range(len(names)) if k != j) for i in range(n)]
            rho = sum(X[j][i] * r[i] for i in range(n)) / n
            w[j] = soft(rho, alpha * l1_ratio) / (1 + alpha * (1 - l1_ratio))
    return w

print("correlation of twins:", round(sum(a * b for a, b in zip(X[0], X[1])) / n, 3))
print(f"{'':10}{'lasso':>9}{'ridge':>9}{'elastic':>9}")
lasso = fit(2.0, 1.0)
ridge = fit(2.0, 0.0)
enet = fit(2.0, 0.5)
for j, name in enumerate(names):
    print(f"{name:10}{lasso[j]:9.3f}{ridge[j]:9.3f}{enet[j]:9.3f}")
print("twin sum :", *(f"{w[0] + w[1]:9.3f}" for w in (lasso, ridge, enet)))

Output

correlation of twins: 0.985
              lasso    ridge  elastic
twin_a        0.642    0.662    0.554
twin_b        0.000    0.656    0.542
noise_1       0.000    0.045    0.000
noise_2       0.000   -0.050    0.000
noise_3       0.000   -0.016    0.000
noise_4       0.000   -0.015    0.000
twin sum :     0.642     1.318     1.096

Reading the table

First, the twins are almost the same column: their correlation is 0.985, where 1.0 would be identical. Now read down the three columns.

  • Lasso

    Keeps twin_a at 0.642 and sets twin_b to exactly 0.000. It picked one twin and dropped the other. It also zeroed all four noise features, which is what we hoped for.

  • Ridge

    Splits the weight almost evenly (0.662 and 0.656), which is good. But it leaves small non-zero weights on the noise features, such as 0.045 and -0.050. Ridge never fully switches a feature off.

  • ElasticNet

    Splits the twins almost evenly (0.554 and 0.542) and zeroes all four noise features. It did both jobs.

Notice that the true link is the same for both twins: each one carries the same signal, so any split between them fits the data about as well. Lasso is free to put everything on one. The squared part of ElasticNet's penalty makes an even split cheaper than a lopsided one, so the weight is shared.

One more thing in the output: the twin sums are 0.642 for lasso, 1.318 for ridge and 1.096 for ElasticNet. They are all well below the true effect because alpha 2.0 is a strong penalty, and shrinking is the whole point. Pushing too hard is how you underfit, which is why the next section matters.

Tip

This is a toy. With one random seed, one dataset and a hand-picked alpha, the exact numbers prove nothing about your data. The pattern is the lesson: lasso picks one twin, ElasticNet shares.

Scale your features first

Look back at step 2 of the code. It put every feature on the same scale before fitting. This is not a detail. The penalty adds up the sizes of the weights, and a weight's size depends on the units of its feature. A feature measured in millimetres needs a tiny weight, and one measured in kilometres needs a big one. Without a common scale, the penalty would treat them unfairly, squeezing some features more than others for no good reason.

The Feature Scaling lesson shows how. In practice you scale with a tool that learns the average and spread from the training rows only, then reuses them on test rows, so no information leaks. The Preprocessing lesson covers keeping those steps together.

Quick check

Why scale features before fitting an ElasticNet?

Setting alpha and l1_ratio

Two dials means more combinations to try. The Tuning lesson explains the whole search loop, so we will not repeat it. What is special about ElasticNet is that scikit-learn has a helper built for it.

Scikit-learn's user guide says the class ElasticNetCV can be used to set the parameters alpha and l1_ratio by cross-validation (splitting the training data several times and averaging the scores). Its l1_ratio can be a list, in which case the different values are tested by cross-validation and the one giving the best prediction score is used.

  1. 1

    1. Scale the features

    Fit the scaler on training rows only.

  2. 2

    2. Give a short list of ratios

    The ElasticNetCV page suggests putting more values close to 1 (lasso) and fewer close to 0 (ridge), as in [.1, .5, .7, .9, .95, .99, 1].

  3. 3

    3. Let it search alpha for you

    For each ratio it tries a whole path of alpha values (100 by default) and scores them.

  4. 4

    4. Check once on the test set

    Do this at the very end, as the Tuning lesson says.

In practice you hand ElasticNetCV your list of ratios and the number of splits you want, give it the scaled training data, and it reports the alpha and ratio that scored best. We did not run it here, because it needs scikit-learn and our example uses only plain Python.

Tip

Start simple. If you cannot afford to tune two dials, set l1_ratio to 0.5 and tune only alpha. You can widen the search later.

Quick recap

  • What it is

    A linear model whose penalty blends L1 (lasso) and L2 (ridge), controlled by alpha for strength and l1_ratio for the mix.

  • When to use it

    Many features, especially groups of correlated ones, where you still want some features switched off.

  • How to use it

    Scale first, then tune both dials with cross-validation, and check once at the end on the untouched test set.

Quick check

In scikit-learn, what does l1_ratio = 1 give you?

Next: decision trees and random forests, models that ask yes-or-no questions.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.