Machine Learning Foundations / Core / 02.27
Lasso: a penalty that switches features off
Lasso adds a penalty that can shrink weights all the way to zero. Features it zeroes are out of the model.
14 min readFree
A penalty that can switch things off
Ridge regression held every weight back with a gentle squeeze. It made weights small, but none of them reached zero. This lesson is about lasso, a close cousin that squeezes in a different way. Its squeeze is strong enough to push some weights to exactly zero, and a feature with a zero weight no longer affects the answer.
Picture a mixing desk with six sliders, one per feature. Ridge nudges every slider down a little. Lasso will pull a slider all the way to the bottom and leave it there.
Key words
- Weight
- A number the model multiplies a feature by. A big weight means the feature matters a lot. A weight of zero means it is ignored.
- Penalty
- An extra cost added to the model's error. The model has to pay it, so it avoids big weights.
- L1 penalty
- The sum of the absolute values of the weights. Absolute value means the number with its minus sign dropped, so -3 counts as 3.
- Sparse
- Mostly zeros. A sparse model uses only a few of its features.
- Alpha
- The dial that sets how strong the penalty is. A bigger alpha means a harder squeeze.
- Coordinate descent
- A way to fit the model by adjusting one weight at a time, over and over, until nothing changes.
- Standardize
- Rescale a feature so its average is 0 and its typical spread is 1.
The L1 penalty
Ridge adds up the squares of the weights. Lasso adds up the absolute values instead. That is the whole difference in the recipe, and it changes the result a lot.
In words: the cost is the model's average squared error (the first part, with a half in front to keep the maths tidy), plus alpha times the sum of the weights with their signs dropped. Training looks for the weights that make the total as small as possible. The scikit-learn guide writes the same thing.
Wikipedia's page on regularization sums up the pair this way: L1 regularization (also called LASSO) leads to sparse models by adding a penalty based on the absolute value of coefficients, while L2 regularization (also called ridge regression) encourages smaller, more evenly distributed weights by adding a penalty based on the square of the coefficients.
Why weights hit exactly zero
Scikit-learn's guide says it directly: the Lasso is a linear model that estimates sparse coefficients, i.e., it is able to set coefficients exactly to zero. Wikipedia adds the contrast: lasso can set coefficients to zero, while the superficially similar ridge regression cannot. Why the difference? A picture in your head answers it.
Take a model with just two weights, w1 and w2. Draw a flat map. Left to right is w1, bottom to top is w2. Every point on the map is one choice of weights.
- 1
1. Draw the budget
Both penalties can be seen as a budget: the weights may only reach so far from the centre. The set of allowed points is a shape on the map.
- 2
2. Two shapes
Ridge's squared penalty gives a circle, which has no corners. Lasso's absolute-value penalty gives a diamond. Wikipedia describes it as a square rotated so that its corners lie on the axes. Each corner sits where one weight is exactly zero.
- 3
3. Add the error
Without the budget, the best weights sit somewhere out on the map. As you shrink the budget, you slide towards the centre until you first touch the edge of the shape. That touching point is the answer.
- 4
4. See where you touch
The error's contours are like rings spreading out from the best spot. They will most likely first touch a diamond at a pointy corner, because corners stick out the most. At a corner, one weight is zero. A circle gives no such preference.
Wikipedia explains it in the same terms: a shape touching the boundary is likely to encounter a corner, where some of the weights are identically zero, but for a circle, the points where some weights are zero are not distinguished from the others.
There is a second, more practical way to see it, and the code below shows it. Lasso's update for one weight uses a rule called soft-thresholding. It pulls a number towards zero by a fixed amount, and if the number is smaller than that amount, it lands on zero. Scikit-learn puts it like this: the soft-thresholding function is exactly zero whenever alpha is at least the size of the number it is shrinking. A weak feature falls below the line, and the rule sets it to zero.
Quick check
Why can ridge not switch a feature off, while lasso can?
Lasso as built-in feature selection
The lesson on Feature selection showed how to choose which columns a model should use. You can do that with a separate step, or you can let the model do it while it learns. Lasso is the second kind. Wikipedia describes it as a method that performs both variable selection and regularization to improve the model's prediction accuracy and interpretability. The scikit-learn guide agrees: as the Lasso regression yields sparse models, it can thus be used to perform feature selection.
Fewer features
Only features with a non-zero weight affect the prediction, so the model is easier to explain.
A catch
The surviving weights are shrunk too, so they sit below what plain linear regression would give. The example below shows it.
Alpha and the path
Alpha is the squeeze dial. At alpha 0 there is no penalty, and lasso gives the same weights as plain linear regression. As you turn alpha up, weights shrink, and one by one they hit zero. At a high enough alpha, everything is zero and the model just predicts the average.
Plot each weight against alpha and you get lines that slide down and flatten onto zero. That set of lines is called the path. Scikit-learn has a function for it, lasso_path, which computes the coefficients along the full path of possible values. You pick alpha the way the lesson on tuning describes: try several values and keep the one that scores best on validation data. The scikit-learn guide covers setting alpha with cross-validation.
How it is solved: one weight at a time
Wikipedia notes that the loss function of the lasso is not differentiable, so it lists several ways to compute a solution. One of them is coordinate descent, and it is the one scikit-learn's Lasso class uses.
The idea is simple. Instead of solving for all the weights at once, solve for just one, pretending the others are fixed. That small problem is easy. Then move to the next weight. Keep looping until the weights stop changing.
- 1
1. Start at zero
Every weight begins at 0.
- 2
2. Pick one feature
Look at feature j. Work out what is left to explain once all the other features have done their part.
- 3
3. Score its pull
Measure how strongly feature j lines up with what is left. Call that number z.
- 4
4. Soft-threshold
Pull z towards zero by alpha. If z is smaller than alpha, the new weight is exactly 0. Otherwise it is z minus alpha (or plus alpha, if z is negative).
- 5
5. Move on and repeat
Do the same for the next feature. After every feature has had a turn, start the loop again. Stop after the weights settle.
In words: take the size of z, subtract alpha, and if that goes below zero, call it zero. Then put the sign of z back. The scikit-learn guide gives this exact function. It works cleanly here because we standardize the features first, so each one has the same spread.
Worked example: six features, two that matter
Here is the whole thing in plain Python, with no libraries. The data is made up. There are 100 rows and six features: size, age, rooms, floor, noise1 and noise2. Only two truly drive the answer. The target is 4 times size, minus 3 times floor, plus a little random noise. The other four features are pure randomness. A good lasso should keep size and floor, and drop the rest. The seed is fixed, so you get the same numbers on every run. The code standardizes each column, then runs coordinate descent for six alpha values.
import random
random.seed(3)
n, p = 100, 6
names = ["size", "age", "rooms", "floor", "noise1", "noise2"]
# Made-up data: only "size" (true weight 4) and "floor" (true weight -3) matter
X = [[random.gauss(0, 1) for _ in range(p)] for _ in range(n)]
y = [4 * r[0] - 3 * r[3] + random.gauss(0, 1) for r in X]
# Scale first: every column gets mean 0 and spread 1; centre y too
for j in range(p):
m = sum(r[j] for r in X) / n
s = (sum((r[j] - m) ** 2 for r in X) / n) ** 0.5
for r in X:
r[j] = (r[j] - m) / s
my = sum(y) / n
y = [v - my for v in y]
def soft(z, a):
# shrink z towards 0 by a; if |z| <= a the answer is exactly 0
if z > a:
return z - a
if z < -a:
return z + a
return 0.0
def lasso(alpha, rounds=200):
w = [0.0] * p
for _ in range(rounds):
for j in range(p):
# residual with feature j's own part put back
z = sum(X[i][j] * (y[i] - sum(X[i][k] * w[k] for k in range(p) if k != j)) for i in range(n)) / n
w[j] = soft(z, alpha)
return w
print("feature " + " ".join(f"{nm:>7}" for nm in names))
for alpha in [0.0, 0.1, 0.5, 1.0, 2.0, 4.0]:
w = lasso(alpha)
kept = sum(1 for v in w if v != 0)
print(f"alpha={alpha:<4}" + " ".join(f"{v:7.3f}" for v in w) + f" kept={kept}")Output
feature size age rooms floor noise1 noise2 alpha=0.0 3.881 -0.045 -0.066 -2.830 -0.055 0.156 kept=6 alpha=0.1 3.775 0.000 0.000 -2.701 0.000 0.044 kept=3 alpha=0.5 3.368 0.000 0.000 -2.286 0.000 0.000 kept=2 alpha=1.0 2.856 0.000 0.000 -1.775 0.000 0.000 kept=2 alpha=2.0 1.833 0.000 0.000 -0.751 0.000 0.000 kept=2 alpha=4.0 0.000 0.000 0.000 0.000 0.000 0.000 kept=0
Read the table from top to bottom. At alpha 0.0 there is no penalty, so all six weights are non-zero. Size is 3.881 and floor is -2.830. Notice that the four useless features get small non-zero weights, such as 0.156 for noise2. Plain regression always fits a little noise.
At alpha 0.1, three weights switch off. Only noise2 (0.044) still slips through. At alpha 0.5, it goes too, and just size and floor remain, which matches how the data was made. Between alpha 0.5 and 2.0 the two real weights shrink steadily: size goes 3.368, 2.856, 1.833, and floor goes -2.286, -1.775, -0.751. At alpha 4.0, even they give up, and all six weights are zero.
Notice the shrinkage cost. The true weights are 4 and -3, but even at alpha 0.5 lasso reports 3.368 and -2.286. That is the price of the squeeze.
The weak spot: near-copy features
Lasso has one well-known weakness. Say two features are almost the same thing, such as a flat's size in square metres and the same size in square feet. Both carry the same information. Lasso tends to keep one of them and push the other to zero, and which one wins can come down to small accidents in the data.
Wikipedia describes the extreme case. If two features are identical for every row, the best weights for them are not uniquely determined: you can slide weight from one to the other and the lasso cost stays the same, so there is a continuum of valid answers. It adds that several variants of the lasso, including the Elastic net, were designed to address this shortcoming.
Here is a small test. The feature twin is size plus a little noise (0.05), so it is a near copy. The target depends on size alone. For four different random seeds, the code fits lasso at alpha 2.0 on just these two features.
import random
n = 100
alpha = 2.0
def scale(col):
m = sum(col) / len(col)
sd = (sum((v - m) ** 2 for v in col) / len(col)) ** 0.5
return [(v - m) / sd for v in col]
def soft(z, a):
return z - a if z > a else z + a if z < -a else 0.0
for seed in [1, 2, 3, 4]:
random.seed(seed)
size = [random.gauss(0, 1) for _ in range(n)]
twin = [s + random.gauss(0, 0.05) for s in size] # a near copy of size
y = [4 * s + random.gauss(0, 1) for s in size]
my = sum(y) / n
y = [v - my for v in y]
cols = [scale(size), scale(twin)]
w = [0.0, 0.0]
for _ in range(300):
for j in range(2):
o = 1 - j
z = sum(cols[j][i] * (y[i] - cols[o][i] * w[o]) for i in range(n)) / n
w[j] = soft(z, alpha)
print(f"seed {seed}: size = {w[0]:6.3f} twin = {w[1]:6.3f}")Output
seed 1: size = 1.739 twin = 0.000 seed 2: size = 2.417 twin = 0.000 seed 3: size = 2.300 twin = 0.000 seed 4: size = 1.345 twin = 0.352
In three of the four runs, the twin gets exactly 0 and size takes all the weight. In seed 4, the weight is split: 1.345 for size and 0.352 for twin. The picks are not stable. Read as "which feature matters", this could fool you: the twin matters just as much, and the model simply chose one.
Tip
Before you trust a lasso result, look for features that are copies of each other, such as the same measurement in two units. If you find some, ElasticNet is the better tool. It is the next lesson.
Scale first
The penalty adds up weights and treats them all alike. That is only fair if the features are on the same scale. Say one feature is a flat's area in square metres (about 50) and another is its number of rooms (about 3). To have equal say, rooms needs the larger weight. The penalty charges more for that, so lasso may drop the feature that happens to have small numbers.
The fix is to standardize every feature before fitting, as the code above does. Wikipedia notes that the covariates are typically standardized so that the solution does not depend on the measurement scale. The lesson on feature scaling covers how. Scale using the training data only, then reuse the same numbers on the test data.
Quick check
You fit lasso on two raw features: area in square metres (values near 50) and rooms (values near 3). What can go wrong?
What to remember
The penalty
Lasso adds alpha times the sum of the absolute weights to the error. The diamond-shaped budget has corners on the axes, which is why some weights land on exactly zero.
The use
Lasso doubles as feature selection. Turn alpha up and more weights switch off. The kept weights are shrunk, so refit if you need unshrunk numbers.
The care
Standardize first, tune alpha on validation data, and watch for near-copy features, where lasso keeps one and may drop the other.
Next: ElasticNet, mixing ridge and lasso.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.