Machine Learning Foundations / Core / 02.03
Feature selection
More columns do not always mean a better model. Feature selection is how we keep the helpful ones and drop the rest.
14 min readFree
Why keep fewer features?
Picture packing a bag for a day trip. You could throw in everything you own. But a heavy bag slows you down, and you spend ten minutes hunting for your keys. A smart traveller packs only what the trip needs.
A model has the same problem. In the last lessons we met features: the columns of numbers a model uses as input, such as the size of a house or its number of rooms. A table can have a few columns or thousands. Feature selection is the job of choosing which of them to keep. Wikipedia defines it as 'the process of selecting a subset of relevant features (variables, predictors) for use in model construction.'
Why not just feed in everything and let the model sort it out? Wikipedia lists several reasons to select features. Two of them are 'simplification of models to make them easier to interpret' and 'shorter training times'. scikit-learn's guide adds that selection 'can help remove irrelevant, redundant and noisy features as well as improve your model build time and performance.'
Let's turn those into plain words.
Less noise
Some columns tell the model nothing useful about the answer. Think of a door-colour code on a house list when you want the price. A model can still try to find a pattern in it, and any pattern it finds there is luck. Dropping the column removes the temptation.
Faster and simpler
Fewer columns means less to store and less to train on. It also means fewer things to explain. If a model uses five features, you can name all five to your client. If it uses five hundred, you cannot.
Less duplication
Two columns can say the same thing in different words, like a size in square metres and the same size in square feet. Wikipedia says the central idea is that data 'contains features that are redundant or irrelevant, and can thus be removed without incurring much loss of information.'
Key words
- Feature selection
- Choosing a smaller set of the original columns to keep. The columns you keep stay exactly as they were.
- Irrelevant feature
- A column that does not help predict the answer, such as a random ID number.
- Redundant feature
- A column that repeats what another column already says.
- Variance
- How spread out a column's numbers are. A column where every row has the same value has a variance of zero.
- Correlation
- How closely two columns move together. We met it in the descriptive statistics lesson.
- Training data
- The rows the model learns from. The test data is held back to check the model later.
One thing to keep straight: feature selection is not the same as making new features. Wikipedia says that feature extraction 'creates new features from functions of the original features, whereas feature selection finds a subset of the features.' Selection picks from the shelf. Extraction builds something new. The next lesson covers the second idea.
Three families of methods
You may wonder: why not just try every possible group of features and keep the best? Wikipedia says this 'exhaustive search' is 'computationally intractable for all but the smallest of feature sets.' With only 30 columns there are over a billion groups to test. So people use shortcuts, and the shortcuts fall into three families, which Wikipedia names 'wrappers, filters and embedded methods'.
Filter methods
Score each feature on its own, before any model is trained, then keep the best scorers. Wikipedia says filters use 'a proxy measure instead of the error rate', chosen to be fast. They are 'usually less computationally intensive than wrappers', but the set they produce is not tuned to one kind of model.
Wrapper methods
Try groups of features with a real model and see how well each group works. Wikipedia says they are 'very computationally intensive, but usually provide the best performing feature set for that particular type of model'.
Embedded methods
The model picks features while it learns. Wikipedia calls them 'a catch-all group of techniques which perform feature selection as part of the model construction process.'
A kitchen helps here. A filter is like a quick look in the fridge: throw out anything that is clearly spoiled before you start. A wrapper is cooking the dish again and again with different ingredients and tasting each time. An embedded method is a cook who drops ingredients as they cook, because the dish itself shows what is not needed.
Quick check
Which family trains a new model for every group of features it tries?
Filter methods
Filters are the easiest place to start. Two simple filters need nothing more than school maths.
The first looks at variance. scikit-learn calls its VarianceThreshold 'a simple baseline approach to feature selection.' It 'removes all features whose variance doesn't meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.' This makes sense. If every house in your table has the same country code, that column cannot help tell one house from another. Python's own docs describe variance this way: 'A large variance indicates that the data is spread out; a small variance indicates it is clustered closely around the mean.'
The second looks at correlation with the label. The label is the answer we want to predict, such as the price. Wikipedia says filters are 'based only on general features like the correlation with the variable to predict.' A feature that moves closely with the price is a good candidate to keep. Python's statistics docs say Pearson's correlation coefficient 'takes values between -1 and +1. It measures the strength and direction of a linear relationship.' Close to +1 or -1 means a strong link. Close to 0 means almost no straight-line link.
Wikipedia also names other scores for filters, such as mutual information. We keep to correlation here because you can compute it by hand.
Tip
Our simple filters look at one column at a time, so they cannot tell that two weak columns might work well together. That is the price of being fast.
Wrapper methods
A wrapper 'wraps' a real model around the search. It tries a group of features, trains the model and checks the score on held-back data. Then it tries another group.
scikit-learn describes two simple ways to walk through the choices, both called sequential feature selection. 'Forward-SFS is a greedy procedure that iteratively finds the best new feature to add to the set of selected features.' You start with no features, add the single best one, then the next best, and stop when you have as many as you want. 'Backward-SFS follows the same idea but works in the opposite direction: instead of starting with no features and greedily adding features, we start with all the features and greedily remove features from the set.' Greedy just means it takes the best move at each step and never looks back.
A third wrapper idea is recursive feature elimination, or RFE. scikit-learn's guide says its goal 'is to select features by recursively considering smaller and smaller sets of features.' You train a model, which gives each feature a weight showing how important it is. Then 'the least important features are pruned from the current set of features.' You repeat on the smaller set until you reach the number you want.
Wrappers have costs. Wikipedia lists two main disadvantages: 'The increasing overfitting risk when the number of observations is insufficient' and 'The significant computation time when the number of variables is large.' Overfitting means the model learned the quirks of the training rows, not the real pattern.
Embedded methods
The best-known embedded method uses L1 penalties, also called Lasso. You do not need the maths yet; a later lesson goes deep. The idea is simple. While the model learns, it is charged a small fee for every weight that is not zero. So it learns to set the weights of unhelpful features to exactly zero.
scikit-learn says: 'Linear models penalized with the L1 norm have sparse solutions: many of their estimated coefficients are zero.' A sparse answer is one that is mostly zeros. Its guide adds that the features with non-zero coefficients can be kept using a tool called SelectFromModel. Wikipedia describes Lasso as a method that 'penalizes the regression coefficients with an L1 penalty, shrinking many of them to zero.' The features that stay non-zero are the ones 'selected'.
Quick check
In an L1 (Lasso) model, what does a coefficient of zero for a feature mean for selection?
Worked example: eight houses
Let's choose features for a made-up table of eight houses. We want to predict the price in thousands of dollars. The numbers are invented for this lesson. There are five candidate features: size in square metres, the same size in square feet, number of rooms, a country code (all our houses are in one country), and a door-colour code that a clerk filled in at random.
Python's standard library has what we need. statistics.pvariance gives the population variance, and statistics.correlation gives Pearson's r. No other packages are needed.
- 1
1. Put the table in code
Each feature is a list with one number per house, in the same order. The price list uses the same order.
- 2
2. Filter by variance
Compute the variance of every feature. A variance of zero means the column never changes.
- 3
3. Filter by correlation with the label
For each feature that survived, compute r against the price.
- 4
4. Check for near-duplicates
Compute r between the surviving features. A value near 1 means one of the two is redundant.
from statistics import correlation, pvariance
table = {
"size_m2": [50, 62, 75, 80, 95, 110, 120, 140],
"size_ft2": [538, 667, 807, 861, 1023, 1184, 1292, 1507],
"rooms": [2, 3, 3, 4, 4, 5, 4, 6],
"country": [1, 1, 1, 1, 1, 1, 1, 1],
"door_code": [7, 2, 9, 4, 1, 8, 3, 5],
}
price = [150, 190, 220, 250, 280, 340, 350, 420]
print("variance of each feature")
for name, col in table.items():
print(f"{name:10} {pvariance(col):10.2f}")
print()
print("correlation with price")
for name, col in table.items():
if pvariance(col) > 0:
print(f"{name:10} {correlation(col, price):6.3f}")
print()
print("size_m2 vs size_ft2:", round(correlation(table["size_m2"], table["size_ft2"]), 3))
print("size_m2 vs rooms:", round(correlation(table["size_m2"], table["rooms"]), 3))Output
variance of each feature size_m2 814.50 size_ft2 94471.36 rooms 1.36 country 0.00 door_code 7.36 correlation with price size_m2 0.997 size_ft2 0.997 rooms 0.943 door_code -0.090 size_m2 vs size_ft2: 1.0 size_m2 vs rooms: 0.918
Now read the output like a detective.
Variance step
The country column has a variance of 0.00, so it is the same in every row. Drop it. Notice the other variances differ a lot, but that is mostly because the columns use different units, so we only use this step to catch the zero. Comparing raw variances across columns is unfair, which is one reason the feature scaling lesson mattered.
Correlation step
size_m2 and size_ft2 both score 0.997 with the price, and rooms scores 0.943. The door_code scores -0.090, which is close to zero. In this table, it has almost no straight-line link to the price. Drop it.
Duplicate step
size_m2 and size_ft2 have r = 1.0 with each other. They carry the same information in two units, so keep only one. The pair size_m2 and rooms is 0.918, high but not identical, so rooms may still add something. That call needs a model to test it.
We went from five features to two: size_m2 and rooms. The final choice for rooms is a judgment call. Wikipedia reminds us that 'one relevant feature may be redundant in the presence of another relevant feature with which it is strongly correlated.' A wrapper method could settle it by training with and without rooms.
Wikipedia also notes that filters 'tend to select redundant variables when they do not consider the relationships between variables', and that more elaborate ones try to fix this by 'removing variables highly correlated to each other'. Our duplicate step did exactly that by hand.
Tip
Correlation only sees straight-line links, as the descriptive statistics lesson warned. A feature with r near 0 could still be linked to the label in a curve. With only eight rows, treat all of these numbers as a demonstration, not as proof.
When a feature looks too good
Sometimes a feature scores very high and you feel lucky. Be careful. A suspiciously perfect score can mean that the feature contains the answer in disguise.
scikit-learn defines this problem, called data leakage: 'Data leakage occurs when information that would not be available at prediction time is used when building the model.' It leads to 'overly optimistic performance estimates'.
Here is a simple case. Suppose our house table had a column with the price including tax. When would you know that number? Only after you already know the price. At prediction time, you would not have it. Look at what correlation does with it.
from statistics import correlation
price = [150, 190, 220, 250, 280, 340, 350, 420]
price_with_tax = [round(p * 1.2) for p in price]
size_m2 = [50, 62, 75, 80, 95, 110, 120, 140]
print("size_m2 vs price:", round(correlation(size_m2, price), 3))
print("price_with_tax vs price:", round(correlation(price_with_tax, price), 3))Output
size_m2 vs price: 0.997 price_with_tax vs price: 1.0
A filter would happily pick price_with_tax, since it scores a perfect 1.0. A model trained on it would look brilliant in testing and be useless in real life. Ask one question of every top-scoring feature: will I really have this value at the moment I need the prediction? If not, remove it.
Quick check
A column called 'refund_issued' scores almost 1.0 with the label 'customer_left'. Refunds are only issued after a customer leaves. What should you do?
Select inside the training data only
There is one more rule, and beginners break it all the time. Feature selection must only look at the training rows.
scikit-learn says it plainly: 'As with any other type of preprocessing, feature selection should only use the training data. Including the test data in feature selection will optimistically bias your model.' Its guide shows a striking case. It builds a dataset of 200 rows with 10,000 random features, where the answers are also random, so nothing can truly be predicted. Selecting features on all the data first still gave 'an accuracy score much higher than chance'. The selection step 'sees' the test data.
Think of an exam. If you peek at the exam questions while choosing what to study, your score on that exam no longer tells you how you will do on a new one. The test rows are the exam. Keep them sealed until the features are chosen.
The same guide gives the safe order: split the data into train and test first, and only then do any preprocessing or selection. It also says 'The scikit-learn pipeline is a great way to prevent data leakage'. Here is our correlation filter again, run on the first six houses only, as if the last two were the test set.
from statistics import correlation
rooms = [2, 3, 3, 4, 4, 5, 4, 6]
door_code = [7, 2, 9, 4, 1, 8, 3, 5]
price = [150, 190, 220, 250, 280, 340, 350, 420]
train = slice(0, 6) # the first 6 houses are the training rows
for name, col in [("rooms", rooms), ("door_code", door_code)]:
r = correlation(col[train], price[train])
print(f"{name:10} r on training rows = {r:6.3f}")Output
rooms r on training rows = 0.977 door_code r on training rows = 0.019
The choice is the same here: keep rooms, drop door_code. But the numbers came only from the training rows, so the test rows stay a fair judge.
Tip
Scaling, filling gaps and feature selection all follow the same rule: learn from the training rows, then apply the result to the test rows.
A checklist
Here is a short routine to follow when you face a wide table.
- 1
1. Split first
Set the test rows aside before any selection.
- 2
2. Remove the obvious
Drop columns with zero variance, and columns that could not be known at prediction time.
- 3
3. Filter
Keep features with a real link to the label, and drop near-duplicates of each other.
- 4
4. Test with a model
Use a wrapper or an embedded method to check that dropping a feature does not hurt the score on held-back data.
- 5
5. Explain it
Write down why each kept feature is there. If you cannot explain it, look again.
Next: dimensionality reduction.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.