Machine Learning Foundations / Core / 02.01
Feature engineering: turning raw columns into signals
A model can only learn from the columns you give it. Learn how to make better ones: ratios, dates, history, buckets, categories, word counts and crosses, and how to avoid the leak that spoils them.
14 min readFree
What feature engineering is
In the last lesson we cleaned our data: we fixed typos, filled gaps and removed duplicates. Now the data is tidy, but tidy is not the same as useful. Think of a chef with a bag of groceries. Clean potatoes and raw onions are fine, but the meal gets better when someone chops, mixes and seasons them. Feature engineering is that kitchen work, done on columns.
Wikipedia says feature engineering is 'a preprocessing step which transforms raw data into a more effective set of inputs', and that each input 'comprises several attributes, known as features'. It adds that by 'providing models with relevant information, feature engineering significantly enhances their predictive accuracy and decision-making capability.' In short: you make the columns easier for a model to learn from.
Key words
- Feature
- One column of input that the model looks at, such as the total of an order. The thing it tries to predict is the label.
- Raw column
- A column exactly as it was collected, such as a date-and-time written as text.
- Derived feature
- A new column you build from raw ones, for example price per item made from the total and the item count.
- Bin (or bucket)
- A range of numbers treated as one group, such as 'orders under 30'.
- One-hot encoding
- Turning a category into a row of 0s and 1s, with a single 1 marking which category it is.
- Data leakage
- Using information that would not exist yet at the moment you make a real prediction.
Wikipedia also warns that feature engineering 'can be a time-consuming and error-prone process, as it requires domain expertise and often involves trial and error'. Domain expertise just means knowing the subject. Someone who runs a shop knows things about orders that no column name tells you. So treat the ideas below as things to try, then check.
Our practice table: six shop orders
We will use one tiny, made-up table of orders from an online shop. Each row is one order. We want a model to guess, at the time an order is placed, whether it will be returned later. That makes 'returned' our label. Everything else we build must be known at the moment of the order. Hold on to that rule; the last third of the lesson is about it.
The orders are plain Python dictionaries. Only the standard library is used, so you can run every block yourself. Later blocks continue this same script, so keep the same file.
Ratios, differences and date parts
The simplest new feature divides or subtracts two columns you already have. A model that sees total = 120 and items = 1 does not automatically see 'one expensive item'. If you hand it price per item, it can.
Dates are the same story. A timestamp like 2026-03-06 22:40 is one long label. Inside it hide smaller facts: the day of the week, the hour and whether it is a weekend. Python's datetime module reads the text into a date object, and then each part is one line. Whether these parts help is a question for your data; we check that at the end.
from datetime import datetime
orders = [
{"id": 1, "who": "asha", "at": "2026-03-02 09:15", "total": 40.0, "items": 2, "pay": "card", "note": "gift for mum"},
{"id": 2, "who": "ben", "at": "2026-03-06 22:40", "total": 120.0, "items": 1, "pay": "card", "note": "late night treat"},
{"id": 3, "who": "asha", "at": "2026-03-14 10:05", "total": 25.5, "items": 3, "pay": "upi", "note": "gift"},
{"id": 4, "who": "chloe", "at": "2026-03-15 23:10", "total": 90.0, "items": 1, "pay": "voucher", "note": "late night gift for me"},
{"id": 5, "who": "asha", "at": "2026-03-21 08:50", "total": 30.0, "items": 2, "pay": "card", "note": ""},
{"id": 6, "who": "ben", "at": "2026-03-22 21:30", "total": 75.0, "items": 1, "pay": "upi", "note": "late night treat"},
]
for o in orders:
when = datetime.strptime(o["at"], "%Y-%m-%d %H:%M")
o["when"] = when
o["price_per_item"] = round(o["total"] / o["items"], 2)
o["weekday"] = when.strftime("%A")
o["hour"] = when.hour
o["is_weekend"] = when.weekday() >= 5
for o in orders:
print(o["id"], o["price_per_item"], o["weekday"], o["hour"], o["is_weekend"])Output
1 20.0 Monday 9 False 2 120.0 Friday 22 False 3 8.5 Saturday 10 True 4 90.0 Sunday 23 True 5 15.0 Saturday 8 True 6 75.0 Sunday 21 True
Tip
Differences work like ratios. Delivery date minus order date gives 'days to deliver'. Always ask: what would a person look at to judge this row? Then write that down as a column.
Time since an event, and counts per group
Two more ideas help with almost any table that has people or things in it. The first is time since an event: how many days since this customer last ordered? The second is a group summary: how many orders has this customer made, and what was their average total? You group the rows by customer, then count or average inside each group.
There is a trap. When you work out 'orders so far' for a row, count only the orders that happened before that row. The code below sorts the orders by time and keeps a running history for each customer, so each row sees only its own past. The days are whole days, so 6 days 22 hours counts as 6.
seen = {}
for o in sorted(orders, key=lambda o: o["when"]):
before = seen.get(o["who"], [])
o["orders_before"] = len(before)
o["days_since_last"] = (o["when"] - before[-1]["when"]).days if before else None
o["avg_before"] = round(sum(b["total"] for b in before) / len(before), 2) if before else None
seen.setdefault(o["who"], []).append(o)
for o in orders:
print(o["id"], o["who"], o["orders_before"], o["days_since_last"], o["avg_before"])Output
1 asha 0 None None 2 ben 0 None None 3 asha 1 12 40.0 4 chloe 0 None None 5 asha 2 6 32.75 6 ben 1 15 120.0
Notice the blanks: None for a first-time customer. A first order has no 'last order'. Do not invent a number such as 0, because 0 days would claim 'ordered again immediately'. You decide how to handle blanks, as in the cleaning lesson, but you decide on purpose.
Binning: turning numbers into groups
Google's Machine Learning Crash Course defines it this way: 'Binning (also called bucketing) is a feature engineering technique that groups different numerical subranges into bins or buckets.' It gives an example where one feature is split into five bins, and notes: 'A model trained on these bins will react no differently to X values of 17 and 29 since both values are in Bin 1.'
Why throw detail away? The course says binning is a good alternative to scaling or clipping when the overall linear relationship between the feature and the label is weak or nonexistent, or when the feature values are clustered. In its words, when a feature 'appears more clumpy than linear, binning is a much better way to represent the data.' Think of age groups on a form: 'under 18', '18 to 64', '65 and over'. A shop might care more about 'small, medium or large order' than about the exact cents.
The same page also describes quantile bucketing, which 'creates bucketing boundaries such that the number of examples in each bucket is exactly or nearly equal.' That keeps every bucket filled with enough examples. Its summary also advises against creating an excessive number of bins, since that can leave too few training examples per bin.
Encoding categories, and rare ones
A model works with numbers, so a column like pay (card, upi, voucher) must become numbers. The crash course warns against the lazy way. If you number the categories and feed the numbers in, the model treats them as amounts: 'The model would then consider "purple" six times more likely than "orange."'
The fix is one-hot encoding. The course says that in a one-hot vector, 'Exactly one of the elements in a one-hot vector has the value 1.0; all the remaining elements have the value 0.0.' One slot per category, one switch turned on.
Rare categories are a problem. A payment type seen once teaches the model very little, and it still adds a column. scikit-learn says its OneHotEncoder and OrdinalEncoder 'support aggregating infrequent categories into a single output for each feature', controlled by settings called min_frequency and max_categories. We do the same idea by hand: any category that appears fewer than twice goes into a group called 'other'. The code also adds a size bucket for the total.
from collections import Counter
def size_bucket(total):
if total < 30:
return "small"
if total < 80:
return "medium"
return "large"
pay_counts = Counter(o["pay"] for o in orders)
print(pay_counts)
for o in orders:
o["size"] = size_bucket(o["total"])
o["pay_group"] = o["pay"] if pay_counts[o["pay"]] >= 2 else "other"
categories = ["card", "upi", "other"]
for o in orders:
print(o["id"], o["size"], [int(o["pay_group"] == c) for c in categories])Output
Counter({'card': 3, 'upi': 2, 'voucher': 1})
1 medium [1, 0, 0]
2 large [1, 0, 0]
3 small [0, 1, 0]
4 large [0, 0, 1]
5 medium [1, 0, 0]
6 medium [0, 1, 0]Voucher appeared only once, so order 4 was folded into 'other'. The three numbers after each size are the one-hot vector for [card, upi, other].
Text basics: counts and bags of words
Free text, such as a note a customer typed, is not a number either. The easiest features are counts: how many words, how many characters, does it contain a certain word. scikit-learn's guide describes a fuller approach called the bag of words. Its steps are 'tokenizing' strings (splitting them into pieces, called tokens), 'counting' the occurrences of tokens, and 'normalizing' and weighting, which gives less importance to tokens that occur in most documents.
Picture tipping every word of a note into a bag. You lose the order but you keep how often each word appears. The guide admits the cost: a collection of single words 'cannot capture phrases and multi-word expressions, effectively disregarding any word order dependence'. Below we build two simple features and a mini bag of words with Counter from the standard library.
for o in orders:
words = o["note"].split()
o["note_words"] = len(words)
o["has_gift"] = "gift" in words
bag = Counter(w for o in orders for w in o["note"].split())
print(bag.most_common(4))
print(orders[3]["note_words"], orders[3]["has_gift"])Output
[('gift', 3), ('late', 3), ('night', 3), ('for', 2)]
5 TrueCounter is a dictionary that counts. Here 'gift', 'late' and 'night' each appear three times across all notes. Order 4's note has five words and does contain 'gift'.
Feature crosses and domain knowledge
Sometimes one feature alone says little, but two together say a lot. An order of 120 is one thing in the afternoon and another at midnight. A feature cross joins them. Google's course says feature crosses 'are created by crossing (taking the Cartesian product of) two or more categorical or bucketed features of the dataset', and that like polynomial transforms they 'allow linear models to handle nonlinearities' and 'encode interactions between features'.
A Cartesian product just means every pairing. Its leaf example crosses edges (smooth, toothed, lobed) with arrangement (opposite, alternate) and gets six pairs, such as Lobed_Alternate. For us, size crossed with time of day gives names like large_night. Because the cross works on categories or buckets, we cross our size bucket, not the raw total.
for o in orders:
o["size_x_night"] = o["size"] + "_" + ("night" if o["hour"] >= 21 else "day")
print(Counter(o["size_x_night"] for o in orders))Output
Counter({'medium_day': 2, 'large_night': 2, 'small_day': 1, 'medium_night': 1})Which crosses are worth making? The course says 'Domain knowledge can suggest a useful combination of features to cross.' It also warns: crossing two sparse features gives an even sparser one. Its example: a 100-element feature crossed with a 200-element feature yields a 20,000-element feature. More crosses means more columns and fewer examples in each.
Tip
Wikipedia names the danger of overdoing it: 'Feature explosion occurs when the number of identified features is too large for effective model estimation or optimization.' Add features for a reason, not because you can.
Leakage: features must exist at prediction time
Here is the rule that matters most. scikit-learn defines it this way: 'Data leakage occurs when information that would not be available at prediction time is used when building the model.' The result is 'overly optimistic performance estimates', so the model looks brilliant in testing and then does poorly on new data.
A weather forecast that uses tomorrow's newspaper is a perfect forecast and no use at all. Our shop has the same risk. A column such as 'refund_issued' is only known after the return, so it would reveal our label. Less obvious is the group summary from earlier. The databases lesson showed a cutoff date; here is the same idea at feature level. 'Orders by this customer' must mean orders before this order.
totals = Counter(o["who"] for o in orders)
for o in orders:
print(o["id"], o["who"], "all-time:", totals[o["who"]], "before this order:", o["orders_before"])Output
1 asha all-time: 3 before this order: 0 2 ben all-time: 2 before this order: 0 3 asha all-time: 3 before this order: 1 4 chloe all-time: 1 before this order: 0 5 asha all-time: 3 before this order: 2 6 ben all-time: 2 before this order: 1
Asha's first order shows 'all-time: 3' in the leaky version. At the moment she placed it, she had made none. A model fed the all-time count learns from orders that had not happened yet. The scikit-learn guide adds a related rule for preprocessing: 'Always split the data into train and test subsets first, particularly before any preprocessing steps.' Learn any averages or category lists from the training rows only.
Quick check
You want to predict, at checkout, whether an order will be returned. Which column is safe to use as a feature?
Checking that a feature helps
A new column is only an idea until you test it. Start with something you can read by eye: split the rows by the feature and compare the label in each group. If every group looks the same, the feature probably says nothing. Our six returned/not-returned labels are made up, so treat this as a drill, not a finding.
returned = {1: 0, 2: 1, 3: 0, 4: 1, 5: 0, 6: 1}
for key in ["has_gift", "size"]:
groups = {}
for o in orders:
groups.setdefault(o[key], []).append(returned[o["id"]])
print(key, {k: f"{sum(v)}/{len(v)}" for k, v in groups.items()})Output
has_gift {True: '1/3', False: '2/3'}
size {'medium': '1/3', 'large': '2/2', 'small': '0/1'}Both results read as 'returned / total'. Gift notes had 1 of 3 orders returned; non-gift had 2 of 3. Large orders were 2 of 2. With six rows, none of this proves anything, and real data would need many more rows and a held-out test set. The habit is the point: build a feature, then show it beats the version without it.
A workflow you can reuse
- 1
1. Name the moment
Write down exactly when the prediction is made. Every feature must be known by then.
- 2
2. Look at one row
Read a single row as a person. What would you want to know? Ratios, dates and history often come from that.
- 3
3. Build a few features
Add one feature at a time, from the list above, with a reason for each.
- 4
4. Handle blanks and rare values
Decide what a missing value means and group rare categories on purpose.
- 5
5. Check for leaks
For each feature, ask: could I compute this at prediction time, using only the past?
- 6
6. Test against a baseline
Compare the model with and without the feature, on data it has not seen.
Here are three things to keep in mind from this lesson:
Make signals visible
Ratios, differences, date parts, history and counts hand the model facts a person would notice.
Fit the model's needs
Bin numbers when ranges matter more than exact values, one-hot encode categories, group rare ones, and turn text into counts.
Guard the timeline
Use only what was known at prediction time, and check every feature before you trust it.
Quick check
A feature cross of a 100-value category and a 200-value category has how many possible values?
Next: feature scaling and normalization.
Sources
- Wikipedia: Feature engineering
- Google Machine Learning Crash Course: Numerical data: Binning
- Google Machine Learning Crash Course: Categorical data: Vocabulary and one-hot encoding
- Google Machine Learning Crash Course: Categorical data: Feature crosses
- scikit-learn: Feature extraction
- scikit-learn: Preprocessing data
- scikit-learn: Common pitfalls and recommended practices
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.