Machine Learning Foundations / Core / 02.19
Classification: sorting into groups
Spam or not spam? Cat, dog or bird? Classification means picking a group for each new thing. Build a tiny sorter yourself.
14 min readFree
Picking a group
Think of a post office worker with a pile of letters and a wall of pigeonholes. Each letter goes into one hole: local, national, abroad. The worker does not invent a new hole for each letter. The holes are fixed, and the job is to choose the right one.
That is classification. The last lesson ended on it: sorting things into groups. A model looks at something new and picks one group from a short, fixed list. Is this email spam or not? Is this photo a cat, a dog or a bird? Will this customer stay or leave?
It is a kind of supervised learning. If that phrase is new, the lesson on supervised learning explains it: the model learns from examples that already have the right answer attached. Here, the answer is a group.
Key words
- Class
- One of the groups a thing can be put in, such as "spam" or "not spam". Also called a category or a label.
- Classifier
- A model that picks a class. The scikit-learn glossary calls it a predictor with a finite set of discrete possible output values.
- Feature
- One fact about the thing you are sorting, such as the number of links in an email.
- Decision boundary
- The line (or surface) where the model switches from one class to another.
- Score
- A number the model gives for how strongly something fits a class, often a probability between 0 and 1.
- Threshold
- The cut-off on a score. Above it, the thing goes in the positive class. Below it, the negative class.
- Confusion matrix
- A small table that counts how often the model was right and wrong, split by kind of mistake.
Wikipedia notes that the word differs from field to field. In machine learning, the things sorted are often called instances, the facts about them features, and the possible groups classes. We use those words here.
Three shapes of the question
Not every sorting job is the same. The number of groups, and how many a thing may belong to, changes the setup. Scikit-learn names three shapes.
Binary: two classes
Yes or no. Spam or not spam. Fraud or fine. One class is called positive (the one you are hunting for) and the other negative.
Multiclass: many classes, one pick
Scikit-learn defines it as a task with more than two classes where each sample can only be labeled as one class. Its example: photos of fruit, each an orange, an apple or a pear.
Multilabel: many classes, any number of picks
Each sample gets m labels from the possible classes, where m can be anything from 0 up to all of them. Scikit-learn describes this as predicting properties that are not mutually exclusive.
A holiday photo shows the difference. As a multiclass question it gets one tag: beach, city or mountain. As a multilabel question it can get several at once: beach, sunset, people. A photo of an empty room may get none.
Tip
Ask two questions before you start. How many groups are there? Can one thing be in two groups at once? The answers tell you which shape you have.
Quick check
A model reads a news article and may tag it with any of: sport, money, health, travel. An article can have two tags, or none. Which shape is this?
A picture: the decision boundary
Here is a way to see what a classifier does. Pick two features and plot each example as a dot on a sheet of paper. For emails, one axis could be the number of links, the other the number of SHOUTING words. Spam emails are red dots. Normal emails are blue dots.
If the dots form two loose clouds, you can draw a line between them. Dots on one side get called spam. Dots on the other side get called not spam. That line is the decision boundary. A new email is just a new dot, and its side tells you its class.
With three classes you need more lines, cutting the paper into three zones. With more features the paper becomes a room, and the line becomes a flat wall or a curved sheet. You cannot draw that, but the idea is the same: a border, and a side for each class.
Dots near the border are the risky ones. A tiny change can tip them across. Dots deep inside one cloud are easy. Learning to draw a good border from examples is what a classifier does when it trains.
Different models draw different borders. Some draw straight ones. Some draw bendy ones. Wikipedia mentions that a family of them, called linear classifiers, works by combining the features with weights into a score. Others work by comparing the new thing to previous examples with a similarity or distance function. We try the second idea in the worked example.
Scores and labels
Many models do not jump straight to a label. They first give a score for each class, and a label comes from that score.
Google's Machine Learning Crash Course gives an example. A spam model outputs a value between 0 and 1, the probability that an email is spam. A 0.50 means a 50% likelihood, a 0.75 means a 75% likelihood. To file the email, you must turn the number into one of two groups. You choose a classification threshold. Emails scoring above it go to the positive class (spam). Emails below it go to the negative class (not spam).
Wikipedia points out the benefit of scores. Algorithms that only output a best class say nothing about how sure they are. Probabilistic ones output a probability for each possible class, and Wikipedia adds that such a classifier can abstain when its confidence is too low. In plain terms: it can say "I am not sure" and hand the case to a person.
- 1
1. The model gives a score
For one email: 0.99.
- 2
2. You pick a threshold
Say 0.5.
- 3
3. Compare
0.99 is above 0.5, so the label is "spam".
- 4
4. Change the threshold
Set it to 0.95. An email scoring 0.51 is now "not spam", while 0.99 stays "spam".
Google's own numbers: with scores of 0.99 and 0.51 and a threshold of 0.5, both emails are spam. With a threshold of 0.95, only the 0.99 one is. It also warns that 0.5 only looks like the natural choice. It is not a good idea when one kind of wrong answer costs more than the other, or when the classes are imbalanced.
Tip
A score is not the truth. Google says it plainly: the probability score is not reality, or ground truth. A 0.9 means the model leans strongly, not that the email is certainly spam.
Classification in daily life
You meet classifiers all day without noticing.
Spam filter (binary)
Each email lands in the inbox or the spam folder. This is the example Google's course uses for its classification lessons.
Fraud check (binary)
A bank looks at a card payment and decides: normal, or worth a second look. The positive class is the rare, worrying one.
Photo tags (multilabel)
A photo app suggests tags like "dog", "grass" and "sunny". One picture, several labels.
In all three, the model is not understanding the thing as you do. It looks at features, compares them with what it saw during training, and picks a side.
Right and wrong: the confusion matrix
How do you know if a classifier is any good? You count its mistakes. But there are two different kinds of mistakes, and they matter differently.
Google's course lays this out for a spam filter. There are four possible outcomes for each email.
- 1
True positive (TP)
A spam email correctly classified as spam. It goes to the spam folder, as it should.
- 2
False positive (FP)
A not-spam email misclassified as spam. A legitimate email ends up in the spam folder.
- 3
False negative (FN)
A spam email misclassified as not-spam. It slips into the inbox.
- 4
True negative (TN)
A not-spam email correctly classified as not-spam. It goes straight to the inbox.
Put the four counts in a two-by-two table and you have a confusion matrix. The rows are what the model predicted. The columns are what was really true. The name fits: it shows where the model gets confused.
A trick to remember the names: the second word is what the model said (positive or negative). The first word says whether it was right (true or false). So a false positive is a wrong "yes".
The two mistakes have different prices. A false negative lets spam through, which is annoying. A false positive hides a real email, such as a job offer, which can be much worse. A table of counts keeps these apart. One number like "90% right" does not.
Tip
Turning a count into a score, such as accuracy, precision and recall, comes in the Model Evaluation lessons. For now, just learn to read the four boxes.
When one class is rare
Fraud is rare. Most payments are fine. When one class is much bigger than the other, the data is called imbalanced. The lesson on data preparation covers how to spot it and how to handle it, so we will not repeat it here.
What matters for classification is a trap. Google gives it as a warning: if a class appears only 1% of the time, a model that predicts negative 100% of the time would score 99% on accuracy, despite being useless. It never finds a single fraud. This is why the confusion matrix matters: it would show zero true positives.
Quick check
Only 1 payment in 100 is fraud. A model says "not fraud" for every payment. What does its confusion matrix show?
Worked example: a nearest-centroid sorter
Time to build one. We choose the simplest idea we can: compare a new thing to the typical member of each group, and pick the group it is closest to.
A centroid is the average position of a group of dots. If the spam emails cluster around 4 links and 4 shouting words, that point is the spam centroid. A new email is sorted by asking which centroid is nearer. That is a nearest-centroid classifier. The decision boundary is a straight line exactly halfway between the two centroids.
The data is made up: 60 normal emails and 40 spam emails, each with two features (links and shouting words). A fixed seed makes the random numbers repeat. We train on 70 emails and keep 30 hidden for a test. If the idea of a test set is hazy, the lesson on train and test data explains it.
import math
import random
random.seed(3)
# Made-up emails: (links, shouty words). 0 = not spam, 1 = spam.
def make(n, label, centre):
return [((random.gauss(centre[0], 1.3), random.gauss(centre[1], 1.3)), label)
for _ in range(n)]
data = make(60, 0, (2, 2)) + make(40, 1, (4.5, 4.5))
random.shuffle(data)
train, test = data[:70], data[70:]
def centroid(rows):
n = len(rows)
return (sum(x for (x, _), _ in rows) / n, sum(y for (_, y), _ in rows) / n)
centres = {c: centroid([r for r in train if r[1] == c]) for c in (0, 1)}
for c, (x, y) in centres.items():
print("class", c, "centre", round(x, 2), round(y, 2))
def score(p):
# Bigger means "more like spam": how much closer to the spam centre.
return math.dist(p, centres[0]) - math.dist(p, centres[1])
def matrix(cut):
tp = fp = fn = tn = 0
for p, truth in test:
guess = 1 if score(p) > cut else 0
if guess == 1 and truth == 1: tp += 1
elif guess == 1 and truth == 0: fp += 1
elif guess == 0 and truth == 1: fn += 1
else: tn += 1
return tp, fp, fn, tn
tp, fp, fn, tn = matrix(0)
print("test emails:", len(test))
print(" actual spam actual not spam")
print("predicted spam ", tp, " ", fp)
print("predicted not ", fn, " ", tn)
print("accuracy:", round((tp + tn) / len(test), 3))
for cut in (-2, 0, 2):
tp, fp, fn, tn = matrix(cut)
print("cut-off", cut, "-> spam caught", tp, "missed", fn, "good mail lost", fp)Output
class 0 centre 1.93 1.74
class 1 centre 4.44 4.39
test emails: 30
actual spam actual not spam
predicted spam 11 0
predicted not 1 18
accuracy: 0.967
cut-off -2 -> spam caught 11 missed 1 good mail lost 7
cut-off 0 -> spam caught 11 missed 1 good mail lost 0
cut-off 2 -> spam caught 8 missed 4 good mail lost 0The centres
The normal centre landed near (1.93, 1.74) and the spam centre near (4.44, 4.39). Those are the two typical dots.
The matrix
On the 30 hidden emails, 11 spam were caught (true positives) and 18 normal emails went through (true negatives). One spam email slipped past (a false negative). No normal email was hidden (zero false positives). Accuracy was 29 right out of 30, or 0.967.
The cut-off
The score is how much closer an email is to the spam centre. We tried three cut-offs. Moving it to -2 catches the same 11 spam but loses 7 good emails. Moving it to 2 loses no good emails but lets 4 spam through. Same model, different trade.
That last row is the lesson in a nutshell. The model did not change. Only the threshold did, and the mistakes moved from one box to another. Which cut-off is best depends on which mistake you can better afford.
Tip
Try it yourself. Change the seed, or move the second centre closer to (2, 2), and watch the matrix. The closer the two clouds, the more mistakes any classifier makes, because no border can split them cleanly.
The model families coming next
Nearest-centroid is a teaching toy. Real classifiers draw smarter borders. The next lessons cover four families, each named here once so you know the map.
k-nearest neighbours (k-NN)
Look at the k closest known examples and let them vote.
Logistic regression
Combine the features into a score, then turn it into a probability. Wikipedia says that in statistics, classification is often done with logistic regression.
Decision trees and support vector machines (SVM)
A tree asks a chain of yes-or-no questions. An SVM is another way to draw a border. Wikipedia lists it among linear classifiers.
Scikit-learn's glossary notes that all its classifiers support multiclass classification, defaulting to a one-vs-rest strategy over the binary problem. In plain words: to sort into five groups, it can train five yes-or-no models, one per group, and pick the group that says yes most strongly.
Quick check
Which statement about a classifier's score and its label is correct?
What to remember
The job
Classification picks one group, or a few, from a fixed list. Two classes is binary, many with one pick is multiclass, and many with any number of picks is multilabel.
The border
A classifier learns a decision boundary. Models often give a score first, and a threshold turns it into a label.
The count
Read the confusion matrix: true and false positives and negatives. When a class is rare, a high accuracy can hide a useless model.
Next: regression, predicting a number.
Sources
- scikit-learn user guide: Multiclass and multioutput algorithms
- scikit-learn glossary
- Google Machine Learning Crash Course: Classification, Thresholds and the confusion matrix
- Google Machine Learning Crash Course: Classification, Accuracy, recall, precision, and related metrics
- Wikipedia: Statistical classification
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.