esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.03

Confusion matrix: the table behind scores

Every accuracy, precision and recall number comes from one small table of counts. Learn to read it, for two classes and for many.

13 min readFree

One table behind every score

The last lesson ended on a promise: the confusion matrix, the table behind every score. Here it is, in full.

Think of a school exam marked by two teachers. A single number, "85 out of 100", says how well a student did. It does not say which questions went wrong. The marked paper does. A confusion matrix is the marked paper for a classifier. It shows what the model got right and, just as useful, how it got things wrong.

Key words

Ground truth
The real answer, as known by a person or a trusted record. Google's course says the model's probability score is not reality, or ground truth.
Positive class
The group you are searching for, such as "ill" or "spam". The other group is the negative class.
Prediction
The group the model picked for one example.
Cell
One box in the table. It holds a count of examples.
Type I and type II error
Two names for the two kinds of mistake. Explained below.
Normalise
Turn counts into shares of a whole, such as percentages, so groups of different sizes can be compared.

Rows, columns and the four cells

A confusion matrix is a grid. One side lists the real answers. The other side lists what the model said. Each box counts how many examples fell in that pair.

Here is the catch. Not everyone puts the real answers on the same side. Google's course lays out the ground truth as columns and the model's prediction as rows. The scikit-learn user guide does the opposite: each row is the true class, and it adds that Wikipedia and other references may use a different convention for axes. This lesson puts truth in the rows and prediction in the columns, like scikit-learn.

Tip

Always check the labels on the axes before reading any confusion matrix. If you read a flipped table as if it were normal, you will swap the false positives and the false negatives.

With two classes there are four cells. Take a medical test for an illness. The positive class is "ill".

  1. 1

    True positive (TP)

    The person is ill and the test says ill. The test caught it.

  2. 2

    False positive (FP)

    The person is healthy but the test says ill. A false alarm.

  3. 3

    False negative (FN)

    The person is ill but the test says healthy. A miss.

  4. 4

    True negative (TN)

    The person is healthy and the test says healthy. A correct all-clear.

The name has two parts. The scikit-learn guide explains them: "positive" and "negative" refer to the classifier's prediction, and "true" and "false" refer to whether that prediction matches the real answer. So a false positive is a wrong "yes", and a false negative is a wrong "no".

A smoke alarm gives the same four cells. The positive event is "there is a fire".

  • True positive: fire, alarm rings

    The alarm did its job.

  • False positive: burnt toast, alarm rings

    No fire, but it shouted anyway. Annoying, and harmless.

  • False negative: fire, alarm silent

    The dangerous one. Nobody is warned.

The fourth cell, a true negative, is a quiet kitchen with a quiet alarm. Nothing happens, and that is the point. Notice that the two mistakes are not equal. Most people would rather have a few false alarms than one silent fire. For a spam filter, the worry can run the other way: hiding a real email can cost more than letting one junk email through.

Type I and type II errors

Statisticians have older names for the two mistakes. Wikipedia explains that in terms of false positives and false negatives, a type I error is equivalent to a false positive, and a type II error is equivalent to a false negative.

  • Type I = false positive

    A false alarm. The medical test says ill, but the person is healthy.

  • Type II = false negative

    A miss. The person is ill, but the test gives a falsely reassuring message that the disease is absent. That is how Wikipedia words its medical example.

  • A memory trick

    Type I is the first kind: the "yes" that should have been "no". Type II is the second kind: the "no" that should have been "yes".

Quick check

A smoke alarm stays silent during a real fire. What is this?

Build a 2x2 matrix

Let us build one. The data is made up: 60 people, of whom 12 really have an illness. Our pretend test gives each person a score from 0 to 1. Ill people tend to score higher, but not always. A fixed seed makes the random numbers repeat, so you get the same table.

To turn a score into a yes or a no, we need a cut-off, also called a threshold. Scores at or above it say "ill". Here the cut-off is 0.5.

Python
import random

random.seed(7)

# Made-up health check: 1 = has the illness, 0 = healthy.
truth = [1] * 12 + [0] * 48
# The test gives each person a score from 0 to 1; ill people score higher.
score = [min(1, max(0, random.gauss(0.65, 0.2))) for _ in range(12)] + \
        [min(1, max(0, random.gauss(0.35, 0.2))) for _ in range(48)]

def count(cut):
    tp = fp = fn = tn = 0
    for t, s in zip(truth, score):
        guess = 1 if s >= cut else 0
        if guess and t: tp += 1
        elif guess and not t: fp += 1
        elif t: fn += 1
        else: tn += 1
    return tp, fp, fn, tn

tp, fp, fn, tn = count(0.5)
print("rows = truth, columns = prediction")
print("               said ill   said healthy")
print("really ill     ", f"{tp:>5}", f"{fn:>10}")
print("really healthy ", f"{fp:>5}", f"{tn:>10}")

Output

rows = truth, columns = prediction
               said ill   said healthy
really ill         11          1
really healthy      8         40

Read it row by row. Of the 12 people who were really ill, 11 were caught and 1 was missed. Of the 48 healthy people, 40 got the all-clear and 8 were told they were ill by mistake. The four counts add up to 60, one per person.

All the scores come from the same four cells

Here is the big idea. Accuracy, precision, recall and specificity are not four separate things to measure. Each is a different way to divide the same four counts. Google's course says true and false positives and negatives are used to calculate several useful metrics for evaluating models.

  • Accuracy

    Of all examples, what share did the model get right? Google defines it as the proportion of all classifications that were correct, whether positive or negative. That is both correct cells, over everything.

  • Recall (also called true positive rate)

    Of the examples that really are positive, what share did it catch? Google calls this the proportion of all actual positives that were classified correctly as positives. It looks at the top row only. Scikit-learn adds that recall is sometimes called sensitivity.

  • Precision

    Of the examples the model called positive, what share really are? It looks at the "said ill" column only. Scikit-learn gives it as tp over tp plus fp.

A fourth one, specificity, is the mirror of recall for the negative group: of the really healthy, how many got the all-clear? Scikit-learn's guide quotes Wikipedia that the false positive rate is one minus the specificity, or true negative rate. Google defines the false positive rate as the share of actual negatives wrongly classified as positives, FP over FP plus TN.

accuracy=TP+TNTP+TN+FP+FNrecall=TPTP+FNprecision=TPTP+FPspecificity=TNTN+FP\begin{aligned} \text{accuracy} &= \frac{TP+TN}{TP+TN+FP+FN} \\ \text{recall} &= \frac{TP}{TP+FN} \\ \text{precision} &= \frac{TP}{TP+FP} \\ \text{specificity} &= \frac{TN}{TN+FP} \end{aligned}

In words: accuracy is the two correct boxes over all four. Recall is true positives over the whole ill row. Precision is true positives over the whole "said ill" column. Specificity is true negatives over the whole healthy row.

Try it on our table. Accuracy is (11 + 40) out of 60, which is 0.85. Recall is 11 out of 12, about 0.92. Precision is 11 out of 19, about 0.58. Specificity is 40 out of 48, about 0.83. The test catches nearly every ill person, but when it says "ill", it is right only about 58% of the time. One accuracy number of 0.85 hid that.

Tip

Each score gets its own lesson. For now, only remember where each one lives in the table.

Quick check

Which cells does recall use?

Moving the threshold moves the counts

The model did not change in the last section. Only a cut-off decided who got a "yes". Move the cut-off, and people move from one cell to another.

Google's course puts it this way: different thresholds usually result in different numbers of true and false positives and true and false negatives. Raise the threshold and the model predicts fewer positives overall, both true and false. It is then less likely to mislabel a healthy person, but more likely to miss a real case.

Python
import random

random.seed(7)

truth = [1] * 12 + [0] * 48
score = [min(1, max(0, random.gauss(0.65, 0.2))) for _ in range(12)] + \
        [min(1, max(0, random.gauss(0.35, 0.2))) for _ in range(48)]

def count(cut):
    tp = fp = fn = tn = 0
    for t, s in zip(truth, score):
        guess = 1 if s >= cut else 0
        if guess and t: tp += 1
        elif guess and not t: fp += 1
        elif t: fn += 1
        else: tn += 1
    return tp, fp, fn, tn

print("cut   TP  FP  FN  TN   accuracy  precision  recall  specificity")
for cut in (0.3, 0.5, 0.7):
    tp, fp, fn, tn = count(cut)
    print(f"{cut}  {tp:>3} {fp:>3} {fn:>3} {tn:>3}   "
          f"{(tp + tn) / 60:>7.2f}  {tp / (tp + fp):>9.2f}  "
          f"{tp / (tp + fn):>6.2f}  {tn / (tn + fp):>10.2f}")

Output

cut   TP  FP  FN  TN   accuracy  precision  recall  specificity
0.3   12  28   0  20      0.53       0.30    1.00        0.42
0.5   11   8   1  40      0.85       0.58    0.92        0.83
0.7    5   0   7  48      0.88       1.00    0.42        1.00

Read down the rows. At 0.3, the test says "ill" for 40 of the 60 people. It catches all 12 ill people (recall 1.00) but sends 28 healthy people for needless worry, so precision drops to 0.30. At 0.7 the opposite happens. It never raises a false alarm, but it misses 7 of the 12 ill people.

Notice accuracy at 0.7 is 0.88, the best of the three, even though the test misses more than half of the ill people. Accuracy looks good because the healthy group is so much bigger. This is the same trap as the last lesson.

  1. 1

    Lower the cut-off

    More "yes" answers. Misses (FN) go down, false alarms (FP) go up.

  2. 2

    Raise the cut-off

    Fewer "yes" answers. False alarms (FP) go down, misses (FN) go up.

  3. 3

    Pick for the cost

    Ask which mistake is worse, and set the cut-off to match. The right answer depends on the job.

More than two classes

The same table works for any number of classes. Three classes make a 3x3 grid, five make a 5x5. Rows are still the truth, columns the prediction.

Now there is no single "positive". Instead, look at the diagonal, the boxes running from top-left to bottom-right. Those are the correct answers, where the truth and the prediction match. Everything off the diagonal is a mistake, and its position says which mix-up it was. A count in the "dog" row and the "cat" column means: a real dog, called a cat.

The scikit-learn guide shows the idea with a tiny example of three classes. By definition, entry i, j is the number of observations actually in group i, but predicted to be in group j.

Let us build one. The labels are made up: 20 photos each of cats, dogs and rabbits. We give the pretend model a different chance of being right for each animal, and make it call dogs "cat" more often than anything else. Then we print the counts, and the same table as percentages of each row.

Python
import random

random.seed(11)

names = ["cat", "dog", "rabbit"]
right = {"cat": 0.85, "dog": 0.55, "rabbit": 0.80}  # chance of a correct guess
truth, guess = [], []
for n in names:
    for _ in range(20):
        truth.append(n)
        if random.random() < right[n]:
            guess.append(n)
        elif n == "dog" and random.random() < 0.85:
            guess.append("cat")  # dogs are often called cats
        else:
            guess.append(random.choice([m for m in names if m != n]))

grid = [[0] * 3 for _ in names]
for t, g in zip(truth, guess):
    grid[names.index(t)][names.index(g)] += 1

print("counts (rows = truth, columns = prediction)")
print(" " * 8 + "".join(f"{n:>8}" for n in names))
for n, row in zip(names, grid):
    print(f"{n:<8}" + "".join(f"{v:>8}" for v in row))

print()
print("row-normalised (each row adds to 100%)")
print(" " * 8 + "".join(f"{n:>8}" for n in names))
for n, row in zip(names, grid):
    total = sum(row)
    print(f"{n:<8}" + "".join(f"{100 * v / total:>7.0f}%" for v in row))

Output

counts (rows = truth, columns = prediction)
             cat     dog  rabbit
cat           17       1       2
dog            8      12       0
rabbit         1       1      18

row-normalised (each row adds to 100%)
             cat     dog  rabbit
cat          85%      5%     10%
dog          40%     60%      0%
rabbit        5%      5%     90%

Read the counts first. Each row adds up to 20. The diagonal has 17, 12 and 18 correct answers. The biggest mistake sits in the dog row, cat column: 8 real dogs were called cats.

Now the percentages. Normalising by row divides each count by its row total, so each row adds to 100%. Now each row answers one question: of the real dogs, what share went where? The answer is 60% dogs, 40% cats and 0% rabbits. Groups of different sizes can now be compared fairly, which helps most when classes are unequal.

  • Diagonal

    The share of each class found correctly. For a row-normalised table, this is each class's recall: 85% for cats, 60% for dogs, 90% for rabbits.

  • Big off-diagonal box

    A pair of classes the model mixes up. Dog and cat here. Look at those photos, or give the model more dog examples.

  • Empty box

    Two classes it never mixes up. No dogs became rabbits.

To use two-class scores on a many-class table, treat one class as positive and all others as negative. The scikit-learn guide says its multilabel confusion matrix function does this: it treats multiclass data as if it were multilabel, a transformation commonly applied to evaluate multiclass problems with binary classification metrics such as precision and recall.

Quick check

In a 3x3 matrix with truth in rows, the box in row "dog" and column "cat" holds 8. What does it mean?

What to remember

  • The layout

    Rows are the truth and columns the prediction here, but libraries differ, so check the labels. Four cells for two classes: TP, FP, FN, TN.

  • The mistakes

    A false positive (type I) is a false alarm. A false negative (type II) is a miss. They rarely cost the same.

  • The scores

    Accuracy, precision, recall and specificity are four ways to divide the same cells. Moving the threshold moves counts between cells. For many classes, read the diagonal and row percentages.

Next: precision, how often a yes is really a yes.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.