esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.02

Accuracy: the simplest score and its traps

Accuracy is right answers divided by all answers. Learn when it works, when a high number hides a useless model, and what to report beside it.

14 min readFree

The simplest score

The last lesson ended with a promise: accuracy, the simplest score and its traps. Here it is. Accuracy is the first number almost everyone reaches for, because you already know how to work it out. It is the same sum a teacher does on a test paper: how many answers were right, out of how many questions.

That simplicity is why it is loved, and also why it is dangerous. A number that is easy to read is easy to trust too much. The model evaluation lesson showed a lazy guess beating a real model. Here we go deeper: what accuracy measures, when it is fair, why it can look great on a useless model, and what to put beside it.

Think of a smoke alarm that never goes off. For 364 days of the year there is no fire, so it is "right" on 364 days out of 365. That is 99.7% accurate, and it is worth nothing on the one day it matters.

Key words

Accuracy
The share of all answers a model got right, as a number from 0 to 1 or a percentage.
Class
One of the groups an answer can fall into, such as "fine", "scratch" or "crack".
Imbalanced data
Data where one class is far more common than the others.
Baseline
A very simple guess, such as always naming the most common class, that a real model must beat.
Hit rate
For one class: out of the things that truly belong to it, the share the model found. The proper name is recall.
Balanced accuracy
The average of the hit rates of every class, so each class counts the same, however rare.
Top-k accuracy
Accuracy where the model gets k guesses, and any of them being right counts.

The formula

Google's Machine Learning Crash Course defines it in one line: accuracy is the proportion of all classifications that were correct, whether positive or negative. In plain words, you count the right answers and divide by the number of answers.

accuracy=right answersall answers\text{accuracy} = \frac{\text{right answers}}{\text{all answers}}

If a model sorts 100 emails and gets 92 right, its accuracy is 92 divided by 100, which is 0.92, or 92%. A perfect model scores 1.0.

  1. 1

    Collect the true answers

    You need examples where the right answer is already known.

  2. 2

    Ask the model for its answers

    Give it each example, without the answer, and write down what it says.

  3. 3

    Compare one by one

    For each example, mark it right if the model's answer matches the true one, and wrong if not.

  4. 4

    Count and divide

    Add up the rights, then divide by the total number of examples. That is your accuracy.

Google's course also writes it with the four boxes of the confusion matrix, which the next lesson covers. The simple version is all you need for now.

Scikit-learn, the popular Python library, has a function for this called accuracy_score. Its guide says it returns either the fraction (the default) or the count of correct predictions.

Tip

Always keep the count beside the fraction. "90% accurate" on 10 examples and "90% accurate" on 10,000 examples are very different amounts of evidence.

When accuracy is a fair score

Accuracy is not bad. It is just narrow. It does a fair job in these cases.

  • The classes are balanced

    Roughly as many examples of each class. The course says that with a balanced dataset, with similar numbers of examples in both classes, accuracy can serve as a coarse-grained measure of model quality. Coarse-grained means rough: a first look.

  • The mistakes cost about the same

    Calling a cat a dog costs about as much as calling a dog a cat. When no mistake is worse than another, counting all mistakes equally makes sense.

  • You need a simple headline

    The course notes accuracy is often the default metric for generic or unspecified models doing generic tasks. If you only need a quick check that something works at all, it is a fine place to start.

A quiz app that tells apart apples, pears and oranges, with the same number of each, is a fair fit. No fruit is rare, and no mistake is serious.

The same course adds the warning that matters for the rest of this lesson. When the dataset is imbalanced, or where one kind of mistake is more costly than the other, which it says is the case in most real-world applications, it is better to optimise for one of the other metrics instead.

Quick check

Which job is the best fit for plain accuracy?

The accuracy paradox

Now the trap. Wikipedia describes the accuracy paradox as the finding that accuracy is not a good metric for predictive models when classifying: a simple model may have a high level of accuracy but too crude to be useful.

Its example is easy to hold on to. If one category is dominant, found in 99% of cases, then predicting that every case is that category will have an accuracy of 99%. Google's course gives the same case: with a rare class at about 1% of the data, a model that predicts negative 100% of the time would score 99% on accuracy, despite being useless.

The cause is class imbalance. The common class carries so much weight in the sum that the rare class hardly moves the number. The model can ignore the rare class completely and still look excellent.

A factory line makes 500 parts. Most are fine. Some have a scratch, which is cosmetic. A few have a crack, which is dangerous, so missing a crack is the expensive mistake. The data is made up, with a fixed random seed so you get the same numbers. First we build the parts and a baseline that never looks at them: it says "fine" every time.

Python
import random

random.seed(5)

# 500 made-up parts from a factory line: mostly fine, some scratched, a few cracked.
classes = ["fine", "scratch", "crack"]
truth = random.choices(classes, weights=[90, 7, 3], k=500)

# The baseline never looks: it always says "fine".
baseline = ["fine"] * len(truth)

def accuracy(truth, guess):
    right = sum(1 for t, g in zip(truth, guess) if t == g)
    return right / len(truth)

for c in classes:
    print(c, truth.count(c))
print("baseline accuracy:", accuracy(truth, baseline))

Output

fine 444
scratch 39
crack 17
baseline accuracy: 0.888

Of the 500 parts, 444 are fine, 39 are scratched and 17 are cracked. The lazy baseline says "fine" for all of them, so it is right 444 times: an accuracy of 0.888, or 88.8%. It has not found one single crack, and it still scores nearly 89%.

Tip

A big accuracy number is a reason to look closer, not a reason to celebrate. Ask first: what would a model that always guesses the common class score?

A real model, beside the baseline

Now we add a made-up model that does try. We give it a different hit chance for each class: it is right about 93% of the time on fine parts, 80% on scratched ones and only 35% on cracked ones. Cracks are rare, so it has had little chance to learn them. When it is wrong, it picks one of the other two classes.

Then we print three things for the baseline and the model: the accuracy, the hit rate for each class, and a new score called balanced accuracy, explained in the next section.

Python
import random

random.seed(5)
classes = ["fine", "scratch", "crack"]
truth = random.choices(classes, weights=[90, 7, 3], k=500)
baseline = ["fine"] * len(truth)

# A made-up model. Chance it gets each kind of part right:
# fine 93%, scratch 80%, crack 35%. When wrong, it picks another class.
hit_chance = {"fine": 0.93, "scratch": 0.80, "crack": 0.35}
random.seed(11)
model = []
for t in truth:
    if random.random() < hit_chance[t]:
        model.append(t)
    else:
        model.append(random.choice([c for c in classes if c != t]))

def accuracy(truth, guess):
    right = sum(1 for t, g in zip(truth, guess) if t == g)
    return right / len(truth)

def hit_rates(truth, guess):
    rates = {}
    for c in classes:
        total = sum(1 for t in truth if t == c)
        hits = sum(1 for t, g in zip(truth, guess) if t == c and g == c)
        rates[c] = hits / total
    return rates

def balanced_accuracy(truth, guess):
    rates = hit_rates(truth, guess)
    return sum(rates.values()) / len(rates)

for name, guess in [("baseline", baseline), ("model", model)]:
    print(name)
    print("  accuracy:", round(accuracy(truth, guess), 3))
    for c, r in hit_rates(truth, guess).items():
        print("  hit rate for", c + ":", round(r, 2))
    print("  balanced accuracy:", round(balanced_accuracy(truth, guess), 3))

Output

baseline
  accuracy: 0.888
  hit rate for fine: 1.0
  hit rate for scratch: 0.0
  hit rate for crack: 0.0
  balanced accuracy: 0.333
model
  accuracy: 0.906
  hit rate for fine: 0.94
  hit rate for scratch: 0.77
  hit rate for crack: 0.35
  balanced accuracy: 0.687

Read the baseline first. Its accuracy is 0.888. Its hit rate is 1.0 for fine parts and 0.0 for the other two classes. The accuracy number hid that.

Now the model. Its accuracy is 0.906. That beats the baseline, but only by 1.8 points (0.906 minus 0.888). On accuracy alone it looks like a tiny win. The hit rates tell the real story: it finds 94% of fine parts, 77% of scratched ones and 35% of cracked ones. It does find cracks, which the baseline never does, though still fewer than half.

Same accuracy, different story. On imbalanced data, break accuracy down by class before you believe it.

Balanced accuracy: every class counts the same

Plain accuracy gives every example one vote. The fine parts have 444 votes and the cracked ones have 17, so the fine parts win every argument. Balanced accuracy changes the rule: every class gets one vote, however few examples it has.

Scikit-learn's guide says the balanced accuracy function avoids inflated performance estimates on imbalanced datasets. It defines the score as the macro-average of recall scores per class. A macro-average just means: work out the score for each class on its own, then take the plain average of those scores, with no extra weight for the big classes.

balanced accuracy=1C∑c=1Chit ratec\text{balanced accuracy} = \frac{1}{C}\sum_{c=1}^{C} \text{hit rate}_c

In words: add up the hit rate of every class, then divide by C, the number of classes. Our factory has three classes, so we add three hit rates and divide by 3. For the model that is (0.94 + 0.77 + 0.35) divided by 3, about 0.687, which matches what the code printed.

Look at what happened to the two competitors. The baseline fell from 0.888 to 0.333, which is one third. The model moved from 0.906 to 0.687. Now the gap between them is large, as it should be.

That 0.333 is no accident. Scikit-learn's guide says that if the classifier gets its conventional accuracy only by taking advantage of an imbalanced test set, then the balanced accuracy will drop to one over the number of classes. With three classes, one over three is 0.333. The guide adds that for balanced datasets, the score equals plain accuracy, so you lose nothing by using it.

  • Plain accuracy

    Every example counts once. The big class dominates. Easy to explain, easy to fool.

  • Balanced accuracy

    Every class counts once. A rare class matters as much as a common one. The floor for a model that learned nothing is one over the number of classes.

  • Per-class hit rates

    The full breakdown. Always worth printing, because the averages hide who is being missed.

Tip

Balanced accuracy still treats all classes as equally important. If missing a crack is far worse, look at the hit rate for cracks on its own.

Accuracy with many classes

Nothing about the idea changes when there are more than two classes. You still count the right answers and divide by all of them. Our factory model already did this with three classes.

A related word shows up in the scikit-learn guide, so it is worth knowing. In multilabel problems, where each example can have several labels at once, the accuracy function returns the subset accuracy: if the whole set of predicted labels strictly matches the true set, the score is 1.0, and otherwise 0.0. A photo tagged beach and sunset only counts as right if the model says exactly beach and sunset, not just beach.

With very many classes, such as a photo app that knows a thousand kinds of object, plain accuracy gets harsh. Two guesses can both be reasonable. That is where top-k accuracy helps.

Top-k accuracy: a few guesses allowed

In top-k accuracy the model ranks its guesses from best to worst, and it counts as right if the true answer is among its first k guesses. Scikit-learn's guide puts it like this: a prediction is considered correct as long as the true label is associated with one of the k highest predicted scores. Plain accuracy is the special case where k is 1.

Think of a search box that shows three suggestions. If the thing you wanted is one of the three, the search worked, even if it was not on top. Here are five photos and a model that ranks three guesses for each.

Python
# Five photos. For each, the model ranks its guesses, best first.
truth = ["cat", "dog", "fox", "owl", "hen"]
ranked_guesses = [
    ["cat", "fox", "dog"],   # right at the first guess
    ["fox", "dog", "cat"],   # right at the second guess
    ["dog", "cat", "fox"],   # right at the third guess
    ["hen", "dog", "cat"],   # the owl is not in the top three
    ["hen", "owl", "fox"],   # right at the first guess
]

def top_k_accuracy(truth, ranked_guesses, k):
    hits = sum(1 for t, g in zip(truth, ranked_guesses) if t in g[:k])
    return hits / len(truth)

for k in [1, 2, 3]:
    print("top-" + str(k), "accuracy:", top_k_accuracy(truth, ranked_guesses, k))

Output

top-1 accuracy: 0.4
top-2 accuracy: 0.6
top-3 accuracy: 0.8

With one guess the model gets 2 of 5 right (0.4). With two guesses it gets 3 of 5 (0.6), and with three guesses it gets 4 of 5 (0.8). The owl is never in the model's top three, so it stays wrong. Always say which k you used.

How to report accuracy honestly

Putting it all together, an honest report has more than one number.

  1. 1

    Count the classes

    Check how many examples each class has. If one class is much bigger than the others, plain accuracy is suspect from the start.

  2. 2

    Print the baseline

    Work out the accuracy of always guessing the most common class. Write it beside your model's number.

  3. 3

    Show the hit rate for each class

    A small table is enough. It shows who is being missed.

  4. 4

    Add balanced accuracy

    One number that gives every class an equal vote, with a floor of one over the number of classes.

  5. 5

    Say what a mistake costs

    If one error is worse, pick the score that tracks it. Later lessons cover precision, recall and F1 for that.

A good score line reads like this: "On 500 test parts, accuracy is 0.906 against a baseline of 0.888, and balanced accuracy is 0.687 against 0.333. Cracks are found 35% of the time." It is longer than "90.6% accurate", and it is the only version that tells a reader whether to trust the model.

Quick check

A model for 3 classes has an accuracy of 0.95, but its balanced accuracy is 0.40. What is the most likely reason?

Next: the confusion matrix, the table behind every score.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.