Machine Learning Foundations / Advanced / 03.04
Precision: is a yes really a yes?
When your model says yes, how often is it right? Learn precision, why false alarms cost money, and how the threshold changes it.
13 min readFree
When the model says yes, can you trust it?
The Confusion Matrix lesson ended with a promise: precision, how often a yes is really a yes. Here it is. Picture a smoke alarm that rings every time you make toast. At first you jump up. After the tenth false alarm you stop listening, and the day there is a real fire, you ignore it too. You cannot trust its yes.
Precision is the score for exactly that trust. It looks only at the times the model said yes, and asks what share of those yeses were correct. The times it said no do not count.
Key words
- Positive
- The answer you are hunting for, such as "this email is spam" or "this payment is fraud". A model says yes when it predicts the positive.
- True positive (TP)
- The model said yes, and it was right.
- False positive (FP)
- The model said yes, and it was wrong. A false alarm.
- Precision
- Of all the yeses the model gave, the share that were right. Wikipedia also calls it the positive predictive value.
- Threshold
- The cut-off on a score. A score at or above it becomes a yes. Below it becomes a no.
- Ranked list
- Results in order, best guess first, like a page of search results.
By the end you will work out precision by hand, know when it matters, see how the threshold moves it, and read precision at k and the averaged versions.
The formula, in words and in numbers
Count the yeses that were right (TP). Count the yeses that were wrong (FP). Add them up to get every yes the model gave. Then divide the right ones by that total.
In words: right yeses, divided by all yeses. Google's Machine Learning Crash Course defines it as the proportion of all the model's positive classifications that are actually positive. Scikit-learn's guide puts the same idea this way: intuitively, precision is the ability of the classifier not to label as positive a sample that is negative.
The answer sits between 0 and 1. Google's course notes that a hypothetical perfect model would have zero false positives, and so a precision of 1.0. A precision of 0.8 means that four out of five yeses were right.
- 1
Pick the yes
Decide which answer counts as the positive, for example "this email is junk".
- 2
Collect the yeses
Gather every case where the model said yes. Ignore all the nos.
- 3
Count the right ones
Check each yes against the truth. The correct ones are TP, the wrong ones are FP.
- 4
Divide
TP divided by (TP + FP). That is the precision.
Let us do it in code. A spam filter flagged 10 emails. Eight were junk and two were real mail someone wanted. The second half of the code handles a filter that flags nothing, which we come back to later.
def precision(tp, fp):
if tp + fp == 0:
return None # the model never said yes
return tp / (tp + fp)
# A spam filter flagged 10 emails. 8 were junk, 2 were real mail.
print("flagged 10, junk 8, real 2")
print("precision:", precision(8, 2))
# A filter that flags nothing has no yes answers to judge.
p = precision(0, 0)
if p is None:
print("never said yes, so precision can't be worked out")Output
flagged 10, junk 8, real 2 precision: 0.8 never said yes, so precision can't be worked out
Eight right yeses out of ten gives 0.8. The two real emails went to the spam folder, and someone may never read them. That is the cost precision measures.
When false alarms are costly
A false alarm always costs something. Whether the cost is small or large decides how much you care about precision. Google's course says to use precision when it is very important for positive predictions to be accurate.
A spam folder
A false positive sends real mail to the junk folder. Google's course lists "ensure that spam-labeled emails are in fact spam" as one of the reasons to put precision first. A lost job offer is worse than one junk email in the inbox.
A shop blocking shoppers
Say a model flags payments as fraud and the shop blocks them. Every false positive is an honest customer turned away at the till. They may never come back. High precision means the shop blocks mostly real thieves.
If a false alarm costs almost nothing, you can accept a lower precision and catch more cases. The cost of the mistake picks the score.
Tip
Before you trust a precision number, write down who pays for one false alarm. If the answer is "a real customer", precision deserves a place at the top of your report.
Quick check
A model flags 20 payments as fraud. 15 really are fraud and 5 are honest shoppers. What is the precision?
The threshold pulls precision up and down
Most models give a score first and a yes or no second. The Classification lesson showed that a threshold turns one into the other. Move the threshold and the yeses change, so precision changes too.
Google's course explains the direction: increasing the classification threshold tends to decrease the number of false positives and increase the number of false negatives, while decreasing the threshold has the opposite effects. Fewer false positives means a higher precision. But more false negatives means more real positives missed, which is what the next lesson, Recall, measures. Google's course sums it up: precision and recall often show an inverse relationship.
Think of a bouncer at a club door. A strict one lets in only people clearly on the list: almost everyone inside belongs (high precision), but some guests are turned away. A relaxed one lets in nearly everyone on the list, and a few gatecrashers too.
Let us watch it happen. We invent 60 emails with a fixed random seed. Junk (1) tends to get a higher score than real mail (0), but the scores overlap, as in real life. Then we try four thresholds.
import random
random.seed(11)
# 60 made-up emails. Junk (1) tends to get a higher score than real mail (0).
labels = [1 if random.random() < 0.4 else 0 for _ in range(60)]
scores = []
for y in labels:
base = 0.68 if y == 1 else 0.32
s = base + random.uniform(-0.3, 0.3)
scores.append(round(min(max(s, 0.0), 0.99), 2))
print("emails:", len(labels), " really junk:", sum(labels))
print("threshold TP FP precision")
for t in [0.3, 0.5, 0.7, 0.9]:
tp = sum(1 for y, s in zip(labels, scores) if s >= t and y == 1)
fp = sum(1 for y, s in zip(labels, scores) if s >= t and y == 0)
if tp + fp == 0:
shown = "can't be worked out"
else:
shown = f"{tp / (tp + fp):.2f}"
print(f"{t:<9} {tp:>2} {fp:>2} {shown}")Output
emails: 60 really junk: 22 threshold TP FP precision 0.3 22 16 0.58 0.5 16 7 0.70 0.7 9 0 1.00 0.9 5 0 1.00
At a threshold of 0.3 the filter says yes to 38 emails (22 right, 16 wrong), a precision of 0.58. At 0.5, 16 right and 7 wrong gives 0.70. At 0.7 the false alarms vanish: 9 right and none wrong, a precision of 1.00. At 0.9 it is still 1.00, but only 5 emails are flagged.
Look at the last two rows with care. A precision of 1.00 sounds perfect, yet the filter at 0.9 found only 5 of the 22 junk emails. Precision does not see the 17 it let through. Google's course also says precision is less meaningful when the number of actual positives is very, very low.
Tip
Never report precision alone. A model that says yes only once, and is right, scores a perfect 1.0. Always pair it with recall.
Quick check
You raise the threshold from 0.5 to 0.9. What usually happens to the number of false positives, and to precision?
When the model never says yes
There is one trap in the formula. Suppose a filter is so strict that it never flags anything. Then TP is 0 and FP is 0, and the bottom of the fraction is 0 + 0. You cannot divide by zero. The answer is not 0 and not 1. It simply can't be worked out, because there were no yeses to judge.
Google's course explains it with the word NaN, short for "not a number". It appears when dividing by 0, which can happen with any of these metrics. The course adds that a model that never predicts positive would have 0 TPs and 0 FPs, so calculating its precision would give NaN. It also warns that this can mean different things: in some cases it can indicate perfect performance, and it can also come from a model that is practically useless.
The first code block showed how to handle it: check for a zero total and return a clear signal instead of crashing. Scikit-learn's guide meets a similar gap with F1. By default it is calculated as 0 when there are no true positives, false positives or false negatives, and the zero_division setting can change that.
Tip
Before you divide, check that the model said yes at least once. If it did not, write "cannot be worked out" in your report. Do not quietly turn it into 0 or 1.
Precision at k: judging a ranked list
Search engines and shop recommendations do not give a yes or no. They give a list, best first, and people look at the top few. So we ask a smaller question: in the first k results, how many are right? That is precision at k, written P@k.
Wikipedia gives an example: P@10, or precision at 10, is the number of relevant results among the top 10 retrieved documents. It also lists two weak spots. P@k ignores the order of the results within the top k, and when a query has fewer relevant results than k, even a perfect system scores less than 1. It is easy to use, though, since you only check the top k results.
Here it is on our 60 emails. We rank them from the highest score to the lowest, and look only at the top.
import random
random.seed(11)
labels = [1 if random.random() < 0.4 else 0 for _ in range(60)]
scores = []
for y in labels:
base = 0.68 if y == 1 else 0.32
s = base + random.uniform(-0.3, 0.3)
scores.append(round(min(max(s, 0.0), 0.99), 2))
# Rank every email from highest score to lowest, then judge only the top k.
ranked = sorted(zip(scores, labels), reverse=True)
def precision_at_k(ranked, k):
return sum(y for _, y in ranked[:k]) / k
for k in [5, 10, 15, 20]:
print("precision@" + str(k) + ":", round(precision_at_k(ranked, k), 2))Output
precision@5: 1.0 precision@10: 1.0 precision@15: 0.93 precision@20: 0.75
The top 5 and the top 10 are all junk, so P@5 and P@10 are 1.0. By the top 15 one real email has slipped in, so P@15 is 0.93. By the top 20, five real emails have slipped in, and P@20 is 0.75.
Precision with more than two answers
What if the model sorts photos into cat, dog or bird? Precision still works. Pick one kind at a time and call it the positive. Scikit-learn's guide says that in a multiclass task precision can be applied to each label independently. For cats: of the photos the model called cat, how many were cats?
That gives one precision per class. To get one number for the whole model you must average them, and there are several ways. Scikit-learn names a few.
Macro average
Work out each class's precision, then take the plain average of them. Every class counts the same, big or small. A rare class matters as much as a common one.
Micro average
Pool all the yeses of all the classes together, then divide the right ones by all of them. Big classes count more. Scikit-learn's guide says that when all labels are included, micro-averaging in a multiclass setting gives precision, recall and F that are all identical to accuracy.
Weighted average
A plain average, except that each class counts in proportion to how many real examples it has. Scikit-learn also offers this.
Here are 10 made-up photos with the real kind and the model's guess.
# Three kinds of photo: cat, dog, bird. Truth and the model's guess for 10 photos.
truth = ["cat", "cat", "cat", "cat", "dog", "dog", "dog", "bird", "bird", "bird"]
guess = ["cat", "cat", "dog", "cat", "dog", "dog", "cat", "bird", "cat", "cat"]
kinds = ["cat", "dog", "bird"]
per_class = {}
total_tp = 0
total_said = 0
for k in kinds:
tp = sum(1 for t, g in zip(truth, guess) if g == k and t == k)
said = sum(1 for g in guess if g == k)
per_class[k] = tp / said
total_tp += tp
total_said += said
print(k, "said", said, "right", tp, "precision", round(tp / said, 2))
macro = sum(per_class.values()) / len(kinds)
micro = total_tp / total_said
print("macro:", round(macro, 2))
print("micro:", round(micro, 2))Output
cat said 6 right 3 precision 0.5 dog said 3 right 2 precision 0.67 bird said 1 right 1 precision 1.0 macro: 0.72 micro: 0.6
The model said cat 6 times and was right 3 times, so cat precision is 0.5. Dog is 2 out of 3, which is 0.67. Bird is 1 out of 1, a perfect 1.0, but based on a single yes. The macro average is the plain mean of those three, 0.72. The micro average pools them: 6 right yeses out of 10 yeses in all, which is 0.6, the same as the accuracy of 6 correct photos out of 10.
The two averages disagree because the model barely says "bird". Macro lets that one lucky bird guess lift the score. Micro mostly follows the busy class, cat. Which one to report depends on whether every class matters equally.
Tip
For a multiclass model, always ask which average a precision number is. The same model gave 0.72 or 0.6 here.
A short checklist
- 1
Count TP and FP
Use only the cases where the model said yes.
- 2
Divide
TP divided by (TP + FP), and check the bottom is not zero first.
- 3
Say what a false alarm costs
Decide if it is a small nuisance or a lost customer.
- 4
Note the threshold
A precision number means little without the cut-off that produced it.
- 5
Look beside it
Report recall, or at least the number of yeses, next to precision.
What to remember
The question
Precision answers one thing: when the model says yes, how often is it right? It is TP divided by (TP + FP).
The cost
Put precision first when a false alarm hurts: real mail in the spam folder, honest shoppers blocked.
The partner
A higher threshold usually raises precision and lowers recall. Never judge precision alone, and be careful when the model says yes only a few times.
Next: recall, how many of the real yeses you catch.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.