esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.07

ROC-AUC: judging every threshold

Slide the cut-off from strict to loose, draw what happens, and measure the area. One number that compares models without picking a threshold.

13 min readFree

One threshold is only a snapshot

The last lesson ended with a promise: judging a model across every threshold. Here it is.

Picture a spam filter. It gives every email a score between 0 and 1 for how spammy it looks. You pick a cut-off, called the threshold. Scores at or above it are marked spam. Scores below it are marked fine. Move the cut-off and the mistakes move with it.

Precision, recall and F1 are all worked out at one threshold. They are a photo, taken at one moment. If you pick a different threshold tomorrow, you get a different photo. Google's Machine Learning Crash Course says those metrics are all calculated at a single classification threshold value, and that evaluating a model across all possible thresholds takes different tools.

Key words

Threshold
The cut-off on a score. At or above it, the answer is yes. Below it, the answer is no.
True positive rate (TPR)
Of all the real yeses, the share the model caught. It is the same thing as recall.
False positive rate (FPR)
Of all the real nos, the share the model wrongly called yes. A false alarm rate.
ROC curve
A line drawn by plotting TPR against FPR at many thresholds. ROC stands for receiver operating characteristic.
AUC
Area under the curve. One number from 0 to 1 that sums up the ROC curve.

Two rates to track

At any threshold, a model catches some real yeses and raises some false alarms. The ROC curve tracks one number for each, as a share.

TPR=yeses caughtall real yesesFPR=false alarmsall real nos\text{TPR} = \frac{\text{yeses caught}}{\text{all real yeses}} \qquad \text{FPR} = \frac{\text{false alarms}}{\text{all real nos}}

In words: TPR asks, of the people who really are sick, how many did we find? FPR asks, of the people who really are healthy, how many did we scare for nothing? You want TPR high and FPR low.

Think of a smoke alarm. A very sensitive one catches every fire, but it also screams when you make toast. A relaxed one never goes off for toast, but it may miss a small fire. Turning the sensitivity dial moves you between those two. The threshold is that dial.

Google adds that the long name, receiver operating characteristic, is a holdover from WWII radar detection.

Build the curve, one step at a time

Google describes the recipe: the ROC curve is drawn by calculating the TPR and FPR at every possible threshold (in practice, at selected intervals), then graphing TPR over FPR. Let us do it with eight made-up emails: four spam (yes), four not (no), each with a score.

  1. 1

    Set the threshold above every score

    Nothing is marked spam. You catch no spam and raise no false alarms. TPR is 0 and FPR is 0. That is the bottom-left corner of the plot.

  2. 2

    Lower it a little

    The email scoring 0.95 is now marked spam. It really is spam, so TPR rises. No false alarm yet, so FPR stays 0. The point moves straight up.

  3. 3

    Lower it again

    Now 0.85 and 0.80 are marked spam. The 0.80 one is a real email, so this time FPR rises too. The point moves up and to the right.

  4. 4

    Finish below every score

    Everything is marked spam. You catch all the spam, and also flag every real email. TPR is 1 and FPR is 1. That is the top-right corner.

  5. 5

    Join the points

    Draw a line through every point in order. That line is the ROC curve.

The same walk in code, one row per threshold.

Python
# 8 made-up emails: 1 = spam (yes), 0 = not spam (no), with the model's spam score.
emails = [(0.95, 1), (0.85, 1), (0.80, 0), (0.70, 1), (0.55, 0), (0.45, 1), (0.30, 0), (0.10, 0)]

yes = sum(label for _, label in emails)
no = len(emails) - yes

print("threshold  caught(TP)  false alarms(FP)  TPR   FPR")
for threshold in [1.0, 0.9, 0.75, 0.6, 0.5, 0.4, 0.2, 0.0]:
    tp = sum(1 for s, label in emails if s >= threshold and label == 1)
    fp = sum(1 for s, label in emails if s >= threshold and label == 0)
    print(f"{threshold:>9.2f}  {tp:>10}  {fp:>16}  {tp / yes:.2f}  {fp / no:.2f}")

Output

threshold  caught(TP)  false alarms(FP)  TPR   FPR
     1.00           0                 0  0.00  0.00
     0.90           1                 0  0.25  0.00
     0.75           2                 1  0.50  0.25
     0.60           3                 1  0.75  0.25
     0.50           3                 2  0.75  0.50
     0.40           4                 2  1.00  0.50
     0.20           4                 3  1.00  0.75
     0.00           4                 4  1.00  1.00

Read the rows from top to bottom. At 0.50 the model has caught 3 of the 4 spam emails (TPR 0.75) and wrongly flagged 2 of the 4 real ones (FPR 0.50). Plot all the (FPR, TPR) pairs and you have the curve.

Tip

A good model's curve climbs steeply at first, then flattens near the top. It catches most of the yeses before it starts making many false alarms. That is a curve hugging the top-left corner.

The line of random guessing

Draw a straight line from the bottom-left corner (0, 0) to the top-right corner (1, 1). This is the line of a model that guesses.

Google says it directly: a model that does exactly as well as random guesses or coin flips has a ROC that is a diagonal line from (0,0) to (1,1). Wikipedia calls it the line of no-discrimination.

So the diagonal is your yardstick. A curve above it is better than guessing. A curve below it is worse than guessing. Google adds that if a classifier reliably puts examples in the wrong classes more often than chance, switching the labels immediately makes it better than chance, with no retraining.

Tip

A perfect model, at some threshold, has a TPR of 1.0 and an FPR of 0.0. That is the top-left corner, point (0, 1). The nearer the curve gets to that corner, the better.

AUC: the area under the curve

With ten models you want one number. The plot is a square with sides of length 1, so its total area is 1. The AUC is the part of that square under your curve.

  • AUC = 1.0

    A perfect model. Google: the perfect model's curve with a square of sides of length 1 has an AUC of 1.0.

  • AUC = 0.5

    A model that guesses. The area under the diagonal is half the square. Google gives the AUC of random guessing as 0.5.

  • AUC < 0.5

    Worse than chance. Google says an AUC lower than 0.5 means the model performs worse than chance.

What the area really means

Here is the plain meaning, straight from Google: the AUC represents the probability that the model, if given a randomly chosen positive and negative example, will rank the positive higher than the negative.

Say it as a game. Pull one real yes and one real no from the pile at random. If the yes has the higher score, the model wins that round. AUC is the share of rounds it would win. At 1.0 it always wins. At 0.5 it wins half the time.

We can play the game exactly on our eight emails. There are 4 yeses and 4 nos, so 16 pairs. We count the pairs where the yes has the higher score. Then we also add up the area under the curve with the trapezoid rule: split the area into thin strips with slanted tops and add the strips. The two methods should land on the same number.

Python
yes_scores = [0.95, 0.85, 0.70, 0.45]
no_scores = [0.80, 0.55, 0.30, 0.10]

# Way 1: count every yes/no pair and see who scores higher.
wins = sum(1 for y in yes_scores for n in no_scores if y > n)
pairs = len(yes_scores) * len(no_scores)
print(f"pairs: yes scored higher in {wins} of {pairs}, AUC = {wins / pairs:.4f}")

# Way 2: slide the threshold down one score at a time, then add up the area.
points = [(0.0, 0.0)]
for cut in sorted(yes_scores + no_scores, reverse=True):
    tpr = sum(1 for s in yes_scores if s >= cut) / len(yes_scores)
    fpr = sum(1 for s in no_scores if s >= cut) / len(no_scores)
    points.append((fpr, tpr))

area = sum((x2 - x1) * (y1 + y2) / 2 for (x1, y1), (x2, y2) in zip(points, points[1:]))
print(f"curve: {len(points)} points, area = {area:.4f}")

Output

pairs: yes scored higher in 13 of 16, AUC = 0.8125
curve: 9 points, area = 0.8125

Both ways give 0.8125. The model wins 13 of 16 rounds, and the area under the curve is the same 13 out of 16, written as a decimal. Wikipedia says the AUC is closely related to the Mann–Whitney U, which tests whether positives are ranked higher than negatives.

AUC=pairs where the yes scores higherall yes/no pairs\text{AUC} = \frac{\text{pairs where the yes scores higher}}{\text{all yes/no pairs}}

In words: count the pairs where the yes beats the no, and divide by the number of pairs. A tie counts as half a win.

Comparing models without a threshold

Here is the big benefit. A metric like F1 changes when you move the threshold. So comparing two models by F1 really compares them at one chosen cut-off. Scikit-learn's guide notes that, compared with metrics such as the F1 score, ROC doesn't require optimizing a threshold for each label.

AUC looks at the ranking only. As Google puts it, the AUC gives the probability of ranking a positive above a negative, independent of where the threshold is set. Google adds that the model with the greater area under the curve is generally the better one, as long as the dataset is roughly balanced.

The curve also helps you pick a threshold later. Google notes that the points on a ROC curve closest to (0,1) represent a range of the best-performing thresholds, and that the one you choose depends on what matters for the use case. If false alarms are very costly, pick a point further left on the curve. If misses are very costly, pick a point further up.

Worked example: three made-up models

Now a bigger test. We invent 200 examples: 60 real yeses and 140 real nos. We give them scores from three made-up models. Model A separates them fairly well, B only a little, and C not at all: it is a coin flip. A fixed seed makes the numbers repeatable.

For each model, the code builds the ROC points, then finds the AUC twice: by the trapezoid rule and by counting pairs. It also prints a few of Model A's points.

Python
import random

random.seed(7)

def make_scores(n_yes, n_no, gap):
    """Made-up scores between 0 and 1. A bigger gap means a better model."""
    yes = [min(1.0, max(0.0, random.gauss(0.5 + gap, 0.2))) for _ in range(n_yes)]
    no = [min(1.0, max(0.0, random.gauss(0.5 - gap, 0.2))) for _ in range(n_no)]
    return yes, no

def roc_points(yes, no):
    """One (FPR, TPR) point for every threshold, from strictest to loosest."""
    cuts = sorted(set(yes + no), reverse=True)
    points = [(0.0, 0.0)]
    for cut in cuts:
        tpr = sum(1 for s in yes if s >= cut) / len(yes)
        fpr = sum(1 for s in no if s >= cut) / len(no)
        points.append((fpr, tpr))
    return points

def auc_trapezoid(points):
    area = 0.0
    for (x1, y1), (x2, y2) in zip(points, points[1:]):
        area += (x2 - x1) * (y1 + y2) / 2
    return area

def auc_pairs(yes, no):
    """Chance that a random yes scores higher than a random no (a tie counts half)."""
    wins = 0.0
    for y in yes:
        for n in no:
            if y > n:
                wins += 1
            elif y == n:
                wins += 0.5
    return wins / (len(yes) * len(no))

models = {
    "A (strong)": make_scores(60, 140, 0.15),
    "B (weak)": make_scores(60, 140, 0.05),
    "C (coin flip)": make_scores(60, 140, 0.0),
}

for name, (yes, no) in models.items():
    points = roc_points(yes, no)
    print(f"Model {name}: trapezoid AUC = {auc_trapezoid(points):.4f}, pair-count AUC = {auc_pairs(yes, no):.4f}")

yes, no = models["A (strong)"]
print()
print("Model A, ROC points at a few thresholds:")
print("threshold   FPR    TPR")
for t in [0.8, 0.5, 0.3]:
    tpr = sum(1 for s in yes if s >= t) / len(yes)
    fpr = sum(1 for s in no if s >= t) / len(no)
    print(f"{t:>9.1f}  {fpr:.3f}  {tpr:.3f}")

Output

Model A (strong): trapezoid AUC = 0.8567, pair-count AUC = 0.8567
Model B (weak): trapezoid AUC = 0.6883, pair-count AUC = 0.6883
Model C (coin flip): trapezoid AUC = 0.5770, pair-count AUC = 0.5770

Model A, ROC points at a few thresholds:
threshold   FPR    TPR
      0.8  0.014  0.167
      0.5  0.250  0.800
      0.3  0.643  0.983

For every model, the two methods match to four decimals.

Model A scores 0.8567: about 86 of every 100 random yes/no pairs are ranked correctly. Model B scores 0.6883, about 69. Model C, the coin flip, scores 0.5770, which is not exactly 0.5. That is normal. With only 60 yeses and 140 nos, a random model wobbles. Wikipedia says that as the sample size increases, a random classifier's ROC point tends towards the diagonal line.

Now the table. At a threshold of 0.5, Model A catches 80% of the yeses (TPR 0.800) and wrongly flags 25% of the nos (FPR 0.250). Lower it to 0.3 and it catches 98.3% of the yeses, but flags 64.3% of the nos.

Quick check

Model A has an AUC of 0.8567. What does that number mean?

When yeses are very rare

ROC-AUC has a weak spot. Google says AUC and ROC work well for comparing models when the dataset is roughly balanced between classes. When the dataset is imbalanced, precision-recall curves (PRCs) and the area under those curves may offer a better comparative visualization of model performance.

A precision-recall curve plots precision on the y-axis and recall on the x-axis across all thresholds. Scikit-learn's example page agrees: Precision-Recall is a useful measure of success of prediction when the classes are very imbalanced.

Why does the ROC view struggle here? FPR divides false alarms by all the real nos. When nos are everywhere, even many false alarms are a tiny share of them, so FPR stays small and the curve looks good. Precision divides by what the model flagged, so a pile of false alarms drags it down at once. Wikipedia warns that a high ROC AUC, such as 0.9, might correspond to low values of precision and negative predictive value, such as 0.2 and 0.1.

Tip

If the yes you care about is rare (fraud, a rare illness), draw the precision-recall curve beside the ROC curve. If they disagree, trust the one that matches the question you are asking.

What AUC cannot see

AUC only cares about the order of the scores. It does not care whether a 0.9 really means a 90% chance. Squash every score towards the middle and the AUC does not move, yet the numbers stop meaning anything as chances. How sure a model was is a different question, and the next lesson, log loss, answers it.

What to remember

  • The curve

    Slide the threshold from strict to loose. At each stop, plot the share of yeses caught (TPR) against the share of nos wrongly flagged (FPR).

  • The area

    AUC is 0.5 for guessing and 1.0 for perfect. It is the chance that a random yes outscores a random no.

  • The limits

    AUC compares models without a threshold, but it can mislead on very rare yeses and says nothing about how sure the scores are.

Next: log loss, scoring how sure a model was.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.