Machine Learning Foundations / Advanced / 03.08
Log loss: how sure was the model?
Two models can get the same number of answers right and still differ a lot. Log loss scores how sure each one was, and punishes confident mistakes.
13 min readFree
Right is not the whole story
The last lesson, on ROC-AUC, ended with a promise: log loss, scoring how sure a model was. Here it is. We start with a small story about two weather apps.
Both apps say "rain" or "no rain" for ten days. Over those days, both are right on seven. Same score. But look closer. App A said things like "60% chance of rain", and when it was wrong, it was a mild miss. App B said "99% chance of sun" on a day that poured. You would trust App A more, and you would be right to. Counting right answers cannot show that difference. Log loss can.
Many models give a chance, such as 0.8, and a cut-off turns it into yes or no. Accuracy only sees the final answer. Log loss looks at the chance itself: it rewards sure and right, and punishes sure and wrong.
Key words
- Probability
- A chance between 0 and 1. 0.8 means "80% likely".
- Log loss
- A score for a model's probabilities. Lower is better. Scikit-learn's guide says it is also called logistic regression loss or cross-entropy loss.
- Confident
- Giving a chance close to 0 or close to 1, so the model leaves itself little room to be wrong.
- Baseline
- A lazy guess a real model has to beat. Here: always say the base rate.
- Base rate
- How common "yes" is in the data overall. If 30 of 100 cases are yes, it is 0.30.
- Calibration
- Whether a model's chances match real life. A model that says 70% should be right about 70% of the time.
- Clipping
- Nudging a chance a tiny bit away from exactly 0 or 1, so the maths never breaks.
A recap: in the Logistic regression lesson, log loss was the loss the model shrinks while it learns. Here the same number judges a finished model on test data.
The formula, in words first
Start with one example. The true answer is yes. The model said the chance of yes is p. Log loss for this one example is the negative of the natural logarithm of p. A logarithm ("log") tells you what power you would need. You do not need to compute it by hand. You only need its shape: when p is close to 1, the log is close to 0, and when p is close to 0, the log falls very fast.
Flipping the sign makes the loss positive: a high chance on the truth gives a small loss, a tiny chance gives a big one.
If the true answer is no, the same idea applies to the chance of no, which is 1 minus p. Both cases fit in one line, and the average over all your examples is the score. Google's Machine Learning Crash Course writes it like this:
Read it in words. N is how many examples you have. For each example i, y is the true label (1 for yes, 0 for no) and y' is the model's chance of yes. The square brackets hold two terms, but only one is live each time, because the other is multiplied by zero. Then you add them all up, divide by N to get the average, and put a minus sign in front.
Scikit-learn's guide calls it the negative log-likelihood of the classifier given the true label, defined on probability estimates. Wikipedia notes that "log loss" and "cross-entropy loss" are used interchangeably.
Tip
Think of it as an average surprise. Each example gets a surprise score: how shocked should we be, given the chance the model gave to what really happened? Log loss is the average of those scores.
Why confident and wrong costs so much
Let us see the cost curve as plain numbers. The true answer is yes every time. We try four chances the model might have given to yes.
import math
# The true answer is "yes". The model says p is the chance of yes.
for p in [0.9, 0.5, 0.1, 0.01]:
loss = -math.log(p)
print(f"p = {p:<4} loss = {loss:.3f}")Output
p = 0.9 loss = 0.105 p = 0.5 loss = 0.693 p = 0.1 loss = 2.303 p = 0.01 loss = 4.605
Saying 0.9 costs 0.105. A shrug at 0.5 costs 0.693. Saying 0.1 costs 2.303. Saying 0.01 costs 4.605, which is more than 40 times the cost of the 0.9 answer.
From 0.9 down to 0.5 the cost rises by about 0.59. From 0.1 down to 0.01 it rises by about 2.3. A wrong answer said quietly is a small mistake. Said loudly, it is a big one. A doctor who says "probably fine, but let us check" and is wrong did less harm than one who says "definitely fine".
Sure and right
A chance near 1 for what happened. The loss is close to 0. This is what we want.
Unsure
A chance of 0.5 either way. The loss is 0.693. It is not a disaster, but it is not skill either.
Sure and wrong
A chance near 0 for what happened. The loss grows without a ceiling. One such answer can ruin an average.
Quick check
The true answer is yes. Which model gets the bigger loss on this one example?
Lower is better, and what is a good number?
Log loss is a cost, so lower is better. A perfect model, with a chance of 1 on every true answer, scores 0, and nothing scores below 0. Scikit-learn's tools treat higher as better, so its name for this score is neg_log_loss: the same number with a minus in front.
But what counts as good? On its own a log loss value is hard to read, because it depends on how common yes is. You need a baseline.
The baseline for log loss is a model that ignores every detail of each case and always gives the same chance: the base rate. If 30% of cases are yes, it says 0.30 every time. It knows nothing about any single case, but it is honest about how common yes is.
With base rate b, its log loss is:
In words: yes costs the log of b and happens a share b of the time; no costs the log of 1 minus b for the rest. Real numbers come in the worked example.
The rule is the one from the Model evaluation lesson: a model that cannot beat its baseline has not learned anything useful. A log loss that is worse than the base-rate baseline means the model's chances are doing harm, even if its yes and no answers look fine.
Clipping: never say exactly 0 or 1
There is a trap. If a model says the chance of yes is exactly 0 and the answer is yes, the loss is the log of 0. That cannot be worked out. In maths it heads to infinity, and in code it usually crashes or gives a useless value that ruins your average.
The fix is clipping. Before taking the log, nudge any chance that is exactly 0 or 1 a tiny step inwards. Libraries do something like this for you. In your own code, a small number such as 1e-15 (a 1 with fifteen zeros in front) works well. Clipping barely changes good answers. It only stops one extreme answer from breaking everything.
import math
EPS = 1e-15
def clip(p):
# Keep p a hair away from 0 and 1 so the log can always be worked out.
return min(max(p, EPS), 1 - EPS)
try:
math.log(0.0)
except ValueError as err:
print("log(0) fails:", err)
print("loss for a sure 'no' when the truth is yes:", round(-math.log(clip(0.0)), 1))
print("loss for a sure 'yes' when the truth is yes:", round(-math.log(clip(1.0)), 12))Output
log(0) fails: expected a positive input, got 0.0 loss for a sure 'no' when the truth is yes: 34.5 loss for a sure 'yes' when the truth is yes: 0.0
The first line is Python refusing the log of 0 (the wording differs between versions). After clipping, the same terrible answer costs 34.5: huge next to 4.605, but a number the average can survive.
Calibration: do the chances mean what they say?
Here is a different question. When a model says 70%, is it right 70% of the time? That match between what a model says and what happens is called calibration.
Scikit-learn's guide gives the test: among the samples a well calibrated classifier gave a probability close to 0.8, about 80% should truly be positive. Wikipedia quotes the same idea from forecasting: of events given a 30 percent chance, about 30 percent should happen in the long run.
Wikipedia adds that a classifier might separate the classes well and still be poorly calibrated. It may sort cases in the right order, which ROC-AUC rewards, while its numbers are far too bold or too timid. Log loss cares about those numbers.
Scikit-learn's guide warns that log loss measures calibration and other things at once, so a low value does not by itself prove good calibration. That is why we also build a table.
A reliability table, step by step
A reliability table checks calibration by counting. Scikit-learn calls the picture version a calibration curve, also known as a reliability diagram, and builds it by binning the predictions. A bin is a range of chances.
- 1
Bin and place
Cut the chances into five bins (0.0 to 0.2, 0.2 to 0.4, and so on) and put each case in its bin.
- 2
Average what the model said
Inside a bin, add up the chances and divide by the number of cases. That is the bin's "said" number.
- 3
Count what really happened
Inside the same bin, find the share of cases that were truly yes. That is the "came true" number.
- 4
Compare the two columns
If they match, the bin is well calibrated. If "said" is far above "came true", the model was too bold.
Small bins are noisy, so read the big ones first.
More than two answers
Log loss is not only for yes or no. Say a model looks at a photo and gives a chance to three answers: cat, dog and bird, adding up to 1. The idea stays the same. Look only at the chance the model gave to the true answer, take the negative log, and average over all the photos.
Scikit-learn's guide writes this with a table where each row marks the true class with a 1 and the others with 0, so the sum picks out the true class's chance. The yes or no formula above is the two-answer case. If the true answer is bird and the model gave the bird only 0.1, that one photo costs the same as the 0.1 case in the cost curve earlier, however the other chances were spread.
Worked example: same accuracy, different confidence
Now the main test. We build two made-up models and give them the same accuracy on purpose. Then we let log loss and a reliability table tell them apart.
The setup: 2,000 cases, each with a hidden true chance of yes, and real answers drawn from those chances with a fixed seed. Honest reports the true chance. Bold pushes each chance away from 0.5 without crossing it. Both say yes and no in the same places, so their accuracy must match.
import math
import random
random.seed(11)
EPS = 1e-15
def clip(p):
return min(max(p, EPS), 1 - EPS)
def log_loss(truth, probs):
total = 0.0
for y, p in zip(truth, probs):
p = clip(p)
total += -(y * math.log(p) + (1 - y) * math.log(1 - p))
return total / len(truth)
def accuracy(truth, probs):
hits = sum(1 for y, p in zip(truth, probs) if (p >= 0.5) == (y == 1))
return hits / len(truth)
def sharpen(p, k):
# Push a probability away from 0.5 without crossing it.
return p**k / (p**k + (1 - p)**k)
def reliability(truth, probs, bins=5):
rows = []
for b in range(bins):
lo, hi = b / bins, (b + 1) / bins
idx = [i for i, p in enumerate(probs)
if lo <= p < hi or (b == bins - 1 and p == 1.0)]
if idx:
said = sum(probs[i] for i in idx) / len(idx)
came = sum(truth[i] for i in idx) / len(idx)
rows.append((lo, hi, len(idx), said, came))
return rows
# 2000 made-up cases. "true_chance" is how likely yes really is for each one.
true_chance = [random.betavariate(1, 2.2) for _ in range(2000)]
truth = [1 if random.random() < q else 0 for q in true_chance]
honest = [min(max(q, 0.001), 0.999) for q in true_chance] # says what it believes
bold = [sharpen(q, 3) for q in honest] # same side of 0.5, far surer
base = sum(truth) / len(truth)
baseline = [base] * len(truth)
print("share of yes (the base rate):", round(base, 3))
print("baseline (always say the base rate) log loss:", round(log_loss(truth, baseline), 3))
print()
for name, probs in [("Honest", honest), ("Bold", bold)]:
print(name)
print(" accuracy:", round(accuracy(truth, probs), 3))
print(" log loss:", round(log_loss(truth, probs), 3))
print(" said range cases said came true")
for lo, hi, n, said, came in reliability(truth, probs):
print(f" {lo:.1f}-{hi:.1f} {n:5d} {said:.2f} {came:.2f}")
print()Output
share of yes (the base rate): 0.325 baseline (always say the base rate) log loss: 0.63 Honest accuracy: 0.75 log loss: 0.499 said range cases said came true 0.0-0.2 790 0.09 0.10 0.2-0.4 548 0.29 0.31 0.4-0.6 382 0.49 0.50 0.6-0.8 219 0.69 0.68 0.8-1.0 61 0.85 0.92 Bold accuracy: 0.75 log loss: 0.732 said range cases said came true 0.0-0.2 1308 0.03 0.18 0.2-0.4 167 0.28 0.46 0.4-0.6 136 0.50 0.48 0.6-0.8 133 0.71 0.59 0.8-1.0 256 0.93 0.74
- 1
The base rate
About 32.5% of the cases were yes. The lazy baseline says 0.325 every time, and its log loss is 0.63. That is the number to beat.
- 2
Accuracy
Both models score 0.75. By accuracy alone they look identical.
- 3
Log loss
Honest scores 0.499, comfortably better than the baseline's 0.63. Bold scores 0.732, which is worse than the lazy baseline, even though it gets exactly as many answers right.
- 4
Honest's table
"Said" and "came true" sit close together in every bin. The top bin has only 61 cases, so its gap (0.85 against 0.92) is likely noise.
- 5
Bold's table
In its biggest bin, 1,308 cases, Bold said about 0.03 but 18% came true. In the top bin it said 0.93 and only 74% did. It is overconfident at both ends.
That is the point: one accuracy, two very different stories. Reporting accuracy alone would hide that Bold's chances are not safe to use. Always print the baseline next to the score.
When to reach for log loss
Reach for log loss when the chance itself matters: risk scores, prices, any number a person will act on, or a tie between two models with the same accuracy. One very confident miss can dominate the average, so also look at the cases that cost the most, and use a reliability table when calibration is the real question.
Use chances
Ask the model for probabilities, not only its final yes or no.
Beat the baseline
Compare its log loss with always saying the base rate.
Check the table
Do "said" and "came true" match in each bin?
In short: accuracy counts how often a model was right. Log loss asks how sure it was while being right or wrong.
Next: validation techniques, testing a model fairly.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.