esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.20

Loss functions: measuring a mistake

A loss function turns "how wrong was the network?" into one number to push down. Which one you pick shapes what the network learns.

13 min readFree

A number for a mistake

The last lesson, Forward propagation, ended with a promise: loss functions, how a network measures its mistakes. A forward pass hands you an answer. Before the network can learn anything, someone has to say how wrong that answer was. And it has to be a single number, because a number is something you can push down.

Think of a darts game. Each throw lands some distance from the bullseye. If all you hear is "miss", you cannot improve. If you hear "7 centimetres left", you can adjust. A loss function is the voice that says how far, and in what way the distance counts.

Key words

Loss
A number for how wrong the model's answer was on one example. Google's Machine Learning Crash Course: a numerical metric that describes how wrong a model's predictions are.
Loss function
The rule that turns a prediction and the true answer into a loss. Wikipedia: a function that maps an event onto a real number representing some cost.
Metric
A score you report to people, such as accuracy. The model does not have to learn from it.
Outlier
An example far from the usual range. Also a prediction that is far from the true value.
Softmax
A step that turns a list of scores into chances that add up to 1.
Regularisation
Adding a penalty for complexity so the model does not just memorise its training data.
Batch
A small group of examples used for one learning step.

A recap you need: in the Regression lesson, MSE and MAE were scores for a finished model. In the Log Loss lesson, log loss scored how sure a model was. Here the same ideas wear a different hat. They steer the learning.

Loss and metric are not the same thing

Google's crash course says the goal of training a model is to minimize the loss, reducing it to its lowest possible value. So the loss is the number training pushes down, thousands of times, while the weights change.

A metric is the number you show to a person: "the model is 92% accurate". It only has to be easy to read.

Accuracy is a good case. It only counts right or wrong, so a small nudge to the weights almost never changes it. The loss reacts to every nudge. That is why a network trains on a loss and reports accuracy.

  • Loss

    The number training pushes down. It must react to small changes in the weights.

  • Metric

    The number you report. It must be meaningful to people.

  • Sometimes the same

    MAE can be both. Google notes that MAE and RMSE are easy to read because they use the same scale as the thing being predicted.

The loss landscape

Picture the loss as the height of the ground. Each setting of the weights is a spot on a map. The loss is how high the ground is there. Training is a walk downhill, looking for the lowest spot. Google's crash course calls this picture a loss surface, and shows that for a linear model it has a single bowl shape.

To walk downhill you must feel the slope under your feet. That slope is the gradient, which the Calculus and Linear regression lessons covered. If the ground has a sharp corner, the slope is not clear at that spot. Wikipedia puts it this way: for most optimization algorithms, it is desirable to have a loss function that is globally continuous and differentiable. In plain words, smooth, with a clear slope everywhere.

Try it. A model has one weight, y = w times x, and the true rule is y = 2 times x. We try several values of w and print MSE and MAE.

Python
xs = [1, 2, 3]
ys = [2, 4, 6]  # the true rule is y = 2 * x

def mse(w):
    return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)

def mae(w):
    return sum(abs(w * x - y) for x, y in zip(xs, ys)) / len(xs)

print("  w    MSE    MAE")
for w in [0, 1, 1.5, 2, 2.5, 3, 4]:
    print(f"{w:>3}  {mse(w):>5.2f}  {mae(w):>5.2f}")

Output

  w    MSE    MAE
  0  18.67   4.00
  1   4.67   2.00
1.5   1.17   1.00
  2   0.00   0.00
2.5   1.17   1.00
  3   4.67   2.00
  4  18.67   4.00

Both bottom out at w = 2, where the model is perfect. But their shapes differ. The MSE column falls and rises smoothly, like a bowl. The MAE column is a V. Going from w = 1.5 to w = 2 drops MAE by 1.00 and from w = 2 to w = 2.5 it rises by 1.00. At the very bottom of a V there is a corner.

Wikipedia names this problem: the absolute loss has the disadvantage that it is not differentiable at a = 0.

Quick check

Why do we want a loss that changes smoothly as the weights change?

Losses for numbers

When the answer is a number, the loss measures the distance between the guess and the truth. Google's crash course says loss focuses on the distance, not the direction, so every method removes the sign: take the absolute value, or square the difference.

Mean squared error (MSE) squares each miss and averages. Mean absolute error (MAE) takes each miss's size and averages. Google's crash course states the difference: when the gap is large, squaring makes the loss even larger. It also says MSE moves the model more toward the outliers, while MAE doesn't.

MSE is a strict teacher who frets over the one disastrous answer. MAE is a calm teacher who counts each unit of error the same. Google also notes that the mathematical properties of MSE often make optimization smoother.

Huber loss: the middle road

What if you want a smooth bowl near the bottom, but you do not want one wild example to dominate? Wikipedia describes the Huber loss as a loss function used in robust regression that is less sensitive to outliers in data than the squared error loss.

It is two losses glued together. For small misses, it acts like squared error. For big misses, it acts like absolute error. Wikipedia writes the rule like this, where a is the miss and delta is the point where it switches:

Lδ(a)={12a2for ∣a∣≤δ,δ(∣a∣−12δ)otherwise.L_\delta(a) = \begin{cases} \tfrac{1}{2}a^2 & \text{for } |a| \le \delta, \\ \delta\left(|a| - \tfrac{1}{2}\delta\right) & \text{otherwise.} \end{cases}

In words: if the miss is no bigger than delta, take half of its square. If it is bigger, take delta times the miss minus half of delta. Wikipedia notes the function is quadratic for small values and linear for large values, with equal values and slopes at the two switch points. So there is no corner where the two halves meet.

One wild miss, three scores

Time to see them side by side. We make six true values and six guesses that are close, then ruin the last guess so it is off by 12. The random seed is fixed, so you will get the same numbers. Delta is 1.

  1. 1

    1. Make the data

    Six true values between 20 and 40, six guesses each within about 1 of the truth.

  2. 2

    2. Add the outlier

    Push the last guess 12 above its true value.

  3. 3

    3. Score three ways

    Work out MSE, MAE and Huber over the six examples.

Python
import random

random.seed(7)

targets = [round(random.uniform(20, 40), 1) for _ in range(6)]
preds = [round(t + random.gauss(0, 1), 1) for t in targets]
preds[5] = round(targets[5] + 12, 1)  # one wild miss: the outlier

def mse(t, p):
    return sum((a - b) ** 2 for a, b in zip(t, p)) / len(t)

def mae(t, p):
    return sum(abs(a - b) for a, b in zip(t, p)) / len(t)

def huber(t, p, delta=1.0):
    total = 0
    for a, b in zip(t, p):
        e = abs(a - b)
        total += 0.5 * e ** 2 if e <= delta else delta * (e - 0.5 * delta)
    return total / len(t)

print("targets:", targets)
print("preds:  ", preds)
print("MSE   :", round(mse(targets, preds), 3))
print("MAE   :", round(mae(targets, preds), 3))
print("Huber :", round(huber(targets, preds), 3))

Output

targets: [26.5, 23.0, 33.0, 21.4, 30.7, 27.3]
preds:   [27.6, 23.4, 34.0, 21.6, 31.1, 39.3]
MSE   : 24.428
MAE   : 2.517
Huber : 2.13

Five guesses are off by 1.1 or less. One is off by 12. MSE comes out at 24.428, because 12 squared is 144, and that one miss swamps everything. MAE says 2.517: the miss is counted once, at its size. Huber says 2.13: the 12 is counted at its size, less a small amount, and the small misses are squared and halved, so they shrink.

One set of guesses, three stories. A network trained on MSE will bend hard to fix that one point. One trained on MAE or Huber will be calmer. Whether that is good depends on whether the 12 is a real signal or a data error.

Binary cross-entropy: yes or no

For two choices, the last step of the network is a sigmoid, the S-shaped curve from the Logistic regression lesson, which turns any number into a chance between 0 and 1. The loss is then log loss, which you already met: take the minus log of the chance given to what really happened.

Wikipedia says cross-entropy can be used to define a loss function in machine learning, and that the terms log loss and cross-entropy loss are used interchangeably. With two choices, it is often called binary cross-entropy.

Categorical cross-entropy and softmax

For many choices, say cat, dog or bird, the last step is softmax. Wikipedia says it converts a tuple of K real numbers into a probability distribution over K possible outcomes, and is often used as the last activation function of a neural network to normalize the output into a probability distribution over predicted output classes.

Categorical cross-entropy then looks at one thing only: the chance given to the true class. The loss is the minus log of that chance. The other chances matter only because they take their share of the total.

Three networks, one picture, true answer cat. The first is confident and right, the second cannot decide, the third is confident and wrong. The code turns each one's raw scores into chances with softmax.

Python
import math

def softmax(scores):
    exps = [math.exp(s) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

# Three classes: cat, dog, bird. The true answer is always "cat" (index 0).
cases = {
    "confident, right": [4.0, 0.5, 0.2],
    "unsure":           [1.0, 0.9, 0.8],
    "confident, wrong": [0.2, 4.0, 0.5],
}
for name, scores in cases.items():
    probs = softmax(scores)
    loss = -math.log(probs[0])
    print(f"{name:<17} probs = {[round(p, 2) for p in probs]}  loss = {loss:.3f}")

Output

confident, right  probs = [0.95, 0.03, 0.02]  loss = 0.051
unsure            probs = [0.37, 0.33, 0.3]  loss = 1.002
confident, wrong  probs = [0.02, 0.95, 0.03]  loss = 3.851

The confident-right network gave cat 0.95 and pays only 0.051. The unsure one gave cat 0.37, about a third, and pays 1.002. The confident-wrong network gave cat only 0.02 and pays 3.851. Same shape as the Log Loss lesson: sure and wrong is punished hardest.

Quick check

In the run above, the true answer is cat. Which output gets the smallest loss?

A name worth knowing: hinge loss

There are many other losses. One more is worth a single mention. Wikipedia says hinge loss is a loss function used for training classifiers, used for maximum-margin classification, most notably for support vector machines. The Support Vector Machines lesson covers the margin idea.

Adding a penalty

A loss can have more than one part. A model that only chases a low loss on its training data may memorise it. Google's crash course says regularization is a mechanism for penalizing model complexity during training. It shows the goal as minimizing loss plus lambda times complexity.

total=loss+λ⋅complexity\text{total} = \text{loss} + \lambda \cdot \text{complexity}

In words: add the loss and a penalty for complexity, and shrink the sum. Lambda, which the crash course calls the regularization rate, sets how much the penalty counts. A high value pushes towards simpler models. L2 regularization, covered in the Ridge regression lesson, measures complexity as the sum of the squares of the weights. That is the same trick as MSE: big weights cost a lot.

Averaging over a batch

A network almost never learns from one example at a time. It looks at a small group, a batch, then takes one step. So it needs one loss for the whole group, and the usual choice is the average.

Google's crash course recommends averaging the losses across all the examples. For log loss it adds a practical reason: using the mean, rather than the total, lets you decouple tuning of the batch size and the learning rate. In plain words, you can change the batch size without the loss number growing and shrinking with it.

The next run does both ideas. It averages four made-up losses from a batch, then adds an L2-style penalty for three weights, with lambda at 0.1.

Python
# Average the loss over a batch, then add a penalty for big weights.
batch_losses = [0.4, 0.1, 2.5, 0.2]
average = sum(batch_losses) / len(batch_losses)
print("batch losses:", batch_losses)
print("average loss:", average)

weights = [3.0, -2.0, 0.5]
lam = 0.1  # how much we care about the penalty
penalty = lam * sum(w ** 2 for w in weights)
print("penalty:", round(penalty, 3))
print("total to push down:", round(average + penalty, 3))

Output

batch losses: [0.4, 0.1, 2.5, 0.2]
average loss: 0.8
penalty: 1.325
total to push down: 2.125

The four losses add up to 3.2, so the average is 0.8. The weights 3.0, -2.0 and 0.5 have squares 9, 4 and 0.25, which sum to 13.25. Times 0.1, that is 1.325. The total the training pushes down is 2.125.

Matching the loss to the output layer

The loss and the last layer of the network are chosen together. The layer decides what shape the answer takes. The loss must be able to compare that shape with the truth. There are three common pairs.

  • A number

    The last layer gives a plain number, with no squashing. Use MSE, MAE or Huber, depending on how you feel about outliers.

  • Yes or no

    The last layer is a sigmoid, giving one chance. Use binary cross-entropy, also known as log loss.

  • One of many

    The last layer is softmax, giving a chance for each choice that adds up to 1. Use categorical cross-entropy.

Quick check

You are predicting tomorrow's temperature in degrees, and a few days in the data are measurement mistakes, far from the rest. Which loss is the safest first pick?

What to remember

Whatever its form, a loss ends as one number: a score for the mistakes, perhaps plus a penalty, averaged over a batch. It tells the network how far off it is.

Next: backpropagation, how a network learns from its mistakes.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.