esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.20

Regression: predicting a number

Regression is the task of predicting a number, like a price or a delivery time. Learn to measure how wrong a guess is, and how to tell a real model from a lazy one.

14 min readFree

From a box to a number

In the last lesson, the model's job was to pick a box: spam or not spam, cat or dog. Now the question changes. You do not want a box. You want an amount. How much will this flat rent for? How many minutes until the pizza arrives? What will the temperature be at noon?

Predicting an amount is called regression. The name sounds odd, but the idea is plain: look at what you know about something, and guess a number for it.

Wikipedia describes regression analysis as a method for estimating the relationship between a dependent variable and one or more independent variables. It adds that the dependent variable is often called 'a label in machine learning parlance', and that the independent variables are often called 'features'. You met both words in the supervised learning lesson. The label is the number you want. The features are the facts you already have.

Key words

Regression
The task of predicting a number from features. Wikipedia calls it estimating the relationship between a label and one or more features.
Prediction
The number the model guesses for one example.
Actual value
The real number, from the data. Also called the true value or the label.
Residual
The actual value minus the prediction. It is the error for one example, with a sign: plus if you guessed too low, minus if too high.
Loss
Google: 'a numerical metric that describes how wrong a model's predictions are'. A smaller loss is better.
Baseline
A very simple guess you compare your model against. If your model cannot beat it, the model is not helping.
Outlier
A value far outside the usual range, or a prediction that lands far from the real value.

How it differs from classification

Both tasks learn from examples that come with answers. The difference is the answer itself.

  • Classification

    The answer is one choice from a short list. Is this photo a bicycle or a car? You are either right or wrong, nothing in between.

  • Regression

    The answer is a number on a scale. A delivery that takes 31 minutes when you said 30 is nearly right. One that takes 60 is very wrong.

  • Why it matters

    In regression, being wrong comes in sizes. So we need a way to measure how wrong, not just whether wrong.

That last point drives the whole lesson. With a number, a guess is almost never exactly right. A house will not sell for exactly the price you predicted. So we stop asking 'right or wrong?' and start asking 'how far off?'

Tip

A quick test from the supervised learning lesson still works: look at the label. If the possible answers fit in a short list, it is classification. If the answer can land anywhere on a scale, it is regression.

Residuals: the error for one example

Say a model guesses a house will sell for 232.5 thousand, and it really sells for 230 thousand. The miss is 2.5 thousand. A residual is that miss, written as the actual value minus the prediction.

residual=actual value−prediction\text{residual} = \text{actual value} - \text{prediction}

In words: take the real number, take away the guess. If the answer is positive, the model guessed too low. If it is negative, the model guessed too high. If it is zero, the guess was perfect.

Here are three houses from the example we build later, with the guess coming from a size-based rule. Prices are in thousands.

Python
# the first three houses from the worked example: size rule vs real price
actual = [230, 181, 252]
guess = [232.5, 177.5, 255.0]

for a, g in zip(actual, guess):
    residual = a - g
    print(f"real {a}, guess {g}, residual {residual:+.1f}")

Output

real 230, guess 232.5, residual -2.5
real 181, guess 177.5, residual +3.5
real 252, guess 255.0, residual -3.0

One number per house. But a model is judged on many houses at once, and we want one score for all of them. The obvious idea is to add the residuals up. That does not work well. A miss of -3 and a miss of +3 cancel to zero, and a model that is wrong in both directions would look perfect. We need to remove the sign first.

Measuring error in one number

There are two common ways to remove the sign: drop it, or square it. Squaring makes every number positive, since a negative times a negative is positive. Each way gives a score. Google's crash course lists both, and the scikit-learn user guide has them as regression metrics.

  1. 1

    1. Mean absolute error (MAE)

    Ignore the sign of each residual. Add them up. Divide by how many examples there are. Google calls this the average of the L1 losses.

  2. 2

    2. Mean squared error (MSE)

    Square each residual. Add them up. Divide by how many examples there are. Google calls this the average of the L2 losses.

  3. 3

    3. Root mean squared error (RMSE)

    Take the MSE, then take its square root. Google's table defines it as 'the square root of the mean squared error'.

Here are the three formulas. Here n is the number of examples, y is the actual value, and the y with a hat is the prediction. The scikit-learn guide writes MAE and MSE in this same way.

Mean absolute error

MAE=1n∑i=1n∣yi−y^i∣\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right|

In words: for each example, find how far the guess was from the real value, ignoring direction. Then take the average. The bars mean 'drop the minus sign'.

MAE is the easiest score to read. It is in the same unit as your label. An MAE of 4.1 on prices in thousands means 'on average, the guess is off by about 4.1 thousand'. You can say that to a client and they will understand.

Mean squared error

MSE=1n∑i=1n(yi−y^i)2\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2

In words: for each example, find the miss and multiply it by itself. Then take the average. Wikipedia describes the MSE as 'the average squared difference between the estimated values and the true value'.

The catch is the unit. If prices are in thousands, squared misses are in 'thousands times thousands'. That is hard to picture. An MSE of 27 does not mean 'off by 27'. This is why the next score exists.

Root mean squared error

RMSE=1n∑i=1n(yi−y^i)2\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2}

In words: do the MSE, then undo the squaring with a square root. Wikipedia notes that taking the square root of the MSE gives the root-mean-square error, 'which has the same units as the quantity being estimated'. So the RMSE is back in thousands, or minutes, or degrees.

Tip

The three scores answer the same question, 'how far off are we?', but they weigh big misses differently. Lower is better for all three, and zero would mean every guess was perfect.

The mean baseline: a lazy guess to beat

A score on its own means little. Is an RMSE of 5.2 good? It depends on the problem. So before we trust any model, we compare it with the laziest guess we can think of: always predict the average label from the training data, and ignore the features completely.

The scikit-learn guide describes exactly this. It says a simple sanity check is 'comparing one's estimator against simple rules of thumb'. Its DummyRegressor has a strategy called mean, which 'always predicts the mean of the training targets'. The guide adds that in all of these strategies, 'the predict method completely ignores the input data'.

Think of a delivery shop that tells every customer '30 minutes', whatever the street. That is the mean baseline. If your model is not clearly better than that, it has learned nothing useful.

  • Beat it

    Your scores are clearly lower than the baseline's. The features carry real information, and the model found it.

  • Tie it

    The scores are about the same. The model is no better than a shrug. Check the data and the features.

  • Lose to it

    Worse than the baseline means something is broken: a bug, bad data or a model that memorised.

Worked example: house size to price

Now let us do it with real numbers. We make up eight houses with a fixed random seed, so the numbers are the same every time you run it. A seed is a starting number for the random generator. Same seed, same 'random' numbers. We use only Python's standard library.

The made-up rule behind the data: a house costs about 2.5 thousand per square metre, plus 30 thousand, plus some noise. The noise stands for everything size cannot explain, like the street or the age of the kitchen. Real data is never perfectly tidy.

Python
import random

random.seed(7)
sizes = [random.randint(40, 140) for _ in range(8)]
prices = [round(2.5 * s + 30 + random.gauss(0, 12)) for s in sizes]

for s, p in zip(sizes, prices):
    print(f"{s} m2 -> {p} thousand")

Output

81 m2 -> 230 thousand
59 m2 -> 181 thousand
90 m2 -> 252 thousand
123 m2 -> 337 thousand
46 m2 -> 141 thousand
49 m2 -> 154 thousand
108 m2 -> 311 thousand
52 m2 -> 167 thousand

Bigger houses cost more. Now we compare two guessers. The mean baseline guesses the same average price for every house. The size rule guesses 2.5 times the size plus 30, which is a straight-line rule we wrote by hand. Choosing such a rule automatically is what the regression models in the next lessons do. Then we score both with all three measures.

Python
import math
import random

random.seed(7)
sizes = [random.randint(40, 140) for _ in range(8)]
prices = [round(2.5 * s + 30 + random.gauss(0, 12)) for s in sizes]

def mae(actual, guess):
    return sum(abs(a - g) for a, g in zip(actual, guess)) / len(actual)

def mse(actual, guess):
    return sum((a - g) ** 2 for a, g in zip(actual, guess)) / len(actual)

def rmse(actual, guess):
    return math.sqrt(mse(actual, guess))

average = sum(prices) / len(prices)
baseline = [average] * len(prices)
rule = [2.5 * s + 30 for s in sizes]

print(f"average price: {average:.1f}")
for name, guess in [("mean baseline", baseline), ("size rule", rule)]:
    print(f"{name}: MAE {mae(prices, guess):.1f}, MSE {mse(prices, guess):.1f}, RMSE {rmse(prices, guess):.1f}")

Output

average price: 221.6
mean baseline: MAE 60.9, MSE 4750.0, RMSE 68.9
size rule: MAE 4.1, MSE 27.0, RMSE 5.2

Read the result in plain words. The lazy guess of 221.6 thousand for every house misses by about 60.9 thousand on average. The size rule misses by about 4.1 thousand. That is roughly fifteen times smaller. The size of the house really does carry information about its price, and even a simple rule picks it up.

Notice that the MSE and RMSE move together. A lower MSE always means a lower RMSE. And the RMSE (5.2) is a bit larger than the MAE (4.1) for the size rule. That gap is not random. It comes from squaring, which we look at next.

Quick check

A model has a mean absolute error of 3 on delivery times in minutes. What does that tell you?

Outliers and why squaring is harsh

Google's crash course defines an outlier with cars. Cars are normally between 2,000 and 5,000 pounds. An 8,000-pound car sits outside the typical range, so it is an outlier. A model can also make an outlier of its own: a prediction that lands far from the real value.

Squaring changes how such a miss counts. A miss of 2 squares to 4. A miss of 10 squares to 100. Ten is five times bigger than two, but its square is twenty-five times bigger. A big miss does not just add up, it takes over.

Wikipedia says it directly: 'Like variance, mean squared error has the disadvantage of heavily weighting outliers.' It explains that this is 'a result of the squaring of each term, which effectively weights large errors more heavily than small ones'. Google's crash course agrees: 'MSE moves the model more toward the outliers, while MAE doesn't'.

Let us see it with four houses. First the misses are all small. Then one of them is far off, which could be a typo in the data or a very odd house. We score both with all three measures.

Python
import math

calm = [3, -2, 4, -3]          # four small misses, in thousands
wild = [3, -2, 4, -30]         # same, but the last one is far off

for name, errors in [("calm", calm), ("one big miss", wild)]:
    mae = sum(abs(e) for e in errors) / len(errors)
    mse = sum(e ** 2 for e in errors) / len(errors)
    print(f"{name}: MAE {mae:.2f}, MSE {mse:.2f}, RMSE {math.sqrt(mse):.2f}")

Output

calm: MAE 3.00, MSE 9.50, RMSE 3.08
one big miss: MAE 9.75, MSE 232.25, RMSE 15.24

One big miss out of four. The MAE rose from 3.00 to 9.75, about three times worse. The MSE rose from 9.50 to 232.25, about 24 times worse. The RMSE sits in between, at 15.24. Same data, but the squared scores shout much louder about the one bad house.

Which score should you use?

There is no single winner. Google says 'deciding whether to use MAE or MSE can depend on the dataset and the way you want to handle certain predictions'. Ask what a big miss costs you.

  1. 1

    1. Is one huge miss much worse than several small ones?

    Think of a bridge, a medicine dose or a flight time. Use MSE or RMSE. They punish the big miss hard.

  2. 2

    2. Are a few odd rows likely to be mistakes or one-offs?

    Think of a data entry typo. MAE is calmer. It will not let one strange row decide the score.

  3. 3

    3. Do you need to explain the number to someone else?

    MAE and RMSE are in the label's own unit, so they are easy to say out loud. Raw MSE is not.

Tip

You can report more than one. Seeing MAE next to RMSE is useful: when RMSE is much bigger than MAE, a few big misses are hiding in the data. Look at those houses first.

Quick check

Your model's MAE is 4 and its RMSE is 20. What is the most likely story?

Pitfalls to remember

Scores are tools, not truth. A few habits keep you honest.

  • Score on new data

    A low error on the examples the model studied proves little. Overfitting, from the tuning lesson, means the model learned its practice data too well and fails on new data. Always score on a test set.

  • Always show the baseline

    A score only means something next to a simple guess. Report both: 'RMSE 5.2, against 68.9 for always guessing the average'.

  • Look at the rows

    A number can hide a story. Sort the houses by residual and read the worst few. The bad rows often teach you more than the average.

Remember too that a prediction is a best guess, not a promise.

The regression models ahead

This lesson was about the task and its scores. The models themselves come next, and each one is a different way to turn features into a number: linear regression, polynomial regression, lasso and ridge. Each one will be judged by the same measures, and against the same baseline.

A pattern worth keeping in your head: training tries to make the loss small. Google says 'the goal of training a model is to minimize the loss'. MSE and MAE are two common choices for that loss in linear regression. So these scores are not only for grading at the end. They also steer the learning.

What to remember

Regression predicts a number. Classification picks a box. In regression, a miss has a size, so we measure it. A residual is the actual value minus the prediction, one per example. MAE averages the sizes of the misses. MSE averages their squares. RMSE is the square root of MSE, so it comes back to the label's unit. Squaring makes big misses count much more. Always compare with the mean baseline, and always score on data the model has not seen.

Next: k-nearest neighbours, the simplest model that learns from examples.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.