esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.22

Linear regression

Draw the best straight line through your data, then use it to predict a number. Here is how the line is found, and when to doubt it.

13 min readFree

A line that makes predictions

Picture a class where everyone sat the same exam. You ask each student how many hours they studied, and you write down their scores. Put the points on a chart: hours along the bottom, score up the side. The dots drift upward from left to right. More study, higher score, roughly.

Now take a ruler and draw one straight line through the middle of the cloud. A student walks in and says "I studied six hours." You find six on the bottom, go up to the line, and read off a score. That is a prediction. That is linear regression.

The last lesson ended on this idea: drawing the best straight line. This lesson is about how a computer decides which line is best. It builds on the supervised learning lesson, where each example has a label (the answer we want to predict) and features (the facts we know). Here the feature is hours studied and the label is the score.

Key words

Linear regression
A way to predict a number by drawing a straight line through the data. Google: it finds the relationship between features and a label.
Weight
How steeply the line climbs. It says how much the prediction moves for each extra unit of a feature.
Bias
Where the line starts. It is the prediction when every feature is zero. Maths class calls it the intercept.
Residual
The gap between a real answer and the line's prediction for it. One gap per data point.
Loss
One number that says how wrong the model is overall. Lower is better.
Least squares
Choosing the line that makes the squared gaps add up to the smallest total.
Gradient descent
Improving a guess in many small steps, each one downhill, until it stops getting better.
Outlier
A data point far away from all the others.

The equation: weight and bias

A straight line needs only two numbers. One says where it starts. The other says how steeply it climbs.

Google writes the model for one feature as y' = b + w1x1, where y' is the predicted label, x1 is the feature, w1 is the weight and b is the bias. It points out that weight is the same idea as the slope in the maths-class line y = mx + b, and bias is the same idea as the y-intercept. Both are parameters: numbers the model works out during training.

y^=w⋅x+b\hat{y} = w \cdot x + b

Say it in words: the predicted score equals the weight times the hours studied, plus the bias.

Think of a taxi: a starting charge (the bias) plus a price per kilometre (the weight). For our exam, suppose training finds a weight of 5 and a bias of 40. A student who studied six hours gets 5 times 6, plus 40. The prediction is 70.

Quick check

A line has a weight of 5 and a bias of 40. What does it predict for a student who studied 2 hours?

Many features

Real predictions use more than one fact. Each feature gets its own weight. Google's car example adds engine size and horsepower to the car's weight.

y^=b+w1x1+w2x2+w3x3+w4x4+w5x5\hat{y} = b + w_1 x_1 + w_2 x_2 + w_3 x_3 + w_4 x_4 + w_5 x_5

In words: start at the bias, then add each feature multiplied by its own weight. Scikit-learn writes the same thing with p features and calls the weights coef_ and the bias intercept_.

What does "best line" mean?

Many lines can go through a cloud of dots. A flat line misses the upward drift. A very steep one misses every dot. We need a fair way to score a line, so we can pick the winner.

Google calls the score loss: a number that describes how wrong the model's predictions are. Loss measures the distance between each prediction and the real label. You can picture it as an arrow from each dot to the line. Short arrows, good line. Long arrows, bad line.

The gap has no direction, so every loss method removes the sign: take the absolute value, or square it. Linear regression uses the squared version. Google's L2 loss is the sum of the squared differences between predicted and actual values. Its mean squared error (MSE) is the average of those squared gaps across all the examples.

MSE=1n∑i=1n(yi−y^i)2\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2

In words: for every student, take the real score minus the predicted score, square it, then average all those numbers.

The method that picks the line with the smallest total of these squared gaps is called least squares. Wikipedia's article on ordinary least squares describes it as minimizing the sum of the squares of the differences between the observed values and the output of the linear function. Scikit-learn says its LinearRegression class does just that: it fits a linear model to minimize the residual sum of squares between the observed targets and the predicted ones.

Finding the line: two ways

  • A closed formula

    Maths gives you the answer directly. Plug the data into a formula and out come the best weight and bias. One calculation, no guessing.

  • Gradient descent

    Start with a bad guess and improve it in many small steps. Slower to explain, but it works for models where no neat formula exists.

  • The same destination

    For a straight line with squared loss, both end at the same line. Google notes that linear models have a loss surface with a single bowl shape, so a model that has converged has found the weights that give the lowest loss.

Way one: the closed formula

Wikipedia says the least-squares estimate for one feature comes from simple formulas, even suitable for working by hand. Write x-bar and y-bar for the averages of the features and labels.

w=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2b=yˉ−w xˉw = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} \qquad b = \bar{y} - w\,\bar{x}

In words: the weight compares how the feature and the label move together with how much the feature moves on its own. The bias is then set so the line passes through the average point of the data.

Way two: gradient descent

Imagine you are standing on a foggy hillside and want to reach the valley. You cannot see it. But you can feel which way the ground slopes under your feet. So you take a small step downhill, feel the slope again, and step again. When the ground goes flat, you have arrived.

That is gradient descent. Here the hill is the loss. Standing at a spot means choosing a weight and a bias. The lower you stand, the smaller the loss. Google describes the same loop in four moves.

  1. 1

    1. Start

    Begin with a weight and bias at or near zero. It will be a bad line, and that is fine.

  2. 2

    2. Measure

    Calculate the loss with the current weight and bias.

  3. 3

    3. Feel the slope

    Work out which direction to move the weight and the bias to reduce the loss.

  4. 4

    4. Step

    Move both a small amount in that direction, then go back to step 2 until the loss cannot shrink any further.

The "small amount" is the learning rate from the tuning lesson. Too low and training takes long. Too high and it bounces around the best values. When the loss stops improving, the model has converged.

w←w−r⋅slopewb←b−r⋅slopebw \leftarrow w - r \cdot \text{slope}_w \qquad b \leftarrow b - r \cdot \text{slope}_b

In words: the new weight is the old weight minus the learning rate r times the slope of the loss for that weight. The bias is updated the same way. Google notes that a steep slope means a big step, and a gentle slope means a small step, because the step is proportional to the slope.

Worked example: both ways, side by side

Let us fit the exam line both ways. The data is made up: twelve students, a score built from 40 plus 5 per hour, plus noise. A fixed seed makes it repeat. Gradient descent starts at zero with a learning rate of 0.01 and takes 3,000 steps.

Python
import random

random.seed(4)

# Made-up data: hours studied -> exam score (true line: 40 + 5 * hours, plus noise)
hours = [random.uniform(0, 10) for _ in range(12)]
scores = [40 + 5 * h + random.gauss(0, 4) for h in hours]
n = len(hours)

def mse(w, b):
    return sum((w * x + b - y) ** 2 for x, y in zip(hours, scores)) / n

# 1. The closed formula (least squares)
mx = sum(hours) / n
my = sum(scores) / n
w_ls = sum((x - mx) * (y - my) for x, y in zip(hours, scores)) / sum((x - mx) ** 2 for x in hours)
b_ls = my - w_ls * mx
print("formula:  w =", round(w_ls, 3), " b =", round(b_ls, 3), " MSE =", round(mse(w_ls, b_ls), 3))

# 2. Gradient descent: start at 0 and take small steps downhill
w, b, rate = 0.0, 0.0, 0.01
print("start:    w =", w, " b =", b, " MSE =", round(mse(w, b), 3))
for step in range(1, 3001):
    gw = sum(2 * (w * x + b - y) * x for x, y in zip(hours, scores)) / n
    gb = sum(2 * (w * x + b - y) for x, y in zip(hours, scores)) / n
    w, b = w - rate * gw, b - rate * gb
    if step in (1, 2, 3, 100, 3000):
        print("step", step, " w =", round(w, 3), " b =", round(b, 3), " MSE =", round(mse(w, b), 3))

Output

formula:  w = 4.404  b = 42.703  MSE = 17.244
start:    w = 0.0  b = 0.0  MSE = 3837.771
step 1  w = 5.597  b = 1.212  MSE = 1370.919
step 2  w = 8.394  b = 1.945  MSE = 740.758
step 3  w = 9.782  b = 2.436  MSE = 576.336
step 100  w = 8.164  b = 20.666  MSE = 170.478
step 3000  w = 4.404  b = 42.703  MSE = 17.244

The weight overshoots early (9.782 at step 3), because the bias is still too low and the weight tries to make up for it. The bias is slow to catch up, which is why we needed so many steps.

  • The formula

    A weight of 4.404, a bias of 42.703 and an error (MSE) of 17.244. One calculation, no loop.

  • Gradient descent

    It starts at an error of 3837.771 with a flat line at zero. After one step the error has dropped to 1370.919, and after 100 steps it is 170.478.

  • The finish

    After 3,000 steps gradient descent reaches the same weight, bias and error as the formula, to three decimal places. Two routes, one line.

Reading the weights

A big reason people like linear regression is that you can read it. The weights are numbers you can look at and understand, which most models do not allow.

Our fitted weight is 4.404. In words: each extra hour of study adds about 4.4 points to the predicted score. The bias of 42.703 says a student who studied zero hours is predicted to score about 43.

The sign matters too. In Google's car example, a heavier car gets fewer miles per gallon, so its weight is negative. Larger engines also go with lower mileage. A positive weight means the label rises with the feature, and a negative weight means it falls.

With many features there is a catch. Wikipedia explains that a weight is the expected change in the label for a one-unit change in that feature when the other features are held fixed. Read each weight as "with everything else the same".

What linear regression assumes

Every model bets on something. Wikipedia says that when you estimate a linear regression with standard methods such as ordinary least squares, you must make a number of assumptions about the features, the label and the relationship between them. The assumptions are what make the results trustworthy. Break them and the line may still be drawn, but it may mislead.

  • A roughly straight relationship

    The model is a straight line, or a flat surface. If the real pattern curves, a straight line will be wrong in a steady, predictable way.

  • Features that are not copies

    Scikit-learn says the coefficient estimates rely on the independence of the features. Strongly related features shake the weights.

  • No wild outliers

    Squaring makes far-away points count a great deal, so a few of them can drag the line. The next section shows it.

Outliers pull the line

Remember that squaring makes a big miss much more costly. Wikipedia spells out the result: using mean squared error on a dataset with many large outliers can give a model that fits the outliers more than the true data, because MSE gives large errors such a high importance.

Let us test it. We use the same twelve students, and add a thirteenth who studied 9 hours but scored only 20. Maybe they fell ill on the day of the exam. Then we fit the closed formula again.

Python
import random

random.seed(4)
hours = [random.uniform(0, 10) for _ in range(12)]
scores = [40 + 5 * h + random.gauss(0, 4) for h in hours]

def fit(xs, ys):
    mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
    w = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
    return w, my - w * mx

w, b = fit(hours, scores)
print("without outlier: w =", round(w, 3), " b =", round(b, 3))

# One odd student: studied 9 hours, scored 20 (say, fell ill on exam day)
w2, b2 = fit(hours + [9], scores + [20])
print("with outlier:    w =", round(w2, 3), " b =", round(b2, 3))

Output

without outlier: w = 4.404  b = 42.703
with outlier:    w = 1.909  b = 48.998

One extra student cut the weight from 4.404 to 1.909. The line went from "each hour adds about 4.4 points" to "each hour adds about 1.9". Twelve students told one story and one student nearly erased it.

Look at your data before fitting; a chart makes an outlier easy to spot. Wikipedia names methods built to cope, such as least absolute deviation regression, which it calls less sensitive to outliers than ordinary least squares. Google's loss page also offers the absolute gap (MAE) instead of the squared one.

When linear regression fails

A straight line is simple, and simple is a strength. It is also the limit. Here are the cases where it lets you down.

  1. 1

    1. The pattern curves

    Sales that rise, peak and fall, or a plant that grows fast then slows. A straight line cuts across the curve and is wrong at both ends. Wikipedia points to nonlinear regression as the generalization for such cases.

  2. 2

    2. Outliers

    A few extreme points drag the line away from everyone else, as we just saw.

  3. 3

    3. Copy-like features

    Correlated features make the weights unstable and hard to read.

  4. 4

    4. One number hides the picture

    Wikipedia shows Anscombe's quartet: datasets designed to have approximately the same regression line, and nearly identical means, standard deviations and correlations, yet graphically very different. It warns about relying solely on a fitted model to understand a relationship.

The short version

Quick check

What does least squares choose?

  • The model

    A prediction is a weight times a feature, plus a bias. With many features, add up each feature times its own weight.

  • The fit

    The best line is the one with the smallest total of squared gaps. A closed formula finds it directly. Gradient descent walks to it in small steps. Both reach the same line.

  • The care

    You can read the weights, but only when features are not copies of each other. Plot your data: a curve, an outlier or a hidden pattern will not show up in a single score.

Next: logistic regression, a straight line that answers yes or no.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.