Machine Learning Foundations / Core / 02.24
Polynomial regression: bending the line
When a straight line misses the shape of your data, add new columns made from the old one. The line bends, and the maths stays the same.
13 min readFree
When a straight line is not enough
A straight line is a fine guess when more of one thing means steadily more of another. Plenty of things do not work that way. Think of a ball thrown in the air: it climbs, slows, and falls. Or the way a plant grows best at a middle amount of sunlight, and worse with too little or too much. A line drawn through those points cuts across the bend and misses it.
The last lesson ended on exactly this: a straight line is not always enough. Here is the fix, and it is surprisingly small. You do not build a new kind of model. You hand the old one some new columns.
Key words
- Feature
- One column of input data, such as a house's size.
- Weight
- A number the model learns, one per feature. It says how much that feature counts.
- Polynomial
- A sum of powers of a number, like 3 + 2x + 5x². The biggest power is called the degree.
- Degree
- The biggest power of x the model may use. Degree 2 allows x and x². Degree 3 adds x³.
- Underfitting
- The model is too simple to catch the pattern, so it does badly even on the data it learned from.
- Overfitting
- The model follows the noise in its training data, so it does well there and badly on new data.
- MSE
- Mean squared error: the average of the squared gaps between the model's guesses and the real answers. Smaller is better.
If "linear model" or "least squares" feels hazy, the lessons on supervised learning and train and test data come first.
Add columns, keep the line
Say each row of your data has one input, x. A straight-line model uses two things: a starting value and a weight on x. Now make two more columns from x: one holding x times x, and one holding x times x times x. Give those to the same model as if they were ordinary inputs. It learns a weight for each one.
Here is what those new columns look like for three values of x, using powers 0 to 3 (the power 0 column is just 1, the starting value):
for x in (1, 2, 3):
print(x, "->", [x ** power for power in range(4)])Output
1 -> [1, 1, 1, 1] 2 -> [1, 2, 4, 8] 3 -> [1, 3, 9, 27]
Scikit-learn, a widely used Python library, has a tool for this called PolynomialFeatures. Its guide describes it as one that transforms an input data matrix into a new data matrix of a given degree. With two inputs, x1 and x2, and degree 2, it turns [x1, x2] into six columns: 1, x1, x2, x1², x1x2 and x2². Notice the mixed column, x1x2. With more than one input, the new columns also multiply inputs together.
In words: the guess is a starting value, plus one weight times x, plus another weight times x², plus another weight times x³. The curve you see on a chart comes from the powers. The weights, the part the model learns, only ever get added up.
Why it is still a linear model
This is the idea people find odd at first. The curve is bent, yet the model is called linear. The word "linear" is about the weights, not the picture.
Scikit-learn's guide says this is a (sometimes surprising) observation: the model is still a linear model. Rename the columns as z1 = x, z2 = x², z3 = x³, and the guess becomes a starting value plus w1·z1 plus w2·z2 plus w3·z3. That is an ordinary straight-line recipe, just in a space with more columns. The guide says this puts polynomial regression in the same class of linear models, and that it can be solved by the same techniques.
Wikipedia puts it the same way: polynomial regression fits a nonlinear relationship between x and y, but as a statistical estimation problem it is linear, because the regression function is linear in the unknown parameters. It is a special case of multiple linear regression.
What bends
The curve in the chart. It comes from the powers of x that you made.
What stays straight
The learning. The weights are added up, never squared or multiplied together.
Why that is good news
Everything you know about fitting a line, including the least-squares method, works unchanged. The scikit-learn guide notes it keeps the generally fast performance of linear methods.
Tip
Think of a sandwich. The model is the same cook, doing the same job. You only changed the ingredients you handed over.
Degree: how much bend to allow
Degree 1 is the plain straight line. Degree 2 allows one smooth bend, like a hill or a bowl. Degree 3 allows an S-shape. Higher degrees allow more wiggles. The degree is a hyperparameter: a setting you choose before training, not a number the model learns. The tuning lesson covers that idea in full.
Choosing it is a balancing act, like fitting a coat. Too tight and it cannot fit your shape. Too loose and it hangs everywhere. The scikit-learn example on underfitting and overfitting shows the whole range on a piece of a cosine curve with polynomial features of different degrees.
Underfitting: too stiff
The scikit-learn example says a linear function (polynomial with degree 1) is not sufficient to fit the training samples, and calls this underfitting. A stiff model is wrong in a steady way. It makes big errors on the data it was trained on, and equally big ones on new data.
The Wikipedia article on overfitting describes underfitting as when a model cannot adequately capture the underlying structure of the data.
Overfitting: too bendy
Give the model too many bends and it starts to chase every little wobble in the data. The scikit-learn example says that for higher degrees the model will overfit the training data, meaning it learns the noise of the training data. Noise is the random wiggle in measurements that carries no real pattern.
Wikipedia says the essence of overfitting is to unknowingly extract some of the residual variation (that is, noise) as if that variation represents the underlying model structure. An overfitted model, it says, contains more parameters than can be justified by the data. For a model made of a polynomial, those parameters represent the degree of the polynomial.
A student who memorises last year's exam answers scores full marks on last year's paper and little on a new one. That is overfitting.
Try it: degree 1, 2 and 6
Let us watch this happen with numbers. The data below is made up: a hill shape, 1 + 20x - 20x², with random noise added. A fixed seed means you get the same numbers every time. We keep 10 points for training and 40 for testing, the split from the train and test lesson.
To fit a degree, we make the columns, then solve least squares by hand. Least squares picks the weights that make the squared gaps smallest. The standard way to solve it is a small set of equations called the normal equations, written out here with a short routine for solving them, no libraries needed.
- 1
1. Make the data
Ten training points and forty test points from the same noisy hill.
- 2
2. Make the columns
For degree d, each x becomes [1, x, x², and so on up to x^d].
- 3
3. Solve for the weights
Build the normal equations from the training data only, then solve them.
- 4
4. Score twice
Measure MSE on the training points, then on the test points the model has never seen.
import random
random.seed(3)
def curve(x):
return 1 + 20 * x - 20 * x * x
def make(n):
xs = [random.uniform(0, 1) for _ in range(n)]
return xs, [curve(x) + random.gauss(0, 0.8) for x in xs]
train_x, train_y = make(10)
test_x, test_y = make(40)
def features(x, degree):
return [x ** p for p in range(degree + 1)]
def solve(a, b):
n = len(b)
m = [row[:] + [b[i]] for i, row in enumerate(a)]
for col in range(n):
pivot = max(range(col, n), key=lambda r: abs(m[r][col]))
m[col], m[pivot] = m[pivot], m[col]
for r in range(col + 1, n):
f = m[r][col] / m[col][col]
for c in range(col, n + 1):
m[r][c] -= f * m[col][c]
w = [0.0] * n
for i in range(n - 1, -1, -1):
w[i] = (m[i][n] - sum(m[i][j] * w[j] for j in range(i + 1, n))) / m[i][i]
return w
def fit(xs, ys, degree):
rows = [features(x, degree) for x in xs]
k = degree + 1
xtx = [[sum(r[i] * r[j] for r in rows) for j in range(k)] for i in range(k)]
xty = [sum(r[i] * y for r, y in zip(rows, ys)) for i in range(k)]
return solve(xtx, xty)
def mse(w, xs, ys):
preds = [sum(wi * f for wi, f in zip(w, features(x, len(w) - 1))) for x in xs]
return sum((p - y) ** 2 for p, y in zip(preds, ys)) / len(ys)
for degree in (1, 2, 6):
w = fit(train_x, train_y, degree)
print(f"degree {degree}: train MSE {mse(w, train_x, train_y):.3f}, test MSE {mse(w, test_x, test_y):.3f}")Output
degree 1: train MSE 2.865, test MSE 3.925 degree 2: train MSE 0.312, test MSE 2.565 degree 6: train MSE 0.279, test MSE 13.624
Read the three lines like a result table. Degree 1 is stiff: both errors are high (2.865 and 3.925), which is underfitting. Degree 2 matches the hill the data came from. Its training error drops to 0.312, and its test error is 2.565, the lowest of the three.
Degree 6 is the surprise. Its training error is lowest of all, 0.279, because six bends can hug ten points. But its test error jumps to 13.624, more than five times worse than degree 2. That gap, a tiny training error beside a large test error, is the signature of overfitting.
Degree 1
Train MSE 2.865, test MSE 3.925. Too stiff. It underfits.
Degree 2
Train MSE 0.312, test MSE 2.565. The right amount of bend for this hill.
Degree 6
Train MSE 0.279, test MSE 13.624. Looks best on the practice data, worst on new data.
Tip
Training error alone always flatters a bendier model. Each extra degree can only help or tie on the training points, so it never warns you. Only data the model has not seen can.
Quick check
A model has the lowest training error of all the ones you tried, but the highest test error. What is the likely cause?
Picking the degree without cheating
You saw that training error cannot choose the degree. And you should not choose on the test set either, because then it stops being a fair final exam.
The way out is the one from the tuning lesson. Hold back a validation set from the data you are allowed to use, fit each candidate degree on the training part, and keep the degree with the lowest validation error. The scikit-learn example takes the same line: it evaluates overfitting and underfitting with cross-validation, calculating the mean squared error on the validation set, where a higher value means the model is less likely to generalize correctly from the training data.
That example finds that a polynomial of degree 4 approximates its cosine curve almost perfectly. Notice that the best degree depends on the data. For our hill it was 2. For a cosine piece it was 4. Try a few and let the validation error decide.
Tip
Start low. Try degrees 1, 2, 3 and 4 before you try 9. If validation error stops falling, or starts rising, stop climbing.
Scaling matters for big powers
Powers grow fast. If x is a house size from 20 to 100, then x⁶ reaches a trillion, while x itself stays at 100. The feature scaling lesson explains why scales matter. Here it matters even more, because the powers make the gaps enormous.
sizes = [20, 50, 80, 100]
for degree in (1, 2, 3, 6):
print(f"size ** {degree}:", [s ** degree for s in sizes])
print()
scaled = [s / 100 for s in sizes]
for degree in (1, 2, 3, 6):
print(f"scaled ** {degree}:", [round(s ** degree, 4) for s in scaled])Output
size ** 1: [20, 50, 80, 100] size ** 2: [400, 2500, 6400, 10000] size ** 3: [8000, 125000, 512000, 1000000] size ** 6: [64000000, 15625000000, 262144000000, 1000000000000] scaled ** 1: [0.2, 0.5, 0.8, 1.0] scaled ** 2: [0.04, 0.25, 0.64, 1.0] scaled ** 3: [0.008, 0.125, 0.512, 1.0] scaled ** 6: [0.0001, 0.0156, 0.2621, 1.0]
Raw, the columns run from 100 up to a trillion. Shrunk to a 0 to 1 range first, every power stays between 0 and 1. That is why our example used x between 0 and 1: its sixth power stays tame.
Scale first
Put x on a small, common range (like 0 to 1) before you make the powers.
Same rule for new data
Scale test and future data with the numbers you worked out on the training data, never fresh ones.
Keep the degree modest
Every extra degree adds a column and a weight, and more weights mean more room to overfit.
Many inputs: columns multiply
With one input, degree 3 gives four columns. With many inputs, the mixed columns pile up, because every combination of inputs counts.
from math import comb
for inputs in (1, 2, 10, 100):
count = comb(inputs + 3, 3)
print(f"{inputs} inputs, degree 3: {count} columns")Output
1 inputs, degree 3: 4 columns 2 inputs, degree 3: 10 columns 10 inputs, degree 3: 286 columns 100 inputs, degree 3: 176851 columns
So 100 inputs at degree 3 means 176,851 columns, most of them mixed. More columns mean more weights to learn, more room to overfit, and more memory. Keep the degree low when you have many inputs, or pick which inputs deserve a bend.
A recipe to follow
- 1
1. Look first
Plot y against x. If the points bend, a line will underfit. If they look straight, stay with a line.
- 2
2. Split the data
Training, validation and test sets, as in the train and test lesson.
- 3
3. Scale
Shrink x to a small range using the training data only.
- 4
4. Try degrees
Fit degrees 1, 2, 3 and so on, and score each on the validation set.
- 5
5. Keep the lowest validation error
Not the lowest training error. Then test once, at the very end.
Quick check
Why is polynomial regression still called a linear model, even though the fitted line curves?
What to remember
The trick
Add columns such as x² and x³. The model stays linear in its weights, yet the fitted shape curves.
The dial
Degree is a hyperparameter. Too low underfits, too high overfits. Choose it with validation data, never with training error.
The care
Scale before you raise to powers, and watch the column count when you have many inputs.
Next: support vector machines, drawing the widest road between groups.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.