Machine Learning Foundations / Core / 02.36
Principal component analysis, in depth
Squeeze many columns into a few. See what a component is, how to pick how many to keep, and find one yourself in plain Python.
12 min readFree
Many columns into a few
The last lesson ended with a promise: squeezing many columns into a few. Here it is. Picture a table of people with fifty columns: height, arm length, leg length, shoe size, glove size and so on. Most of them say nearly the same thing, which is how big the person is. Do you really need fifty numbers to say that?
Principal component analysis, or PCA, finds a few new columns that hold most of the story. The Dimensionality Reduction lesson introduced it: PCA builds new columns from the old ones, each pointing along a direction where the data spreads out, and you keep the first few. Here we go deeper. How does it pick directions? How many should you keep? What do the new columns mean? Where does it fail?
Key words
- Component
- A new column made by mixing the old columns. PCA makes one for each old column, and you keep the best few.
- Variance
- How much a column spreads out. If everybody has the same value, the variance is zero.
- Covariance matrix
- A grid showing how much each column spreads and how each pair of columns moves together.
- Eigenvector
- A direction that a matrix only stretches and never turns. Each one is a direction PCA can use.
- Eigenvalue
- How much the matrix stretches its eigenvector. For PCA, it is the spread along that direction.
- Explained variance
- The share of the total spread that one component holds, such as 96.8%.
- Loading
- How much one old column adds to one component.
Tip
If the covariance matrix and eigenvalues feel hazy, the lessons on Eigenvalues and Diagonalization and on Singular Value Decomposition cover them. You can follow this lesson without them, because we rebuild what we need in code.
The direction of most spread
Picture a swarm of fireflies drifting in a dark room, a flashlight and a wall. Wherever you stand, the swarm throws a flat shadow on the wall. Stand where the swarm points straight at you, and all the shadows land in one clump. You cannot tell the fireflies apart. Stand to the side of a long, thin swarm, and the shadows spread across the whole wall. You can still see who sits where.
PCA looks for the best spot to stand. Wikipedia says the first principal component can equivalently be defined as a direction that maximizes the variance of the projected data. In plain words: the shadow with the most spread. That direction is the first component.
Then it looks for the next best spot, with one rule: at a right angle to the first. Wikipedia says the i-th principal component can be taken as a direction orthogonal to the first i − 1 principal components that maximizes the variance of the projected data. Orthogonal just means at right angles. A direction leaning toward the first would repeat it, so at a right angle, every component brings news. Wikipedia adds that the directions are linearly uncorrelated: knowing one score tells you nothing about another.
First component
The direction with the most spread. The best single column you could build.
Second component
The most spread that is left, at a right angle to the first.
Every next one
The same rule again. Each is at right angles to all before it, and holds less.
Before looking, PCA moves the swarm so its middle sits at zero: for each column, subtract that column's average. Wikipedia describes the first step as centering the values of each variable on 0 by subtracting the mean. Why? We want to measure the spread around the middle, not where the middle happens to be.
Finding it in plain Python
Let us build one. We make up 30 people with three columns: height, arm span and shoe size. One hidden "how big is this person" number lifts all three together, plus a little noise. A fixed seed gives you the same numbers each run.
How do we find the direction of most spread with no libraries? Wikipedia describes a very simple method called power iteration. Start with any vector, multiply it by the matrix, shrink it back to length 1, and repeat. It homes in on the eigenvector with the biggest eigenvalue, as long as the start is not at a right angle to it. For a covariance matrix, that eigenvector is the first component.
In words: multiply the current guess by the matrix A, then divide by its length so it stays at length 1. Wikipedia warns that it may converge slowly, with speed set by the ratio of the second-largest to the largest eigenvalue. Here the gap is big, so it settles quickly.
- 1
1. Make and centre the data
30 rows, 3 columns, each column minus its average.
- 2
2. Build the covariance matrix
A 3 by 3 grid. Each column's spread sits on the diagonal.
- 3
3. Power iteration
Multiply, shrink to length 1, repeat until the direction stops changing.
- 4
4. Read the result
The direction, its share of the spread, and one new number for each person.
import math
import random
random.seed(7)
# 30 made-up people: height, arm span and shoe size all grow together
rows = []
for _ in range(30):
size = random.gauss(0, 1) # a hidden "how big is this person"
rows.append([170 + 8 * size + random.gauss(0, 2), # height, cm
171 + 8.5 * size + random.gauss(0, 2), # arm span, cm
41 + 2 * size + random.gauss(0, 0.8)]) # shoe size
n = len(rows)
def centre(data):
means = [sum(r[j] for r in data) / n for j in range(3)]
return [[r[j] - means[j] for j in range(3)] for r in data]
def covariance(c):
return [[sum(r[i] * r[j] for r in c) / (n - 1) for j in range(3)] for i in range(3)]
def times(m, v):
return [sum(m[i][j] * v[j] for j in range(3)) for i in range(3)]
def power_iteration(cov, show=False):
v = [1.0, 0.0, 0.0]
for step in range(1, 201):
w = times(cov, v)
length = math.sqrt(sum(x * x for x in w))
new_v = [x / length for x in w]
change = max(abs(new_v[j] - v[j]) for j in range(3))
v = new_v
if show and step <= 3:
print("step", step, [round(x, 4) for x in v])
if change < 1e-12:
break
w = times(cov, v)
return v, sum(v[i] * w[i] for i in range(3)), step
centred = centre(rows)
cov = covariance(centred)
total = cov[0][0] + cov[1][1] + cov[2][2]
print("total spread:", round(total, 2))
# The first component
v, value, steps = power_iteration(cov, show=True)
print("settled after", steps, "steps")
print("direction:", [round(x, 3) for x in v])
scores = [sum(r[j] * v[j] for j in range(3)) for r in centred]
print("first five scores:", [round(s, 2) for s in scores[:5]])
# All three, taking away each one's spread before looking for the next
running = 0.0
for k in range(1, 4):
v, value, _ = power_iteration(cov)
running += value / total
print("component", k, "share", round(value / total, 3), "running total", round(running, 3))
cov = [[cov[i][j] - value * v[i] * v[j] for j in range(3)] for i in range(3)]
# Scale first: divide each column by its own spread, then look again
spreads = [math.sqrt(covariance(centred)[j][j]) for j in range(3)]
scaled = [[r[j] / spreads[j] for j in range(3)] for r in centred]
v, value, _ = power_iteration(covariance(scaled))
print("after scaling:", [round(x, 3) for x in v], "share", round(value / 3, 3))Output
total spread: 134.95 step 1 [0.7273, 0.6654, 0.168] step 2 [0.7091, 0.685, 0.1672] step 3 [0.7086, 0.6856, 0.1671] settled after 9 steps direction: [0.709, 0.686, 0.167] first five scores: [0.51, -6.56, 16.35, -14.57, -20.6] component 1 share 0.968 running total 0.968 component 2 share 0.028 running total 0.996 component 3 share 0.004 running total 1.0 after scaling: [0.582, 0.577, 0.573] share 0.947
The total spread is 134.95, the sum of the three column spreads. The guess jumped in step 1, nearly settled by step 3, and stopped moving after 9 steps. It landed on (0.709, 0.686, 0.167): the first component is mostly height and arm span, with a little shoe size. The last scores line is the shadow itself. Each person became a single number, so three columns became one.
That is one-dimensional data, found by hand. Wikipedia says the same move with two components finds the two-dimensional plane through the high-dimensional dataset in which the data are most spread out.
How many components to keep
A table with p columns gives p components. Keeping all of them squeezes nothing, so you keep the first few. Wikipedia says the proportion of the variance that each eigenvector represents can be calculated by dividing the eigenvalue corresponding to that eigenvector by the sum of all eigenvalues.
In words: one component's spread divided by the spread of all of them together. That is the explained variance. Our code prints it: 0.968, 0.028 and 0.004. To get the next components, the code takes away what each one explained, then runs power iteration on what is left.
Add the shares from the top and you get a running total: 0.968, then 0.996, then 1.0. A popular rule is to keep the fewest components whose total passes a target, such as 95%. We pass it with one component. At 99% we would keep two. The target is your choice and depends on how much loss you can accept.
A picture helps too. Wikipedia describes a scree plot, which lines up the eigenvalues from big to small and is meant to help decide how many components to retain. The start of the bend in the line, the knee, should indicate how many are retained.
Quick check
Three components hold 70%, 20% and 8% of the spread, and the rest is tiny. You want to keep at least 95%. How many do you keep?
What a component is made of
A component is a recipe. Our first one is 0.709 parts height, 0.686 parts arm span and 0.167 parts shoe size. These recipe numbers are close cousins of the loadings. Wikipedia defines loadings as eigenvectors scaled up by the square roots of their eigenvalues. The exact numbers change with that scaling, but the idea holds: a big loading means that column has a big say.
So the new columns are blends, and blends have no names. Wikipedia warns that the new data variables cannot be interpreted in the same ways that the originals were, because they are linear interpretations of the original variables. We might call ours "overall body size", but that is our guess, not something PCA tells us.
That makes a model built on components harder to explain. If a client asks which input mattered, the answer is "a blend of fifty of them". When each column must keep its meaning, choosing a few original columns, as in the Feature Selection lesson, is the better tool.
Scale first
PCA hunts for spread, so a column with big numbers looks like it has a lot to say, even when the big numbers are only a choice of unit. Height in millimetres spreads far more than height in metres. Wikipedia says PCA is sensitive to the scaling of the variables, and that this can be cured by scaling each feature by its standard deviation. The scikit-learn guide is clear that its PCA centers but does not scale the input data for each feature, so scaling is your job.
The last line of our output shows it. Without scaling, shoe size got a weight of only 0.167, because its numbers move over a small range. After dividing every column by its own spread, the direction became (0.582, 0.577, 0.573). All three columns have almost equal say, and the share is 0.947. When the columns use different units, scale first.
Tip
The Feature Scaling lesson covers the ways to do it. For PCA, a common choice is to centre each column and divide by its standard deviation.
What PCA cannot do
PCA draws straight lines. Wikipedia says it relies on a linear model: if a dataset has a pattern hidden inside it that is nonlinear, PCA can steer the analysis in the complete opposite direction of progress. Picture dots along a curved banana. They really have one dial, how far along the bend, but a straight line cannot follow the bend.
A bent shape needs a bent tool. Scikit-learn describes kernel PCA as an extension of PCA that achieves non-linear dimensionality reduction through the use of kernels. The next lesson shows another way, with neural networks that learn to compress, called autoencoders.
A second weak spot is outliers. Wikipedia says PCA is sensitive to outliers in the data that produce large errors, so it is common practice to remove outliers before computing PCA. One far-away point can drag a direction toward itself.
What PCA is good for
Speed up a model
Fewer columns means less to compute. Feed a model the first few components instead of hundreds of columns.
Draw many columns in 2D
Keep two components and you get a flat picture with one point per row. Wikipedia notes that many studies use the first two principal components in order to plot the data in two dimensions.
Remove noise
Wikipedia explains that when the first few components hold more of the total variance than the noise does, the proportionate effect of the noise is less. Drop the weak components and some noise goes with them.
One habit applies to all three. Learn the centring, scaling and directions from the training rows only, then reuse them on new rows. The scikit-learn guide says its PCA learns the components in its fit method and can be used on new data to project it on these components. If you fit on all the rows before splitting, the test data leaks in.
What to take away
PCA centres the data, then finds the direction of most spread, the spot where the flashlight casts the widest shadow. Each next direction is at a right angle and holds less. Explained variance says how much each holds, and a running total such as 95% says how many to keep. Components are recipes, so they are hard to name. Scale first, because big units shout. PCA draws straight lines only, so curves beat it.
Next: autoencoders, networks that learn to compress.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.