esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.04

Dimensionality reduction

Data with hundreds of columns is hard to see, slow to use and easy to get wrong. Dimensionality reduction keeps the story and drops the extra pages.

14 min readFree

What a dimension is

Picture a spreadsheet of people. One column holds age, another holds height, another holds hours of sleep. Each column is one dimension. You can think of a dimension as one direction you can move in. Age is a line you can slide along. Age and height together make a flat map. Add a third column and the map becomes a room.

We can draw one, two or three columns. We cannot draw fifty. Yet real data can have many more. A grey photo of 100 by 100 pixels already has 10,000 numbers, and each one is a dimension.

Dimensionality reduction means using fewer columns while keeping what matters. Wikipedia defines it as 'the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data'. Think of a shadow. A 3-D chair becomes a flat shadow, and you can often still tell it is a chair.

Key words

Dimension
One column of the data, or one direction you can move in. Ten columns means ten dimensions.
Variance
How spread out the numbers in a column are. If everyone has the same value, the variance is zero.
Covariance
Whether two columns move together. When one goes up and the other tends to go up too, the covariance is positive.
Component
A new direction that PCA builds out of the old columns.

Why many columns cause trouble

More columns sounds like more information. Often it is more trouble. The problem has a name: the curse of dimensionality. Wikipedia says it 'refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience.' The expression was coined by Richard E. Bellman.

Here is the gentle version. Test a 100-metre road for potholes every metre and you need 100 spots. A square field of that size needs 100 times 100, which is 10,000. A cube needs a million. Every new direction multiplies the work.

Wikipedia gives the same example with numbers. Sampling a line with points 0.01 apart takes 10² = 100 points. 'An equivalent sampling of a 10-dimensional unit hypercube' with the same spacing 'would require 10²⁰' sample points. Its summary: 'when the dimensionality increases, the volume of the space increases so fast that the available data becomes sparse', and so 'the amount of data needed often grows exponentially with the dimensionality.'

Sparse means thinly spread. With many columns, your rows are like a handful of seeds thrown across a huge field. They end up far from each other, so patterns are hard to find.

A second trouble is about distance. Wikipedia says that when a distance 'is defined using many coordinates, there is little difference in the distances between different pairs of points.' If everyone is about equally far from everyone else, asking 'who is nearest?' stops being useful. It adds a fair warning: recent research found this holds only in an artificial case, where the columns are independent. When columns are correlated, data can become easier. So the curse is real, but it is not a law that hits every dataset.

Wikipedia lists what reducing dimensions can help with: noise reduction, data visualization and cluster analysis.

Two ways to use fewer columns

Wikipedia says linear approaches can be divided into 'feature selection and feature extraction'. They sound alike, but they do different jobs.

  • Selection: keep some columns

    Wikipedia says feature selection 'aims to find a suitable subset of the input variables'. You pick the useful columns and throw the others away. Nothing is changed. It is like choosing five outfits from a wardrobe. The previous lesson on feature selection covers this.

  • Extraction: build new columns

    Wikipedia says feature projection, also called feature extraction, 'transforms the data from the high-dimensional space to a space of fewer dimensions'. You mix the old columns into a few new ones. It is like blending fruit into a smoothie. You get fewer things, but each one contains a bit of everything.

  • Which is which

    Selection keeps the original columns, so you can still say 'this is age'. Extraction makes new columns that can be better at holding the information, but they no longer have a simple name. PCA, in the next part, is the best known way to extract.

PCA: new directions that keep the spread

Principal component analysis, or PCA, is the main way to extract. Wikipedia calls it 'a linear dimensionality reduction technique' and says the data are transformed 'onto a new coordinate system such that the directions (principal components) capturing the largest variation in the data can be easily identified.' The scikit-learn guide puts it as a set of 'successive orthogonal components that explain a maximum amount of the variance'.

Here is a picture. Imagine a swarm of bees shaped like a long cigar, floating in a room. You must describe where each bee is, using one number. The best choice is not left to right or up and down. It is the long axis of the cigar, because that is where the bees differ most. Two bees at the far ends get very different numbers. That is what PCA looks for: the direction with the most spread. This is the first component.

Then it looks for the next direction with the most spread at a right angle to the first ('orthogonal' means at right angles). If the cigar is thin, the second component carries little, and you can drop it.

You have met the tools already. The spread in each direction comes from the covariance matrix. Its eigenvectors are the new directions, and its eigenvalues say how much spread each direction holds. The eigenvalues lesson shows how to find them. The singular value decomposition lesson shows a second way to reach the same place. Wikipedia notes that PCA 'is a form of feature selection by collapsing correlated features'. We will not re-teach those tools here. We will use them.

explained variance of component i=λiλ1+λ2+⋯+λp\text{explained variance of component } i = \dfrac{\lambda_i}{\lambda_1 + \lambda_2 + \dots + \lambda_p}

In words: each eigenvalue is a component's share of the spread. Divided by the sum of all of them, it is the explained variance ratio.

  1. 1

    1. Centre the data

    Subtract each column's average from its values, so every column is centred on zero. Wikipedia describes this step: first 'center the values of each variable in the dataset on 0 by subtracting the mean'.

  2. 2

    2. Build the covariance matrix

    Measure how much each column spreads and how each pair moves together. With two columns you get a 2 by 2 grid.

  3. 3

    3. Find eigenvalues and eigenvectors

    The eigenvectors are the new directions. The eigenvalues tell you how much spread each holds.

  4. 4

    4. Keep the biggest few

    Sort by eigenvalue and keep the top ones. Wikipedia says the eigenvectors 'that correspond to the largest eigenvalues' can be used 'to reconstruct a large fraction of the variance of the original data'.

  5. 5

    5. Project

    Turn each row into its position along the kept directions. These are the new, shorter columns.

Tip

Scale first. scikit-learn says PCA 'centers but does not scale the input data for each feature'. Wikipedia says that 'the results of PCA depend on the scaling of the variables', and that this can be cured 'by scaling each feature by its standard deviation'. If one column is in thousands and another in tenths, the big numbers will dominate the spread only because of their units.

Worked example in plain Python

Let us do PCA by hand on a tiny made-up table: six people, with hours of sleep and hours of screen time. We use only Python's built-in math module, and a 2 by 2 matrix is small enough to solve with the quadratic formula.

For a 2 by 2 covariance matrix with numbers a, b and d, the eigenvalues solve this equation. It is the same characteristic equation from the eigenvalues lesson, written out for our grid:

λ2−(a+d) λ+(ad−b2)=0λ=(a+d)±(a+d)2−4(ad−b2)2\lambda^2 - (a + d)\,\lambda + (ad - b^2) = 0 \qquad \lambda = \dfrac{(a + d) \pm \sqrt{(a + d)^2 - 4(ad - b^2)}}{2}

In words: add the two spreads, a and d. Subtract the square of the shared movement b from their product. Put those into the formula and it gives the two eigenvalues. The code follows the five steps above. (The direction of the first component comes from the vector (b, λ − a), which solves the equation for the larger eigenvalue.)

Python
import math

# Hours of sleep and hours of screen time for six people (made-up)
points = [(2, 1), (3, 3), (4, 3), (5, 5), (6, 5), (8, 7)]
n = len(points)

# 1. Centre each column on 0
mean_x = sum(p[0] for p in points) / n
mean_y = sum(p[1] for p in points) / n
centred = [(x - mean_x, y - mean_y) for x, y in points]
print("means:", mean_x, mean_y)

# 2. The 2x2 covariance matrix [[a, b], [b, d]]
a = sum(x * x for x, y in centred) / (n - 1)
b = sum(x * y for x, y in centred) / (n - 1)
d = sum(y * y for x, y in centred) / (n - 1)
print("a, b, d:", round(a, 3), round(b, 3), round(d, 3))

# 3. Eigenvalues: lambda^2 - (a + d) lambda + (a d - b^2) = 0
trace = a + d
det = a * d - b * b
root = math.sqrt(trace * trace - 4 * det)
big = (trace + root) / 2
small = (trace - root) / 2
print("eigenvalues:", round(big, 3), round(small, 3))

# 4. Explained variance ratio
print("ratio:", round(big / (big + small), 3), round(small / (big + small), 3))

# 5. Direction of the first component: (b, big - a) solves (A - big I) v = 0
vx, vy = b, big - a
length = math.hypot(vx, vy)
vx, vy = vx / length, vy / length
print("first direction:", round(vx, 3), round(vy, 3))

# 6. Squash each person onto that one direction: one number each
print("one-number version:", [round(x * vx + y * vy, 2) for x, y in centred])

Output

means: 4.666666666666667 4.0
a, b, d: 4.667 4.4 4.4
eigenvalues: 8.935 0.131
ratio: 0.986 0.014
first direction: 0.718 0.696
one-number version: [-4.0, -1.89, -1.17, 0.94, 1.65, 4.48]

The covariance numbers are 4.667, 4.4 and 4.4. Because the shared movement (4.4) is almost as big as each spread, the two columns rise and fall together.

The eigenvalues are 8.935 and 0.131. Their sum, 9.066, equals 4.667 + 4.4, the sum of the two spreads. That is expected: PCA only moves the spread around, it does not make or lose any. The first direction holds 8.935 ÷ 9.066 = 0.986, which is 98.6% of the total. The second holds just 1.4%.

The first direction is (0.718, 0.696), so it runs diagonally and gives almost equal weight to each column. The last line squashes each person onto it. Six pairs of numbers became six single numbers, and we lost only 1.4% of the spread. That is dimensionality reduction from 2 columns to 1.

Quick check

In the example, the eigenvalues were 8.935 and 0.131. What does the second one tell us?

How many components to keep

A common rule uses the running total of the explained variance ratio. Wikipedia describes the cumulative share and says to choose 'a value of L as small as possible while achieving a reasonably high value of g on a percentage basis.' Its example is to keep enough components so that the total passes 90 percent. If three components give 85% and four give 92%, you keep four.

Another way is to look at a picture. A scree plot lines up the eigenvalues from biggest to smallest. Wikipedia says it is 'meant to help interpret the PCA and decide how many components to retain', and that the start of the bend in the line, the 'knee', 'should indicate how many components are retained'.

Tip

There is no perfect number. Ask what you need the result for, try a few values and compare.

Other methods in one line each

  • t-SNE: a picture of your data

    scikit-learn says that 'The main purpose of t-SNE is visualization of high-dimensional data.' It is 'particularly sensitive to local structure', so neighbours stay near each other. Its limits: it is 'computationally expensive', its results can differ from run to run because 'The algorithm is stochastic', and 'Global structure is not explicitly preserved.'

  • UMAP: also pictures, and more

    The UMAP docs call it 'a dimension reduction technique that can be used for visualisation similarly to t-SNE, but also for general non-linear dimension reduction.'

  • Autoencoders: a learned squeeze

    Wikipedia describes them as 'a special kind of feedforward neural networks with a bottleneck hidden layer'. They get their own lesson later in this course.

t-SNE, UMAP and their cousins belong to manifold learning, which scikit-learn describes as 'an approach to non-linear dimensionality reduction' based on the idea 'that the dimensionality of many data sets is only artificially high.' The same page says that straight-line methods such as PCA 'often miss important non-linear structure in the data.'

When to use it, and when not to

Use it when you want a flat picture of your data, when many columns move together, or when noise hides the signal.

Be careful when the meaning of each column matters. Wikipedia warns that after PCA 'the new data variables cannot be interpreted in the same ways that the originals were. They are linear interpretations of the original variables.' A new column is a blend. 'Component 1' is not 'age'. If your boss asks which input drove the result, a blend is hard to explain. In that case, feature selection may be the better tool.

Wikipedia lists more limits for PCA. It 'relies on a linear model', so a bent pattern may fool it. It can be 'sensitive to outliers', so removing outliers first is common practice. And if it 'is not performed properly, there is a high likelihood of information loss'.

Quick check

You need to tell a client which input column affects the result most. Which tool fits better?

Fit on training data only

One rule keeps you honest. In scikit-learn, PCA 'is implemented as a transformer object that learns n components in its fit method, and can be used on new data to project it on these components.' So there are two moves. First it learns the directions from data. Then it applies them to any data.

Learn the directions from the training data only. Then use those same directions on the test data and on any new data. If you run PCA on all the data before splitting, facts about the test rows leak into the directions you chose. It is like a student who saw the exam questions while writing the study guide. The score will look better than it really is.

Text
1. Split the rows into training and test sets.
2. Learn the centring, scaling and components from training rows only.
3. Apply those same numbers to the training rows.
4. Apply those same numbers to the test rows and to new data.

What to take away

A dimension is one column. Many columns bring the curse of dimensionality: data gets thin, and the data you need grows quickly. You can fight it two ways: select some of the columns, or extract a few new ones. PCA extracts the directions with the most spread, found with the covariance matrix and its eigenvalues. The explained variance ratio tells you how much each one holds, and a running total helps you choose how many to keep.

Next: what machine learning is, and its main types.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.