Machine Learning Foundations / Core / 02.37
Autoencoders: learning to squeeze
A network that squeezes data through a narrow middle and rebuilds it. How it learns, and why a bad rebuild is a useful alarm.
14 min readFree
From squeezing to learning to squeeze
The last lesson, Principal Component Analysis, squeezed data into fewer numbers by a fixed recipe. It ended with a promise: autoencoders, networks that learn to compress. This is that lesson.
Think of packing a suitcase for a weekend away. You fold things small and keep only what matters. At the other end, you unpack and hope you can still dress well. If you packed badly, something important is missing.
An autoencoder does exactly that with data. It packs each example into a few numbers, then unpacks it and tries to get the original back. How close it gets tells you how well it packed. And, as you will see, it also tells you when something strange walks in.
Key words
- Neural network
- A model built from layers of small calculating units. Google's glossary: a model containing at least one hidden layer.
- Layer
- A row of units that all take in the numbers from the row before and pass new numbers on.
- Encoder
- The first half. It squeezes the input down into a short list of numbers.
- Code
- The short list in the middle. Also called the bottleneck or the latent representation.
- Decoder
- The second half. It rebuilds the original from the code.
- Reconstruction
- The rebuilt version of the input.
- Reconstruction error
- How far the rebuilt version is from the original. Smaller is better.
- Anomaly
- An example that does not fit the usual pattern.
You do not need to know neural networks yet. The lesson called Neural Network Basics comes later in this course. The next section gives you just enough to follow this one.
A tiny network in plain words
A neural network is built from one simple move, repeated. Take some numbers. Multiply each by a weight, which is a number the model learns. Add the results up. That is a weighted sum. Then pass the total through a bend, which Google's glossary calls an activation function. In its words, activation functions let networks learn nonlinear, or curvy, relationships.
One common bend is called ReLU. It leaves positive numbers alone and turns negative numbers into zero. Here is one unit doing its job, with weights 0.5, 0.2 and 0.3.
weights = [0.5, 0.2, 0.3]
def neuron(inputs):
total = sum(w * x for w, x in zip(weights, inputs))
return round(total, 2), round(max(0, total), 2)
print("inputs [2, -1, 3] -> weighted sum, after bend:", neuron([2, -1, 3]))
print("inputs [-2, 1, -3] -> weighted sum, after bend:", neuron([-2, 1, -3]))Output
inputs [2, -1, 3] -> weighted sum, after bend: (1.7, 1.7) inputs [-2, 1, -3] -> weighted sum, after bend: (-1.7, 0)
The first inputs give 1.7, and the bend lets it through. The second give -1.7, and the bend turns it into 0. A layer is many such units side by side. A network is layers stacked, each feeding the next.
The idea: squeeze, then rebuild
Wikipedia describes an autoencoder as a type of artificial neural network used to learn efficient codings of unlabeled data. It learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation.
The TensorFlow tutorial says it more bluntly: an autoencoder is a special type of neural network that is trained to copy its input to its output. Copying sounds pointless. Here is the trick: the network is not allowed to copy straight through.
Between the encoder and the decoder sits a narrow middle. Say each example has 4 numbers, and the middle holds only 2. Whatever the decoder rebuilds from, it can only use those 2 numbers. So the encoder has to keep what matters and throw away the rest.
- 1
1. In
A row of numbers goes in, for example 4 numbers describing one example.
- 2
2. Encode
The encoder turns the 4 numbers into a short code, say 2 numbers.
- 3
3. Decode
The decoder turns the 2-number code back into 4 numbers.
- 4
4. Compare
Check the 4 numbers that came out against the 4 that went in. The gap is the reconstruction error.
Wikipedia calls an autoencoder whose code is smaller than the input undercomplete, and notes it can be interpreted as compressing the message, or reducing its dimensionality. Google's glossary adds a useful point: because the middle is smaller, the autoencoder is forced to learn what information in the input is essential, and the output won't be perfectly identical to the input.
The shape of it
Drawn as a picture, an autoencoder looks like an hourglass. Wide at the ends, narrow in the middle.
input (4) -> encoder -> code (2) -> decoder -> output (4)
wide narrow wide
"the original" "the suitcase" "the rebuild"The input is its own answer, so nobody has to attach a label to any example. That is why autoencoders count as learning from unlabeled data.
The score: reconstruction error
A model needs a score to improve. For an autoencoder, the score is how badly it rebuilt the input. The TensorFlow tutorial puts the goal in one line: an autoencoder learns to compress the data while minimizing the reconstruction error.
The usual way to measure the gap is to take each number's difference from the original, square it, and average. Squaring does two jobs: it makes every difference positive, and it punishes big misses more than small ones. It is the same idea as the error in the Linear regression lesson.
In words: for each of the 4 numbers, take the rebuilt value minus the original, square it, then average the four results. Zero means a perfect rebuild.
How it learns
At the start the weights are small random numbers, so the rebuild is rubbish. Training nudges the weights, again and again, so the error shrinks. Wikipedia says this search is accomplished by any mathematical optimization technique, but usually by gradient descent, and that the process is referred to as training the autoencoder.
If gradient descent is hazy, the Linear regression lesson covers it, and the tuning lesson shows what a learning rate does. The short version is a walk downhill in fog. At each spot you feel which way the ground slopes, then take a step that way. Here the "ground" is the reconstruction error, and the steps change the weights.
- 1
1. Start random
Give every weight a small random number.
- 2
2. Rebuild
Push every example through the encoder and decoder.
- 3
3. Measure
Work out the reconstruction error.
- 4
4. Find the slope
For each weight, work out which way a small change would make the error grow.
- 5
5. Step
Move each weight a small step the other way. The size of the step is the learning rate.
- 6
6. Repeat
Go back to step 2 until the error stops improving.
Tip
Nobody tells the network what the code should mean. It invents its own short summary, because that is the only way to get a good rebuild through a narrow middle.
How it relates to PCA
PCA and an autoencoder have the same goal: keep few numbers, lose little. Wikipedia says how they link. If linear activations are used, or only a single sigmoid hidden layer, then the optimal solution to an autoencoder is strongly related to principal component analysis. Linear here means no bend: only weighted sums.
It goes further. The weights of an autoencoder with a single hidden layer of size p span the same space as the first p principal components. But the autoencoder weights are not equal to the principal components, and are generally not orthogonal. Orthogonal means at right angles to one another.
So a simple autoencoder does PCA's job by learning instead of by a formula. The extra power comes when you add the bend. Wikipedia: the potential of autoencoders resides in their non-linearity, allowing the model to learn more powerful generalizations compared to PCA. It also reports one experiment in which a 30-number code gave a smaller reconstruction error than the first 30 components of a PCA.
Build one in plain Python
Time to build a real one, using only Python's built-in tools. We make 200 rows of made-up data with 4 numbers each. But the 4 numbers are secretly driven by just 2 hidden things, a and b, plus a little noise. The second number is about twice the first. The fourth is about the first minus the third. That pattern is what the network has to find.
The network is 4 to 2 to 4, with no bend, so it is the simple linear kind from the last section. The code prints the average error every 200 steps.
import random
random.seed(3)
# 200 made-up rows: 4 numbers each, but really driven by just 2 hidden things.
rows = []
for _ in range(200):
a = random.gauss(0, 1)
b = random.gauss(0, 1)
rows.append([a + 0.1 * random.gauss(0, 1),
2 * a + 0.1 * random.gauss(0, 1),
b + 0.1 * random.gauss(0, 1),
-b + a + 0.1 * random.gauss(0, 1)])
# Encoder weights (2 x 4) and decoder weights (4 x 2), small random start.
enc = [[random.gauss(0, 0.3) for _ in range(4)] for _ in range(2)]
dec = [[random.gauss(0, 0.3) for _ in range(2)] for _ in range(4)]
def encode(x):
return [sum(enc[k][j] * x[j] for j in range(4)) for k in range(2)]
def decode(z):
return [sum(dec[i][k] * z[k] for k in range(2)) for i in range(4)]
def error(x):
r = decode(encode(x))
return sum((r[i] - x[i]) ** 2 for i in range(4)) / 4
def average_error(data):
return sum(error(x) for x in data) / len(data)
rate = 0.05
n = len(rows)
for step in range(1201):
if step % 200 == 0:
print("step", step, "error", round(average_error(rows), 4))
g_enc = [[0.0] * 4 for _ in range(2)]
g_dec = [[0.0] * 2 for _ in range(4)]
for x in rows:
z = encode(x)
r = decode(z)
e = [2 * (r[i] - x[i]) / 4 for i in range(4)] # slope of the error
for i in range(4):
for k in range(2):
g_dec[i][k] += e[i] * z[k] / n
dz = [sum(dec[i][k] * e[i] for i in range(4)) for k in range(2)]
for k in range(2):
for j in range(4):
g_enc[k][j] += dz[k] * x[j] / n
for i in range(4):
for k in range(2):
dec[i][k] -= rate * g_dec[i][k]
for k in range(2):
for j in range(4):
enc[k][j] -= rate * g_enc[k][j]
x = rows[0]
print("row ", [round(v, 2) for v in x])
print("code ", [round(v, 2) for v in encode(x)])
print("rebuilt ", [round(v, 2) for v in decode(encode(x))])
odd = [1.0, -2.0, 1.0, 3.0] # breaks the pattern: 2nd number should be about twice the 1st
print("typical row error:", round(average_error(rows), 4))
print("odd row error :", round(error(odd), 4))Output
step 0 error 3.2273 step 200 error 0.0057 step 400 error 0.0056 step 600 error 0.0055 step 800 error 0.0054 step 1000 error 0.0054 step 1200 error 0.0053 row [0.0, 0.29, 1.22, -1.18] code [-0.77, 1.5] rebuilt [0.1, 0.2, 1.27, -1.15] typical row error: 0.0053 odd row error : 3.3721
Reading the result
The error starts at 3.2273, the rebuild of a network that knows nothing. By step 200 it is down to 0.0057, and by step 1200 it is 0.0053. Most of the learning happens early, then it settles. The error never reaches exactly zero, because the noise we added is not something 2 numbers can capture.
Look at the first row. The original was [0.0, 0.29, 1.22, -1.18]. The encoder squeezed it to just two numbers, [-0.77, 1.5]. The decoder rebuilt [0.1, 0.2, 1.27, -1.15]. Not exact, but close. Four numbers were rebuilt from two, because the data really had only two things going on.
Tip
We made the data from two hidden things on purpose, so a 2-number code is the right size. Choosing the size of the middle is a setting you tune.
Spotting the odd one out
The last two lines of the output are the best part. The odd row was [1.0, -2.0, 1.0, 3.0]. It breaks the pattern: the second number should be about twice the first, and here it is minus two times it. The network never saw anything like it while learning.
Its rebuild error is 3.3721. A typical row's error is 0.0053. The odd row is rebuilt more than 600 times worse. The network learned the pattern of normal data, so when something breaks the pattern, it cannot pack and unpack it well.
That is anomaly detection. Wikipedia defines it as the identification of rare items, events or observations which deviate significantly from the majority of the data.
Quick check
A trained autoencoder rebuilds one new example with a very large error. What does that most likely mean?
What autoencoders are used for
Wikipedia says autoencoders are applied to many problems, including facial recognition, feature detection, anomaly detection, and learning the meaning of words. Three uses are easiest to picture.
Compression
Keep the short code instead of the full data. Wikipedia says the learned representation is typically used for dimensionality reduction, to generate lower-dimensional embeddings for subsequent use by other machine learning algorithms. Google notes the rebuild is lossy, meaning some detail is lost.
Denoising
The TensorFlow tutorial shows an autoencoder trained to remove noise from images: noisy images go in, and the clean originals are the target. Wikipedia describes the idea: a denoising autoencoder is trained by intentionally corrupting the inputs. The network learns to recover the original message without the noise.
Anomaly detection
Train on normal examples, then flag anything that rebuilds badly. Wikipedia: after training, the autoencoder will accurately reconstruct normal data, while failing to do so with unfamiliar anomalous data.
How to decide what counts as odd
A big error is clear. A medium one is a judgement call, so you need a cut-off. Wikipedia describes the usual recipe: on a validation set, record how the reconstruction errors are spread, then take for example the 95th percentile as the threshold. An example whose error is above it gets flagged.
- 1
1. Train on normal data
Mostly or only normal examples. Wikipedia notes that in most cases only normal instances are used to train the autoencoder.
- 2
2. Measure on held-out data
Use data kept back from training, as in the train and test lesson. Record the reconstruction errors.
- 3
3. Pick a threshold
For example, the error that 95 out of 100 normal examples stay below.
- 4
4. Flag
Anything above the threshold is a suspect to look at by hand.
Wikipedia warns that setting the threshold "correctly" is genuinely hard, because it is only an estimate from data. Treat a flag as a reason to look, not as proof.
Quick check
Why train an anomaly-spotting autoencoder mostly on normal examples?
One more name: variational autoencoders
You will hear the term variational autoencoder, or VAE. Wikipedia describes it as an artificial neural network architecture introduced by Diederik P. Kingma and Max Welling in 2013. Where the basic autoencoder is mostly about squeezing and rebuilding, Wikipedia says variational autoencoders can be used as generative models, which means models that make new data.
Things to watch for
Wrong middle size
Too narrow and everything rebuilds badly. Wikipedia warns that if the code is as big as the input or bigger, an autoencoder can learn the identity function and become useless: it just copies.
Not a perfect copy
By design, the rebuild is lossy. If you need every detail back exactly, use a normal file compressor instead.
What to remember
The idea
An encoder squeezes the input into a short code. A decoder rebuilds it. The network is trained to make the rebuild match the input.
The score
Reconstruction error: how far the rebuild is from the original. Gradient descent shrinks it by nudging the weights.
The uses
Compression, denoising and anomaly detection. A rebuild that fails badly is the sign of something unusual.
Next: how to tell whether a model is any good.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.