esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.37

Autoencoders: learning to squeeze

A network that squeezes data through a narrow middle and rebuilds it. How it learns, and why a bad rebuild is a useful alarm.

14 min readFree

From squeezing to learning to squeeze

The last lesson, Principal Component Analysis, squeezed data into fewer numbers by a fixed recipe. It ended with a promise: autoencoders, networks that learn to compress. This is that lesson.

Think of packing a suitcase for a weekend away. You fold things small and keep only what matters. At the other end, you unpack and hope you can still dress well. If you packed badly, something important is missing.

An autoencoder does exactly that with data. It packs each example into a few numbers, then unpacks it and tries to get the original back. How close it gets tells you how well it packed. And, as you will see, it also tells you when something strange walks in.

Key words

Neural network
A model built from layers of small calculating units. Google's glossary: a model containing at least one hidden layer.
Layer
A row of units that all take in the numbers from the row before and pass new numbers on.
Encoder
The first half. It squeezes the input down into a short list of numbers.
Code
The short list in the middle. Also called the bottleneck or the latent representation.
Decoder
The second half. It rebuilds the original from the code.
Reconstruction
The rebuilt version of the input.
Reconstruction error
How far the rebuilt version is from the original. Smaller is better.
Anomaly
An example that does not fit the usual pattern.

You do not need to know neural networks yet. The lesson called Neural Network Basics comes later in this course. The next section gives you just enough to follow this one.

A tiny network in plain words

A neural network is built from one simple move, repeated. Take some numbers. Multiply each by a weight, which is a number the model learns. Add the results up. That is a weighted sum. Then pass the total through a bend, which Google's glossary calls an activation function. In its words, activation functions let networks learn nonlinear, or curvy, relationships.

One common bend is called ReLU. It leaves positive numbers alone and turns negative numbers into zero. Here is one unit doing its job, with weights 0.5, 0.2 and 0.3.

Python
weights = [0.5, 0.2, 0.3]

def neuron(inputs):
    total = sum(w * x for w, x in zip(weights, inputs))
    return round(total, 2), round(max(0, total), 2)

print("inputs [2, -1, 3]  -> weighted sum, after bend:", neuron([2, -1, 3]))
print("inputs [-2, 1, -3] -> weighted sum, after bend:", neuron([-2, 1, -3]))

Output

inputs [2, -1, 3]  -> weighted sum, after bend: (1.7, 1.7)
inputs [-2, 1, -3] -> weighted sum, after bend: (-1.7, 0)

The first inputs give 1.7, and the bend lets it through. The second give -1.7, and the bend turns it into 0. A layer is many such units side by side. A network is layers stacked, each feeding the next.

The idea: squeeze, then rebuild

Wikipedia describes an autoencoder as a type of artificial neural network used to learn efficient codings of unlabeled data. It learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation.

The TensorFlow tutorial says it more bluntly: an autoencoder is a special type of neural network that is trained to copy its input to its output. Copying sounds pointless. Here is the trick: the network is not allowed to copy straight through.

Between the encoder and the decoder sits a narrow middle. Say each example has 4 numbers, and the middle holds only 2. Whatever the decoder rebuilds from, it can only use those 2 numbers. So the encoder has to keep what matters and throw away the rest.

  1. 1

    1. In

    A row of numbers goes in, for example 4 numbers describing one example.

  2. 2

    2. Encode

    The encoder turns the 4 numbers into a short code, say 2 numbers.

  3. 3

    3. Decode

    The decoder turns the 2-number code back into 4 numbers.

  4. 4

    4. Compare

    Check the 4 numbers that came out against the 4 that went in. The gap is the reconstruction error.

Wikipedia calls an autoencoder whose code is smaller than the input undercomplete, and notes it can be interpreted as compressing the message, or reducing its dimensionality. Google's glossary adds a useful point: because the middle is smaller, the autoencoder is forced to learn what information in the input is essential, and the output won't be perfectly identical to the input.

The shape of it

Drawn as a picture, an autoencoder looks like an hourglass. Wide at the ends, narrow in the middle.

Text
input (4)  ->  encoder  ->  code (2)  ->  decoder  ->  output (4)
  wide                      narrow                      wide
  "the original"            "the suitcase"              "the rebuild"

The input is its own answer, so nobody has to attach a label to any example. That is why autoencoders count as learning from unlabeled data.

The score: reconstruction error

A model needs a score to improve. For an autoencoder, the score is how badly it rebuilt the input. The TensorFlow tutorial puts the goal in one line: an autoencoder learns to compress the data while minimizing the reconstruction error.

The usual way to measure the gap is to take each number's difference from the original, square it, and average. Squaring does two jobs: it makes every difference positive, and it punishes big misses more than small ones. It is the same idea as the error in the Linear regression lesson.

error=14∑i=14(rebuilti−originali)2\text{error} = \frac{1}{4}\sum_{i=1}^{4}\left(\text{rebuilt}_i - \text{original}_i\right)^2

In words: for each of the 4 numbers, take the rebuilt value minus the original, square it, then average the four results. Zero means a perfect rebuild.

How it learns

At the start the weights are small random numbers, so the rebuild is rubbish. Training nudges the weights, again and again, so the error shrinks. Wikipedia says this search is accomplished by any mathematical optimization technique, but usually by gradient descent, and that the process is referred to as training the autoencoder.

If gradient descent is hazy, the Linear regression lesson covers it, and the tuning lesson shows what a learning rate does. The short version is a walk downhill in fog. At each spot you feel which way the ground slopes, then take a step that way. Here the "ground" is the reconstruction error, and the steps change the weights.

  1. 1

    1. Start random

    Give every weight a small random number.

  2. 2

    2. Rebuild

    Push every example through the encoder and decoder.

  3. 3

    3. Measure

    Work out the reconstruction error.

  4. 4

    4. Find the slope

    For each weight, work out which way a small change would make the error grow.

  5. 5

    5. Step

    Move each weight a small step the other way. The size of the step is the learning rate.

  6. 6

    6. Repeat

    Go back to step 2 until the error stops improving.

Tip

Nobody tells the network what the code should mean. It invents its own short summary, because that is the only way to get a good rebuild through a narrow middle.

How it relates to PCA

PCA and an autoencoder have the same goal: keep few numbers, lose little. Wikipedia says how they link. If linear activations are used, or only a single sigmoid hidden layer, then the optimal solution to an autoencoder is strongly related to principal component analysis. Linear here means no bend: only weighted sums.

It goes further. The weights of an autoencoder with a single hidden layer of size p span the same space as the first p principal components. But the autoencoder weights are not equal to the principal components, and are generally not orthogonal. Orthogonal means at right angles to one another.

So a simple autoencoder does PCA's job by learning instead of by a formula. The extra power comes when you add the bend. Wikipedia: the potential of autoencoders resides in their non-linearity, allowing the model to learn more powerful generalizations compared to PCA. It also reports one experiment in which a 30-number code gave a smaller reconstruction error than the first 30 components of a PCA.

Build one in plain Python

Time to build a real one, using only Python's built-in tools. We make 200 rows of made-up data with 4 numbers each. But the 4 numbers are secretly driven by just 2 hidden things, a and b, plus a little noise. The second number is about twice the first. The fourth is about the first minus the third. That pattern is what the network has to find.

The network is 4 to 2 to 4, with no bend, so it is the simple linear kind from the last section. The code prints the average error every 200 steps.

Python
import random

random.seed(3)

# 200 made-up rows: 4 numbers each, but really driven by just 2 hidden things.
rows = []
for _ in range(200):
    a = random.gauss(0, 1)
    b = random.gauss(0, 1)
    rows.append([a + 0.1 * random.gauss(0, 1),
                 2 * a + 0.1 * random.gauss(0, 1),
                 b + 0.1 * random.gauss(0, 1),
                 -b + a + 0.1 * random.gauss(0, 1)])

# Encoder weights (2 x 4) and decoder weights (4 x 2), small random start.
enc = [[random.gauss(0, 0.3) for _ in range(4)] for _ in range(2)]
dec = [[random.gauss(0, 0.3) for _ in range(2)] for _ in range(4)]

def encode(x):
    return [sum(enc[k][j] * x[j] for j in range(4)) for k in range(2)]

def decode(z):
    return [sum(dec[i][k] * z[k] for k in range(2)) for i in range(4)]

def error(x):
    r = decode(encode(x))
    return sum((r[i] - x[i]) ** 2 for i in range(4)) / 4

def average_error(data):
    return sum(error(x) for x in data) / len(data)

rate = 0.05
n = len(rows)
for step in range(1201):
    if step % 200 == 0:
        print("step", step, "error", round(average_error(rows), 4))
    g_enc = [[0.0] * 4 for _ in range(2)]
    g_dec = [[0.0] * 2 for _ in range(4)]
    for x in rows:
        z = encode(x)
        r = decode(z)
        e = [2 * (r[i] - x[i]) / 4 for i in range(4)]   # slope of the error
        for i in range(4):
            for k in range(2):
                g_dec[i][k] += e[i] * z[k] / n
        dz = [sum(dec[i][k] * e[i] for i in range(4)) for k in range(2)]
        for k in range(2):
            for j in range(4):
                g_enc[k][j] += dz[k] * x[j] / n
    for i in range(4):
        for k in range(2):
            dec[i][k] -= rate * g_dec[i][k]
    for k in range(2):
        for j in range(4):
            enc[k][j] -= rate * g_enc[k][j]

x = rows[0]
print("row     ", [round(v, 2) for v in x])
print("code    ", [round(v, 2) for v in encode(x)])
print("rebuilt ", [round(v, 2) for v in decode(encode(x))])

odd = [1.0, -2.0, 1.0, 3.0]   # breaks the pattern: 2nd number should be about twice the 1st
print("typical row error:", round(average_error(rows), 4))
print("odd row error    :", round(error(odd), 4))

Output

step 0 error 3.2273
step 200 error 0.0057
step 400 error 0.0056
step 600 error 0.0055
step 800 error 0.0054
step 1000 error 0.0054
step 1200 error 0.0053
row      [0.0, 0.29, 1.22, -1.18]
code     [-0.77, 1.5]
rebuilt  [0.1, 0.2, 1.27, -1.15]
typical row error: 0.0053
odd row error    : 3.3721

Reading the result

The error starts at 3.2273, the rebuild of a network that knows nothing. By step 200 it is down to 0.0057, and by step 1200 it is 0.0053. Most of the learning happens early, then it settles. The error never reaches exactly zero, because the noise we added is not something 2 numbers can capture.

Look at the first row. The original was [0.0, 0.29, 1.22, -1.18]. The encoder squeezed it to just two numbers, [-0.77, 1.5]. The decoder rebuilt [0.1, 0.2, 1.27, -1.15]. Not exact, but close. Four numbers were rebuilt from two, because the data really had only two things going on.

Tip

We made the data from two hidden things on purpose, so a 2-number code is the right size. Choosing the size of the middle is a setting you tune.

Spotting the odd one out

The last two lines of the output are the best part. The odd row was [1.0, -2.0, 1.0, 3.0]. It breaks the pattern: the second number should be about twice the first, and here it is minus two times it. The network never saw anything like it while learning.

Its rebuild error is 3.3721. A typical row's error is 0.0053. The odd row is rebuilt more than 600 times worse. The network learned the pattern of normal data, so when something breaks the pattern, it cannot pack and unpack it well.

That is anomaly detection. Wikipedia defines it as the identification of rare items, events or observations which deviate significantly from the majority of the data.

Quick check

A trained autoencoder rebuilds one new example with a very large error. What does that most likely mean?

What autoencoders are used for

Wikipedia says autoencoders are applied to many problems, including facial recognition, feature detection, anomaly detection, and learning the meaning of words. Three uses are easiest to picture.

  • Compression

    Keep the short code instead of the full data. Wikipedia says the learned representation is typically used for dimensionality reduction, to generate lower-dimensional embeddings for subsequent use by other machine learning algorithms. Google notes the rebuild is lossy, meaning some detail is lost.

  • Denoising

    The TensorFlow tutorial shows an autoencoder trained to remove noise from images: noisy images go in, and the clean originals are the target. Wikipedia describes the idea: a denoising autoencoder is trained by intentionally corrupting the inputs. The network learns to recover the original message without the noise.

  • Anomaly detection

    Train on normal examples, then flag anything that rebuilds badly. Wikipedia: after training, the autoencoder will accurately reconstruct normal data, while failing to do so with unfamiliar anomalous data.

How to decide what counts as odd

A big error is clear. A medium one is a judgement call, so you need a cut-off. Wikipedia describes the usual recipe: on a validation set, record how the reconstruction errors are spread, then take for example the 95th percentile as the threshold. An example whose error is above it gets flagged.

  1. 1

    1. Train on normal data

    Mostly or only normal examples. Wikipedia notes that in most cases only normal instances are used to train the autoencoder.

  2. 2

    2. Measure on held-out data

    Use data kept back from training, as in the train and test lesson. Record the reconstruction errors.

  3. 3

    3. Pick a threshold

    For example, the error that 95 out of 100 normal examples stay below.

  4. 4

    4. Flag

    Anything above the threshold is a suspect to look at by hand.

Wikipedia warns that setting the threshold "correctly" is genuinely hard, because it is only an estimate from data. Treat a flag as a reason to look, not as proof.

Quick check

Why train an anomaly-spotting autoencoder mostly on normal examples?

One more name: variational autoencoders

You will hear the term variational autoencoder, or VAE. Wikipedia describes it as an artificial neural network architecture introduced by Diederik P. Kingma and Max Welling in 2013. Where the basic autoencoder is mostly about squeezing and rebuilding, Wikipedia says variational autoencoders can be used as generative models, which means models that make new data.

Things to watch for

  • Wrong middle size

    Too narrow and everything rebuilds badly. Wikipedia warns that if the code is as big as the input or bigger, an autoencoder can learn the identity function and become useless: it just copies.

  • Not a perfect copy

    By design, the rebuild is lossy. If you need every detail back exactly, use a normal file compressor instead.

What to remember

  • The idea

    An encoder squeezes the input into a short code. A decoder rebuilds it. The network is trained to make the rebuild match the input.

  • The score

    Reconstruction error: how far the rebuild is from the original. Gradient descent shrinks it by nudging the weights.

  • The uses

    Compression, denoising and anomaly detection. A rebuild that fails badly is the sign of something unusual.

Next: how to tell whether a model is any good.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.