esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.22

Deep Learning Architectures

A map of the main shapes networks come in: dense, convolutional, recurrent, transformer, autoencoder and GAN, and which data suits which.

13 min readFree

From learning to the shapes of networks

The last lesson, Back Propagation, showed how a network learns: it checks its mistake and nudges every weight so the mistake gets smaller. It ended with a promise: deep learning architectures, the main shapes networks come in. This is that lesson.

Think of buildings. A house, a school and a hospital are all made of bricks, but nobody builds them the same way. A school needs many similar rooms in a row. A hospital needs wide corridors and lifts. The bricks are the same. The plan is what changes. Networks work like that. The bricks are the small calculating units you already know. The plan is the architecture.

This lesson is a map, not a deep dive. Each shape gets a short, plain introduction, and most of them get their own lesson later. By the end you should be able to look at a problem, such as photos, sentences or music, and say which shape is the usual first pick.

Key words

Architecture
The shape and wiring of a network: how many layers, how big each one is, and which units connect to which.
Layer
A row of units that take in numbers from the row before and pass new numbers on.
Deep learning
Learning with a network of many layers. Wikipedia says "deep" refers to the use of multiple layers, from three to several hundred or thousands.
Weight
A number the network learns. Each connection between two units has one.
Dense layer
A layer where every unit connects to every unit in the layer before. Also called fully connected.
Filter
A small grid of weights that slides over the data looking for one pattern. Also called a kernel.
Sequence
Data where the order matters, such as the words in a sentence or the notes in a tune.

What an architecture is

A network is made of units and the connections between them. The wiring is the architecture.

Wikipedia lists some common deep learning architectures: fully connected networks, deep belief networks, recurrent neural networks, convolutional neural networks, generative adversarial networks, transformers and neural radiance fields. You do not need all of them. Six are enough to start.

One reason architectures matter is that deep learning lets the network find its own useful features. Wikipedia puts it this way: before deep learning, machine learning often involved hand-crafted feature engineering. In the deep learning approach, the model discovers useful feature representations from the data automatically. The architecture decides what kind of features it is able to discover.

Feedforward networks: the plain one

You have met this one already. In a feedforward network, numbers go in at one end, pass through layer after layer, and come out at the other end. Nothing loops back.

Wikipedia describes a multilayer perceptron (MLP) as a kind of modern feedforward neural network consisting of fully connected neurons with nonlinear activation functions, organized in layers. It adds that MLPs are often called dense networks, and colloquially "vanilla" networks, to tell them apart from more specialized architectures. Vanilla means plain, like the flavour of ice cream.

Fully connected means each unit in one layer connects to every unit in the next. That is the strength and the weakness. It is flexible: it assumes nothing about your data, so it can work on tables of numbers like house prices or customer records. But it is wasteful when the data has a built-in shape, like a picture. The next section shows how wasteful.

Convolutional networks: for images

A photo is a grid of pixels. Neighbouring pixels usually belong together: a few dark pixels in a row might be an edge. A tiny patch of the picture tells you a lot, no matter where on the picture the patch sits. A cat's ear is a cat's ear in the top left or the bottom right.

A convolutional neural network, or CNN, is built around that idea. Wikipedia says it is a type of feedforward neural network that learns features via filter (or kernel) optimization. It describes how it works: as the convolution kernel slides along the input matrix, the operation generates a feature map, which feeds the next layer. In plain words, a small filter of weights slides across the picture, one patch at a time, and records how strongly each patch matches its pattern.

The big win is a smaller number of weights. Wikipedia gives an example: for each neuron in a fully connected layer, 10,000 weights would be required to process an image of 100 × 100 pixels. With convolution kernels, only 25 weights are required for 5 × 5 tiles. The reason is that every spot the filter visits uses the same weights. Wikipedia calls this shared weights.

Count the weights

Take a small picture, 28 × 28 pixels, which is 784 numbers. We count weights only, and leave out the extra "bias" number each unit also has. A 3 × 3 filter can sit in 26 × 26 = 676 spots when it never goes off the edge.

Python
pixels = 28 * 28
print("pixels in one 28x28 picture:", pixels)

# A dense layer with 128 units: every unit looks at every pixel.
units = 128
dense = pixels * units
print("dense layer, 128 units, weights:", dense)

# A dense layer that makes one output per pixel (784 outputs).
dense_same = pixels * pixels
print("dense layer, 784 units, weights:", dense_same)

# One 3x3 filter: 9 weights, reused at every spot.
conv = 3 * 3
print("one 3x3 filter, weights:", conv)

spots = (28 - 3 + 1) ** 2
print("spots the filter visits:", spots)
print("dense (784 units) needs", dense_same // conv, "times more weights than one filter")

Output

pixels in one 28x28 picture: 784
dense layer, 128 units, weights: 100352
dense layer, 784 units, weights: 614656
one 3x3 filter, weights: 9
spots the filter visits: 676
dense (784 units) needs 68295 times more weights than one filter

A dense layer with 128 units needs 100,352 weights. One with a unit for every pixel needs 614,656. A single 3 × 3 filter needs 9, and it still covers all 676 spots. A real convolutional layer uses many filters, each looking for its own pattern, but even 100 filters would need only 900 weights. Wikipedia makes the same point: convolution reduces the number of free parameters, allowing the network to be deeper.

What sliding looks like

Images are 2D, which is hard to print. A line of numbers shows the same move in one dimension. We slide a filter of 3 numbers along a list of 7. At each stop we multiply the filter's weights by the three numbers under it, and add the results up. The filter [1, 0, -1] means "left number minus right number", so it reacts where the signal rises or falls.

Python
signal = [1, 2, 3, 4, 3, 2, 1]
filt = [1, 0, -1]   # a 3-number filter: left minus right

for start in range(len(signal) - len(filt) + 1):
    window = signal[start:start + len(filt)]
    total = sum(w * f for w, f in zip(window, filt))
    print("window", window, "->", total)

Output

window [1, 2, 3] -> -2
window [2, 3, 4] -> -2
window [3, 4, 3] -> 0
window [4, 3, 2] -> 2
window [3, 2, 1] -> 2

Read the output as a story of the signal. While the numbers climb (1, 2, 3, 4) the filter gives -2. At the top, where it stops climbing, it gives 0. While the numbers fall, it gives 2. One small filter, five stops, and the pattern "going up" or "going down" shows up. In a trained network the filter's weights are not chosen by hand. They are learned, so the network finds the patterns that help it.

Recurrent networks: for sequences

Now a different kind of data. Read this: "the dog bit the man". Now this: "the man bit the dog". Same words, different story. When order matters, a network that looks at every input on its own is missing something.

Wikipedia says recurrent neural networks (RNNs) are designed for processing sequential data, such as text, speech and time series, where the order of elements is important. Unlike feedforward networks, which process inputs independently, RNNs use recurrent connections: the output of a neuron at one time step is fed back as input at the next. This lets them capture patterns across time.

Wikipedia names the building block the recurrent unit, which maintains a hidden state, a form of memory that is updated at each time step from the current input and the previous hidden state. It is like reading with a notepad: at each word you update your notes.

Here is the smallest possible version. The memory is one number. At each step the old memory fades to half, and the new input is added.

Python
def run(sequence):
    memory = 0.0
    for x in sequence:
        memory = 0.5 * memory + x   # old memory fades by half, new input is added
    return memory

print("[1, 0, 0] ->", run([1, 0, 0]))
print("[0, 0, 1] ->", run([0, 0, 1]))

Output

[1, 0, 0] -> 0.25
[0, 0, 1] -> 1.0

Plain RNNs have a known weakness. Wikipedia says traditional RNNs suffer from the vanishing gradient problem, which limits their ability to learn long-range dependencies. In plain words: the learning signal fades as it travels back over many steps, so the network struggles to connect things far apart. The long short-term memory (LSTM) architecture, developed in 1997, was made to fix this. Gated recurrent units (GRUs), introduced in 2014, came later. Wikipedia says a GRU has fewer parameters than an LSTM, and found their performance similar on some tasks, such as speech signal modelling.

Transformers: attention to everything at once

An RNN reads one step at a time, like a person reading with a finger under each word. A transformer takes a different route. Wikipedia describes it as a family of neural network architectures based on the multi-head attention mechanism. The input, such as text, images or audio, is turned into a sequence of numerical pieces called tokens. Then, at each layer, every token is looked at in the context of the other tokens, so the signal for key tokens is amplified and less important tokens are diminished.

Attention means each piece of the input decides how much to look at each other piece. In "The cat sat on the mat because it was tired", to understand "it" you look back at "cat". A transformer can do that directly.

Because transformers do not process tokens one at a time, Wikipedia says their computations can be parallelized more readily than those of RNNs like the LSTM. Parallel means many calculations at once. It also notes that attention alone ignores order, so transformers add positional information, such as positional encodings, so the order of tokens can affect the output.

The original transformer came from the 2017 paper "Attention Is All You Need" by researchers at Google. Wikipedia says it has since been used in language, computer vision, audio and robotics, and led to systems such as generative pre-trained transformers (GPTs). Transformers get their own lesson later.

Autoencoders: squeeze and rebuild

You met these in the Autoencoders lesson, so this is a quick recap. An autoencoder is a network that squeezes its input into a short code, then tries to rebuild the original from that code. Wikipedia describes it as a type of artificial neural network used to learn efficient codings of unlabeled data.

Its shape is an hourglass: wide, narrow in the middle, wide again. The narrow middle forces it to keep only what matters.

GANs: two networks competing

Imagine a forger who paints fake banknotes and a detective who tries to spot them. At first the fakes are poor and the detective wins every time. The forger studies what gave the fakes away, and gets better. The detective has to get sharper in turn. Round after round, the fakes become hard to tell from the real thing.

That is a generative adversarial network, or GAN. Wikipedia says it was initially developed by Ian Goodfellow and colleagues at Université de Montréal in June 2014. In a GAN, two neural networks compete with each other in the form of a zero-sum game, where one agent's gain is another agent's loss. One network, the generator, makes new data. The other, the discriminator, judges how realistic the input seems.

A key point from Wikipedia: the generator is not trained to match a specific image, but to fool the discriminator. Given a training set, the technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers. Wikipedia compares it to mimicry in evolutionary biology, an arms race between the two networks. GANs get their own lesson later.

Matching data to architecture

Start with the kind of data you have, then pick the shape built for it. This is a starting point, not a law. Wikipedia notes that CNNs have only recently been replaced in some cases by newer architectures such as the transformer, and that RNNs remain relevant where computational efficiency or real-time processing is crucial.

  • Tables and plain numbers: dense (MLP)

    Rows of numbers with no built-in shape, such as prices, ages or scores. A dense network assumes nothing about the layout, so it fits.

  • Pictures and grids: convolutional

    Photos, scans and other grids where nearby pixels belong together. Sliding filters with shared weights spot patterns anywhere in the image. Wikipedia lists image and video recognition, classification and medical image analysis among its uses.

  • Order and language: recurrent or transformer

    Text, speech and time series. RNNs, LSTMs and GRUs read step by step with a memory. Transformers look at all the pieces and decide what to attend to. Wikipedia calls transformers the dominant architecture for many sequence tasks, especially in language.

Why the shape is a bet

Each shape in this lesson carries an assumption about your data. A convolutional network assumes that a pattern means the same thing wherever it appears. A recurrent network assumes that order matters and that the past helps explain the present. A dense network assumes very little. Machine learning has a name for this kind of assumption: inductive bias.

Wikipedia defines the inductive bias of a learning algorithm as the set of assumptions that the learner uses to predict outputs of given inputs that it has not encountered. It explains why this is needed: without any additional assumptions, the problem cannot be solved, since unseen situations might have an arbitrary output value.

So choosing an architecture is a bet about your data. A good bet helps the network learn. A bad bet works against you.

Check yourself

Quick check

You want a network to read a customer's review and judge the mood. Which shape fits best?

Quick check

Why does a 3 × 3 filter need so few weights compared with a dense layer?

Recap

  1. 1

    1. Wiring

    Architecture is the shape of a network. Same bricks, different plan.

  2. 2

    2. Pick by data

    Dense for tables, convolutional for grids and images, recurrent or transformer for sequences.

  3. 3

    3. Goals

    Autoencoders squeeze and rebuild. GANs make two networks compete to generate data.

  4. 4

    4. A bet

    Each shape assumes something about the data. Pick the one whose bet fits.

Next: deep learning libraries, the tools that build networks for you.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.