esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.42

Transformers

The model built on attention: how one block is put together, how it knows word order, and why it took over from recurrent networks.

14 min readFree

The architecture built on attention

The last lesson, Multi-head Attention, ended with a promise: next, transformers, the architecture built on attention. Here it is.

You already know the key part. Attention lets every word in a sentence look at every other word and decide which ones matter. A transformer stacks it into a repeating block and builds a whole model out of copies of that block.

By the end you will know what is inside a block, how the model learns word order, the three families, and why this design replaced recurrent networks for many jobs.

Key words

Transformer
A neural network built from attention and small feed-forward layers, with no recurrence. Introduced in 2017.
Encoder
The part that reads the input and turns every token into a richer list of numbers.
Decoder
The part that writes the output, one token at a time, while looking at what it has written so far.
Residual connection
A shortcut that adds a layer's input back onto its output, so the layer only has to learn the change.
Layer normalisation
A tidy-up step that rescales each token's list of numbers so they stay in a steady range.
Positional encoding
Extra numbers added to each token that say where it sits in the sentence.
Pre-training
Training a model first on a huge amount of general data, before any specific job.
Fine-tuning
A second, smaller round of training that adapts the pre-trained model to one job.

The 2017 paper

The transformer comes from a 2017 paper with a bold title: "Attention Is All You Need". It has eight authors, led by Ashish Vaswani. Before it, the paper says, the dominant sequence models were complex recurrent or convolutional networks in an encoder-decoder setup.

The authors proposed a simpler design. In their words, a new network architecture, the Transformer, "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". No loop that carries a memory forward, as in the Recurrent Neural Networks lesson. Just attention.

They tested it on translation. The abstract reports a BLEU translation score of 28.4 on the WMT 2014 English-to-German task, over 2 BLEU above the best earlier results. On English-to-French it set a new single-model best of 41.8 after 3.5 days of training on eight GPUs, which the authors call a small fraction of earlier training costs. The models, it adds, were more parallelizable and needed significantly less time to train.

Encoder and decoder stacks

The original transformer has two halves, and each half is a stack: the same layer repeated one on top of the other.

  • Encoder stack

    The paper uses 6 identical layers. The encoder reads the whole input sentence at once and builds a rich description of every token, using the other tokens as context.

  • Decoder stack

    Also 6 identical layers. It writes the output one token at a time. It has an encoder layer's two parts plus a third that looks at the encoder's output.

  • Masking

    While writing, the decoder is stopped from peeking at later positions. The paper says this means the prediction for position i can depend only on the known outputs before i.

Inside one block

Each encoder layer, the paper says, has two sub-layers. The first is multi-head self-attention. The second is a simple, position-wise, fully connected feed-forward network. Position-wise means the same small network is applied to each token on its own.

You met attention in the last two lessons: each token gathers information from the others. The feed-forward layer is the part that then thinks about what was gathered. Attention is the meeting. The feed-forward layer is each person back at their desk.

Around each of the two sub-layers, the paper wraps two more ideas. First, a residual connection. Then layer normalisation.

  1. 1

    Keep a copy of the input

    Before the sub-layer runs, hold on to its input x. This is the shortcut.

  2. 2

    Run the sub-layer

    Attention in the first sub-layer, the feed-forward network in the second. Call what it returns Sublayer(x).

  3. 3

    Add the copy back

    Compute x + Sublayer(x). That is the residual connection. The sub-layer only has to learn what to change, not rebuild everything. Think of editing a draft with tracked changes instead of retyping the whole page.

  4. 4

    Tidy with layer normalisation

    Rescale each token's numbers so they have an average near 0 and a similar spread. This keeps the numbers from drifting too large or too small as layers pile up.

The paper writes the whole recipe in one line. The output of each sub-layer is:

LayerNorm(x+Sublayer(x))\mathrm{LayerNorm}\bigl(x + \mathrm{Sublayer}(x)\bigr)

In words: take the input, add the sub-layer's answer to it, then normalise the result. To make this adding possible, the paper says every sub-layer and the embedding layers produce outputs of the same size, 512 numbers per token.

Positional encoding: where am I?

Here is a puzzle. Attention looks at all tokens together, with no left and right. Wikipedia says self-attention alone is permutation-invariant, which means that if you shuffle the words, it cannot tell. But "the dog bit the man" and "the man bit the dog" must not look the same.

A recurrent network knew the order for free, reading one word at a time. The transformer has no recurrence and no convolution, so, the paper says, we must inject some information about the relative or absolute position of the tokens. It does this by adding positional encodings to the input embeddings. They are the same size as the embeddings so the two can be summed.

The paper allows many choices, learned and fixed. Its own choice is sine and cosine waves of different frequencies:

PE(pos, 2i)=sin⁡ ⁣(pos100002i/dmodel)PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)
PE(pos, 2i+1)=cos⁡ ⁣(pos100002i/dmodel)PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)

Said in words: pos is the position of the token, 0, 1, 2 and so on. d_model is how many numbers each token has. Numbers in the list come in pairs. For pair number i, divide the position by 10000 raised to the power 2i over d_model, then take the sine for the even slot and the cosine for the odd slot.

Let us compute it. This code builds the encoding for positions 0 to 5 with a model size of 8, so four pairs of numbers, and prints each row rounded to 3 places.

Python
import math

d_model = 8

def positional_encoding(pos, d_model):
    row = []
    for i in range(d_model // 2):
        angle = pos / (10000 ** (2 * i / d_model))
        row.append(math.sin(angle))
        row.append(math.cos(angle))
    return row

for pos in range(6):
    row = positional_encoding(pos, d_model)
    print(pos, [round(v, 3) for v in row])

Output

0 [0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0]
1 [0.841, 0.54, 0.1, 0.995, 0.01, 1.0, 0.001, 1.0]
2 [0.909, -0.416, 0.199, 0.98, 0.02, 1.0, 0.002, 1.0]
3 [0.141, -0.99, 0.296, 0.955, 0.03, 1.0, 0.003, 1.0]
4 [-0.757, -0.654, 0.389, 0.921, 0.04, 0.999, 0.004, 1.0]
5 [-0.959, 0.284, 0.479, 0.878, 0.05, 0.999, 0.005, 1.0]

Read the first two numbers of each row. They are the fastest pair. They go 0.0 and 1.0 at position 0, then 0.841 and 0.54 at position 1, then 0.909 and -0.416 at position 2. They swing like a wave. The last pair, the slowest, barely moves: 0.001 and 1.0 at position 1, then 0.005 and 1.0 at position 5. Position 0 has sine 0 and cosine 1 in every pair, so its row is 0.0, 1.0 repeated. Every row is different, so each position has its own pattern.

One tiny block, by hand

Now let us run one small block on 3 tokens. To keep every number readable, each token is a list of just 4 numbers, and the weights are fixed by hand rather than learned. A real model learns its weights. The steps, though, are exactly the ones above.

To stay short, this uses one attention head, not several. The Multi-head Attention lesson showed how several heads run side by side. Here the attention uses softmax, the Activation Functions lesson's tool for turning scores into chances that add up to 1.

Python
import math

# 3 tokens, each a list of 4 numbers (already embedded, position added)
x = [
    [1.0, 0.0, 1.0, 0.0],
    [0.0, 2.0, 0.0, 1.0],
    [1.0, 1.0, 0.0, 1.0],
]

def matmul(a, b):
    return [[sum(a[i][k] * b[k][j] for k in range(len(b))) for j in range(len(b[0]))]
            for i in range(len(a))]

def transpose(a):
    return [list(r) for r in zip(*a)]

def softmax(row):
    m = max(row)
    exps = [math.exp(v - m) for v in row]
    total = sum(exps)
    return [e / total for e in exps]

def layer_norm(row):
    mean = sum(row) / len(row)
    var = sum((v - mean) ** 2 for v in row) / len(row)
    return [(v - mean) / math.sqrt(var + 1e-5) for v in row]

def show(name, m):
    print(name, f"shape {len(m)}x{len(m[0])}")
    for r in m:
        print("  ", [round(v, 3) for v in r])

# fixed weights (chosen by hand, not learned)
Wq = [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]
Wk = [[0, 1, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1], [0, 0, 1, 0]]
Wv = [[0.5, 0, 0, 0], [0, 0.5, 0, 0], [0, 0, 0.5, 0], [0, 0, 0, 0.5]]
W1 = [[1, -1, 0, 0, 1, 0], [0, 1, 1, 0, 0, -1], [1, 0, 0, 1, -1, 0], [0, 0, 1, -1, 0, 1]]
W2 = [[0.1, 0, 0, 0], [0, 0.1, 0, 0], [0, 0, 0.1, 0], [0, 0, 0, 0.1], [0.1, 0, 0, 0], [0, 0, 0, 0.1]]

show("input x", x)

# 1. attention
q, k, v = matmul(x, Wq), matmul(x, Wk), matmul(x, Wv)
scores = [[v_ / math.sqrt(4) for v_ in r] for r in matmul(q, transpose(k))]
weights = [softmax(r) for r in scores]
show("attention weights", weights)
attn = matmul(weights, v)
show("attention output", attn)

# 2. residual + layer norm
added = [[a + b for a, b in zip(r1, r2)] for r1, r2 in zip(x, attn)]
h = [layer_norm(r) for r in added]
show("after add and norm", h)

# 3. feed-forward (expand to 6, ReLU, shrink to 4)
hidden = [[max(0.0, v_) for v_ in r] for r in matmul(h, W1)]
show("feed-forward hidden", hidden)
ff = matmul(hidden, W2)

# 4. residual + layer norm again
out = [layer_norm([a + b for a, b in zip(r1, r2)]) for r1, r2 in zip(h, ff)]
show("block output", out)

Output

input x shape 3x4
   [1.0, 0.0, 1.0, 0.0]
   [0.0, 2.0, 0.0, 1.0]
   [1.0, 1.0, 0.0, 1.0]
attention weights shape 3x3
   [0.122, 0.547, 0.331]
   [0.547, 0.122, 0.331]
   [0.333, 0.333, 0.333]
attention output shape 3x4
   [0.227, 0.712, 0.061, 0.439]
   [0.439, 0.288, 0.273, 0.227]
   [0.333, 0.5, 0.167, 0.333]
after add and norm shape 3x4
   [1.2, -0.482, 0.658, -1.376]
   [-0.775, 1.545, -0.983, 0.213]
   [0.469, 0.781, -1.718, 0.469]
feed-forward hidden shape 3x6
   [1.858, 0.0, 0.0, 2.034, 0.542, 0.0]
   [0.0, 2.32, 1.758, 0.0, 0.208, 0.0]
   [0.0, 0.312, 1.249, 0.0, 2.186, 0.0]
block output shape 3x4
   [1.319, -0.588, 0.543, -1.273]
   [-0.823, 1.596, -0.874, 0.102]
   [0.605, 0.732, -1.718, 0.382]
  • Attention weights

    Each row adds up to 1. Token 1 gives 0.547 of its attention to token 2, and only 0.122 to itself. Token 3 splits evenly, 0.333 each.

  • After add and norm

    The attention output was added to the input, then each row was rescaled. Every row now has an average of about 0 and values mostly between -2 and 2, like 1.2, -0.482, 0.658, -1.376.

  • Feed-forward and the second residual

    Each token was widened from 4 numbers to 6, the negatives were cut to zero (ReLU, from the Activation Functions lesson), then shrunk back to 4. That was added to its input again and normalised once more.

Look at the shapes. The input is 3 by 4: three tokens, four numbers each. The block output is also 3 by 4. A block takes in a list of tokens and hands back a list of the same size. That is the reason blocks can be stacked as high as you like.

Quick check

Why is the block's output the same shape as its input?

Three families

The original transformer used both halves, encoder and decoder. Later models often keep only one half, or both. Wikipedia names these arrangements: encoder-only, decoder-only and encoder-decoder models use different arrangements of the layers. Three well-known examples:

  • Encoder-only: BERT

    BERT is a language model introduced in October 2018 by researchers at Google. It uses the encoder-only transformer architecture and is trained by masked token prediction and next sentence prediction.

  • Decoder-only: GPT

    Wikipedia says GPT models use only the decoder part of the transformer architecture, pre-trained to predict the next token in a sequence.

  • Encoder-decoder: T5

    T5, short for Text-to-Text Transfer Transformer, is a series of language models developed by Google AI and introduced in 2019. Like the original transformer, the encoder processes the input text and the decoder generates the output text.

Quick check

Which family fits a model that reads a sentence in one language and writes it in another?

Why transformers replaced RNNs

Two reasons stand out, and both come from the paper.

The first is speed of training. The paper explains that a recurrent network builds a hidden state at each position from the one before it, and this inherently sequential nature precludes parallelization within training examples. In plain words: step 5 must wait for step 4. A transformer looks at all the positions at once, so the work can be spread across many chips at the same time. Wikipedia adds that this parallelizability was an important factor in its widespread use.

The second is long-range links. The paper notes that the shorter the paths between any two positions in the input and output, the easier it is to learn long-range dependencies. In attention, any token can look straight at any other token in one step. In a recurrent network, the news from a word 50 steps back has to be passed along 50 times, and the Recurrent Neural Networks lesson showed how the memory fades on the way.

Pre-training, then fine-tuning

Big transformers are rarely trained from scratch for one narrow job. Wikipedia describes the usual two-step recipe for GPT-style models. First, the model is trained on a large, unlabeled dataset, the pre-training step, to learn to generate data points. Then this pre-trained model is adapted to a specific task using a labeled dataset, the fine-tuning step.

It is like a new hire. First they spend years at school and in life learning how language and the world work. Then they spend a few weeks learning your company's way of doing one job. The long first stage is expensive, but it is done once. The short second stage is cheap and can be repeated for many jobs. Wikipedia's example: ChatGPT is a chatbot based on a fine-tuned variant of GPT-3.5.

Beyond text, and on to big language models

Transformers did not stay with words. Wikipedia says a vision transformer decomposes an input image into a series of patches, instead of text into tokens, turns each patch into a vector, and then processes these vectors with a transformer encoder as if they were tokens. A photo becomes a row of small tiles, and the tiles are read like words in a sentence.

And words are where the biggest result came. Wikipedia notes that, starting in 2018, the OpenAI GPT series of decoder-only transformers became state of the art in natural language generation, and that ChatGPT's popularity at the end of 2022 triggered a boom around large language models. A large language model, in plain words, is a transformer pre-trained on a huge amount of text. The blocks you learned in this lesson are inside it, just made much bigger.

Recap and what is next

A transformer is a stack of blocks: attention, then a feed-forward layer, each wrapped in a residual shortcut and layer normalisation. Positional encodings tell it the order. BERT, GPT and T5 are its three families. It trains in parallel and links far-apart tokens directly, which is why it replaced recurrent networks for many jobs.

Next: generative adversarial networks, two networks that train each other.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.