esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.41

Multi-head Attention

Why a model looks at a sentence with several attention heads at once, how their answers are joined, and what each head ends up doing.

14 min readFree

Many ways of looking at once

The last lesson, Self-Attention, ended with a promise: next, multi-head attention, many ways of looking at once. Here it is.

Picture a group of friends watching the same film. One follows the plot. One watches the clothes. One listens to the music. Afterwards they sit down and share notes. Together they know far more about the film than any one of them could.

Multi-head attention works like that group. Instead of one attention calculation per word, the model runs several at the same time. Each one is called a head. Each head looks at the sentence in its own way, and then their notes are joined together.

By the end you will know why one head is not enough, how the model size is shared out between heads, how the answers are joined, and what it costs. You will also run a tiny two-head example in plain Python.

Key words

Attention head
One full attention calculation with its own set of settings. A model runs several of them side by side.
Query, key, value (Q, K, V)
Three lists of numbers made from each word. The query asks, the key answers who is relevant, and the value is what gets passed along.
Weight matrix
A grid of numbers the model learns. Multiplying a word's numbers by it turns them into a new list.
Attention weights
How much one word looks at each other word. Each row adds up to 1.
Concatenate
Join lists end to end. Joining [1, 2] and [3, 4] gives [1, 2, 3, 4].
Projection
Multiplying by a matrix to turn a list of numbers into a new list, often of a different length.

What one head does

Quick reminder of the single-head version. For every word, the model makes a query, a key and a value, using three weight matrices it has learned. The query of one word is compared with the keys of all the words. That gives scores. The scores go through softmax, which turns them into weights that add up to 1. Then the model adds up the values, using those weights.

Wikipedia says that for each attention unit, the transformer model learns three weight matrices: the query weights, the key weights and the value weights. The original paper also scales the scores. It divides the dot products by the square root of the key size, to stop the numbers growing too large.

A single head produces one blend per word. Alammar points out a weakness. In his example, the blend for a word can contain a little of every other word, but it could be dominated by the word itself.

The original paper makes a related point. With a single attention head, averaging inhibits the model from attending to information from different representation subspaces at different positions. In plain words: when one head blends everything into one average, different kinds of information get squashed together.

Tip

Think of a mixing desk. One slider has to serve every instrument at once. If you want the singer louder and the drums quieter, you need more sliders. Heads are extra sliders.

Several heads in parallel

The fix is simple. Run the attention calculation several times, side by side. The paper's wording: instead of performing a single attention function, the authors found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections. On each projected version, the attention function runs in parallel.

Each head has its own Q, K and V matrices. That is the key point. Alammar says the model keeps separate Q/K/V weight matrices for each head. Each set starts out random, and after training, each set projects the input into a different representation subspace.

A representation subspace sounds heavy, so here is a daily-life version. Describe a person by their height and age, and you will group people one way. Describe them by their job and city, and you will group them another way. Same people, two different views. A head is one view.

  1. 1

    Make views

    Each head has its own Q, K and V matrices. Each turns every word into three short lists.

  2. 2

    Each head attends

    Inside every head, queries meet keys, softmax makes weights, and the weights mix the values.

  3. 3

    Join the answers

    The outputs of all heads are placed end to end.

  4. 4

    Mix with the output matrix

    One last matrix blends the joined list into the final answer for each word.

Wikipedia adds a helpful way to see it. While each attention head attends to the tokens that are relevant to each token, multiple heads allow the model to do this for different definitions of "relevance". A token is the small piece of text, often a word, that the model reads.

Sharing out the model size

Each word inside the model is a list of numbers. The paper calls the length of that list d_model. In the original Transformer it is 512.

You might expect eight heads to need eight times the work. They do not, because the list is split up. The paper uses h = 8 parallel attention layers, or heads. For each of them it sets the key and value size to d_model divided by h, which is 64.

So 512 is shared out as 8 heads of 64. It is like cutting one pizza into eight slices. Each head gets a slice, and nobody gets the whole pizza. The paper adds the payoff: due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality.

Python
d_model = 512
heads = 8
d_head = d_model // heads
print("size of each head:", d_head)
print("heads side by side:", heads * d_head)

# Numbers to compare one pair of words in the scoring step
one_big_head = d_model
many_small_heads = heads * d_head
print("multiplications per pair, 1 head:", one_big_head)
print("multiplications per pair, 8 heads:", many_small_heads)

Output

size of each head: 64
heads side by side: 512
multiplications per pair, 1 head: 512
multiplications per pair, 8 heads: 512

The first two lines show the split: 512 divided by 8 is 64, and eight heads of 64 side by side make 512 again. The last two lines show the idea behind the paper's sentence. When scoring one pair of words, one big head works with 512 numbers. Eight small heads work with 64 numbers each, 512 in all. The scoring work is the same size. This is only a count for one step, not the whole cost of a model.

Wikipedia gives a real example. The smallest GPT-2 model has an embedding size of 768, 12 heads and a head size of 64. Since 12 times 64 is 768, the heads fit back into the full size exactly.

Joining the heads

Each head hands back its own list for each word. The next layer wants one list per word, not eight. Alammar describes the problem like this: the next layer is not expecting eight matrices, it is expecting a single one. So the model concatenates the head outputs, then multiplies by one more weight matrix, written W^O.

Here is the formula from the paper, then the same thing in words.

MultiHead(Q,K,V)=Concat(head1,…,headh) WOwhereheadi=Attention(QWiQ, KWiK, VWiV)\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_h)\,W^O \quad\text{where}\quad \mathrm{head}_i = \mathrm{Attention}(QW_i^Q,\,KW_i^K,\,VW_i^V)

In words: for each head i, multiply the queries, keys and values by that head's own matrices, and run attention. Place all the head results end to end. Then multiply the long list by the output matrix W^O.

The paper gives the sizes. W^O has h times d_v rows and d_model columns. With 8 heads of 64, that is 512 in and 512 out. The joined list is 512 long, and so is the final answer. The word keeps the same length it started with, so the next layer can use it without changes.

Tip

The output matrix is more than glue. Without it, the heads would just sit side by side and never talk to each other. W^O lets the model mix what different heads found.

Two heads in plain Python

Now a small example you can run. The sentence is "the dog chased cats". Each word is four made-up numbers: two stand for meaning, two for where the word sits in the sentence. Real models learn such numbers. Ours are picked by hand so you can follow along.

There are two heads. Head 1 is built to read the two position numbers. Head 2 is built to read the two meaning numbers. Each has its own small Q, K and V matrices, also picked by hand. Then the code joins the two answers and applies a 4 by 4 output matrix. It uses only Python's built-in math module.

Python
import math

words = ["the", "dog", "chased", "cats"]
# Made-up 4 numbers per word: two for meaning, two for where it sits in the sentence.
X = [
    [0.9, 0.1, 1.0, 0.0],
    [0.2, 0.9, 0.7, 0.3],
    [0.1, 0.8, 0.3, 0.7],
    [0.3, 0.9, 0.0, 1.0],
]
def matmul(A, B):
    return [[sum(a * b for a, b in zip(row, col)) for col in zip(*B)] for row in A]

def softmax(row):
    exps = [math.exp(v) for v in row]
    total = sum(exps)
    return [e / total for e in exps]

def head(X, Wq, Wk, Wv):
    Q, K, V = matmul(X, Wq), matmul(X, Wk), matmul(X, Wv)
    d = len(Q[0])
    scores = [[sum(q * k for q, k in zip(qr, kr)) / math.sqrt(d) for kr in K] for qr in Q]
    weights = [softmax(r) for r in scores]
    return weights, matmul(weights, V)

# Head 1 reads the position numbers. Head 2 reads the meaning numbers.
W1q = [[0, 0], [0, 0], [2, 0], [0, 2]]
W1k = [[0, 0], [0, 0], [2, 0], [0, 2]]
W1v = [[1, 0], [0, 1], [0, 0], [0, 0]]
W2q = [[3, 0], [0, 3], [0, 0], [0, 0]]
W2k = [[0, 3], [3, 0], [0, 0], [0, 0]]
W2v = [[0, 0], [0, 0], [1, 0], [0, 1]]
w1, z1 = head(X, W1q, W1k, W1v)
w2, z2 = head(X, W2q, W2k, W2v)

for name, w in (("Head 1", w1), ("Head 2", w2)):
    print(name, "attention (each row: one word looks at the, dog, chased, cats)")
    for word, row in zip(words, w):
        print("  %-7s" % word, [round(v, 2) for v in row])

concat = [a + b for a, b in zip(z1, z2)]
print("Concatenated shape:", len(concat), "x", len(concat[0]))
Wo = [[1.0, 0.5, 0.0, 0.0], [0.0, 1.0, 0.5, 0.0], [0.0, 0.0, 1.0, 0.5], [0.5, 0.0, 0.0, 1.0]]
out = matmul(concat, Wo)
print("Final output shape:", len(out), "x", len(out[0]))
for word, row in zip(words, out):
    print("  %-7s" % word, [round(v, 2) for v in row])

Output

Head 1 attention (each row: one word looks at the, dog, chased, cats)
  the     [0.62, 0.26, 0.08, 0.04]
  dog     [0.4, 0.29, 0.18, 0.13]
  chased  [0.13, 0.18, 0.29, 0.4]
  cats    [0.04, 0.08, 0.26, 0.62]
Head 2 attention (each row: one word looks at the, dog, chased, cats)
  the     [0.01, 0.38, 0.2, 0.41]
  dog     [0.86, 0.04, 0.02, 0.08]
  chased  [0.87, 0.04, 0.02, 0.07]
  cats    [0.79, 0.07, 0.03, 0.12]
Concatenated shape: 4 x 4
Final output shape: 4 x 4
  the     [0.96, 0.71, 0.53, 0.83]
  dog     [0.53, 0.8, 1.18, 0.55]
  chased  [0.35, 0.92, 1.29, 0.55]
  cats    [0.34, 0.97, 1.26, 0.58]

First, look at the two attention tables. Each row is one word looking at the, dog, chased and cats. Every row adds up to about 1.

Head 1 follows the position numbers. The word the puts 0.62 of its attention on itself and 0.04 on cats, the last word. The word cats does the opposite, with 0.62 on itself and only 0.04 on the. Words look mostly at words near them in the sentence.

Head 2 follows the meaning numbers, with its query and key matrices set differently. Look at the column for the. The words dog, chased and cats put 0.86, 0.87 and 0.79 of their attention on it. Meanwhile the word the itself spreads its attention over dog and cats, with 0.38 and 0.41. The two tables look nothing alike, and that is the point of having two heads.

These patterns come from numbers we chose. They are not real grammar. They only show that two sets of matrices can look at the same four words in two different ways.

Next, the join. Each head gives a 4 by 2 result, so the joined list is 4 by 4: one list of four numbers for each of the four words. The output matrix then mixes those four numbers. The final result is again 4 by 4, so the sentence keeps its shape. For example, the word dog ends up as 0.53, 0.8, 1.18 and 0.55.

Quick check

In the code, why is the final output 4 by 4?

Cost, and why it stays parallel

Does running several heads make the model slow? Mostly no, for three reasons.

  • Same size of work

    The model size is split across the heads, so the scoring work is about the same as one full-size head. The paper says the total computational cost is similar.

  • Heads do not wait on each other

    Head 3 does not need head 2's answer. Wikipedia says the computations for each attention head can be performed in parallel, which allows for fast processing.

  • No walking along the sentence

    A recurrent network must read word by word. Attention looks at all words at once. The paper's table lists a self-attention layer with a constant number of sequential steps, while a recurrent layer needs a number that grows with the sentence length.

Parallel means many workers doing jobs at the same time instead of one after another. Think of eight cashiers serving eight customers at once, versus one cashier serving them in a line. Wikipedia notes that the multi-head model was easier to parallelize because of the use of independent heads and the lack of recurrence, and that this parallelizability was an important factor in its widespread use in large neural networks.

One honest limit. The same paper table lists the cost of a self-attention layer as growing with the square of the sentence length. Every word looks at every other word, so doubling the sentence makes the scoring about four times bigger.

What do heads actually learn?

It is tempting to say: head 1 learns grammar, head 2 learns meaning, head 3 learns position. Tidy, but only partly true.

The paper reports hopeful signs. In its words, not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences. Syntactic means about grammar. Semantic means about meaning.

Wikipedia is more careful. It says analyses of some models have found attention heads that focus on patterns such as nearby words or syntactic relations, and that these patterns vary across heads and models.

And Alammar gives a gentle warning. He shows two heads focusing on different words when the model reads the word "it" in a sentence. Then he notes that if you add all the heads to the picture, things can be harder to interpret.

Tip

A fair summary: some heads show readable patterns, and many are hard to put a name on. Treat any story like "this head does grammar" as a guess about one model, not a rule for all of them.

Quick check

What does the output matrix W^O do?

Recap

  • Several heads

    Each head has its own Q, K and V matrices, so it can look at the sentence in its own way.

  • Shared out size

    In the original paper, 512 is split into 8 heads of 64, so the work stays about the same.

  • Join and mix

    Head outputs are placed end to end, then one output matrix blends them into the final answer.

Next: transformers, the architecture built on attention.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.