Machine Learning Foundations / Advanced / 03.40
Self-Attention
How every word in a sentence asks questions of all the others, scores the answers, and blends them, all at the same time.
13 min readFree
Every word looks at every other word
The last lesson, Attention Mechanisms, ended with a promise: next, self-attention, where every word looks at every other word. Here it is.
Picture a group chat with four people. Before anyone replies, each person reads the whole chat and decides whose messages matter most for what they want to say. Nobody waits their turn in a line. Everyone reads everyone, at once. That is self-attention.
Why does it matter? Take the sentence "The animal didn't cross the street because it was too tired." What does "it" mean, the street or the animal? A person knows at once. A program has to work it out. Jay Alammar, who wrote a famous illustrated guide to the Transformer, says that when the model is processing the word "it", self-attention allows it to associate "it" with "animal". The word borrows meaning from the other words around it.
Key words
- Self-attention
- A way for each word in a sentence to look at the other words of the same sentence and collect clues from them.
- Embedding
- A short list of numbers that stands for a word. Words with similar meanings get similar lists.
- Query
- A list of numbers made from one word. It stands for the question that word is asking.
- Key
- A list of numbers made from one word. It stands for the label that word shows to the others.
- Value
- A list of numbers made from one word. It stands for the content that word hands over when someone picks it.
- Dot product
- Multiply two lists of numbers place by place, then add the results. A bigger answer means the two lists match more.
- Softmax
- A step that turns any list of scores into shares that are all positive and add up to 1.
One sentence, three roles
In the attention you met last time, the questions came from one place and the answers from another, as when a translator reads a French sentence while writing English. In self-attention, there is only one sentence. Wikipedia puts it in one line: the query, key and value vectors all come from the same model. A vector is just a list of numbers.
So how can one word be a question, a label and a piece of content all at once? Each word starts as an embedding. We then multiply that embedding by three different grids of numbers, called weight matrices. Alammar says these were trained during the training process, just like the weights you learned about in the Backpropagation lesson. The three results are the word's query, key and value.
A library is a handy picture. This is our own analogy, not from the sources. You walk in with a question (the query). Every book has a label on its spine (the key). You compare your question with each label. The books that match best are the ones whose pages (the values) you read most.
Query: what am I looking for?
Made from the word that is doing the looking. The word "it" might ask for a noun it could stand for.
Key: what do I offer?
Made from every word, including the one doing the looking. It is the label that gets compared with the query.
Value: what do I hand over?
Also made from every word. Once the scores are in, the values are what get mixed together to make the result.
Alammar calls query, key and value abstractions that are useful for calculating and thinking about attention. They are a helpful picture, not a law.
Sizes, from the original paper as Alammar describes it: each word's embedding has 512 numbers, and its query, key and value each have 64.
The recipe, step by step
Alammar walks through the calculation for one word at a time in six steps. Here is the same walk in our words. Think of the word you are working on as the reader.
- 1
Make Q, K and V
Multiply every word's embedding by the three weight matrices. Each word now has a query, a key and a value.
- 2
Score
Take the reader's query and compute its dot product with every word's key. That gives one score per word. The score decides how much focus to put on that word.
- 3
Divide
Divide each score by the square root of the key size. Alammar divides by 8, since the keys in the paper have 64 numbers.
- 4
Softmax
Turn the scores into shares that are positive and add up to 1. These shares are the attention weights.
- 5
Weigh the values
Multiply each word's value by its share. In Alammar's words, this keeps the words we want to focus on and drowns out irrelevant words.
- 6
Add up
Sum the weighted values. That sum is the output for the reader's position.
The formula
Stack all the queries into one grid Q, all the keys into K, and all the values into V. Then the six steps above fit on one line. This is scaled dot-product attention, as Wikipedia calls it:
In words: multiply Q by the flipped K. That gives every word's score against every other word at once. Divide all of them by the square root of d_k, the number of values in a key. Run softmax on each row, so every row becomes shares that add up to 1. Finally, multiply by V to blend the values with those shares. The small T on K means the grid is flipped, so the multiplication lines up.
Why divide by the square root?
The division looks like a small detail. What do the sources say about it? Alammar says that dividing the scores by 8, the square root of 64, leads to having more stable gradients. Gradients are the blame numbers that the network uses to learn, as in the Backpropagation lesson. He adds that there could be other possible values, but this is the default.
Wikipedia agrees on the size of the fix: the scale factor is usually set to one over the square root of the key size. The sources we used say no more than that, so we will not invent a longer story.
Who looks at whom: the attention matrix
After softmax, each reader has one row of shares. Put all the rows together and you get a square table. Row = the word that is looking. Column = the word being looked at. Each row adds up to 1. This is the attention matrix, and it is the picture people draw when they show "which words look at which". Wikipedia calls this setup all-to-all attention, as the vector at every position can attend to every other.
In Alammar's example, a trained model processing "it" gives part of its attention to "The animal" and bakes a little of that word into the result. In our code below, nothing is trained. The numbers are made up by hand, so the demo shows how the maths works, not what a real model has learned.
Self-attention in plain Python
Here is the whole thing for a four-word sentence, "the cat sat down". Everything is our own made-up demo: each word gets an embedding of three numbers, and the three weight matrices are small fixed grids of whole numbers that we typed in. Real weights are learned, and real embeddings are far longer.
The code multiplies, scores, divides by the square root of 3, runs softmax on each row, and blends the values. It uses only Python's built-in math module.
import math
words = ["the", "cat", "sat", "down"]
X = [[1.0, 0.0, 1.0],
[0.0, 2.0, 1.0],
[1.0, 1.0, 0.0],
[0.0, 1.0, 2.0]]
W_Q = [[1, 0, 1], [0, 1, 0], [1, 1, 0]]
W_K = [[0, 1, 0], [1, 0, 1], [0, 1, 1]]
W_V = [[1, 0, 0], [0, 2, 0], [0, 0, 1]]
def matmul(A, B):
return [[sum(a * b for a, b in zip(row, col)) for col in zip(*B)] for row in A]
def softmax(row):
exps = [math.exp(v) for v in row]
total = sum(exps)
return [e / total for e in exps]
def show(title, M):
print(title)
print(" " + "".join(f"{w:>7}" for w in words))
for w, row in zip(words, M):
print(f"{w:>7} " + "".join(f"{v:7.2f}" for v in row))
Q, K, V = matmul(X, W_Q), matmul(X, W_K), matmul(X, W_V)
print("Q", Q)
print("K", K)
print("V", V)
d = len(K[0])
scores = [[v / math.sqrt(d) for v in row] for row in matmul(Q, list(zip(*K)))]
show("scaled scores", scores)
weights = [softmax(row) for row in scores]
show("attention (each row adds to 1)", weights)
out = matmul(weights, V)
print("outputs")
for w, row in zip(words, out):
print(f"{w:>7}", [round(v, 2) for v in row])Output
Q [[2.0, 1.0, 1.0], [1.0, 3.0, 0.0], [1.0, 1.0, 1.0], [2.0, 3.0, 0.0]]
K [[0.0, 2.0, 1.0], [2.0, 1.0, 3.0], [1.0, 1.0, 1.0], [1.0, 2.0, 3.0]]
V [[1.0, 0.0, 1.0], [0.0, 4.0, 1.0], [1.0, 2.0, 0.0], [0.0, 2.0, 2.0]]
scaled scores
the cat sat down
the 1.73 4.62 2.31 4.04
cat 3.46 2.89 2.31 4.04
sat 1.73 3.46 1.73 3.46
down 3.46 4.04 2.89 4.62
attention (each row adds to 1)
the cat sat down
the 0.03 0.58 0.06 0.33
cat 0.27 0.15 0.09 0.49
sat 0.08 0.42 0.08 0.42
down 0.15 0.27 0.09 0.49
outputs
the [0.09, 3.1, 1.27]
cat [0.36, 1.76, 1.4]
sat [0.15, 2.7, 1.35]
down [0.24, 2.24, 1.4]Read it from the top. The first three lines print Q, K and V: each word now has three numbers for each role. Then come the scaled scores. The word "the" scores 4.62 against "cat" and only 1.73 against itself. After softmax, its row says it gives 0.58 of its attention to "cat", 0.33 to "down", 0.06 to "sat" and 0.03 to itself. Add those four and you get 1, as every row must.
Look at the row for "sat". It splits its attention evenly between "cat" and "down", 0.42 each, and gives just 0.08 to "the" and 0.08 to itself. The last row, "down", puts 0.49 on itself and 0.27 on "cat". None of this means anything about English. We typed the matrices by hand. It shows that each word picks its own mix.
The outputs are the blended values. The output for "the" is [0.09, 3.1, 1.27]. It is mostly the value of "cat", whose second number is 4, because "the" paid most attention to "cat".
Quick check
In our run, the row for "sat" reads 0.08, 0.42, 0.08, 0.42. What does that tell you?
All words at once
Look back at the code. There is no loop that walks through the sentence one word at a time, handing a memory note forward. In the Recurrent Neural Networks lesson, the network had to read word 1 before word 2, and word 2 before word 3. Wikipedia says RNNs operate one token at a time, from first to last, and cannot operate in parallel over all tokens in a sequence. A token is one small piece of text, usually a word or part of one.
Self-attention is a handful of grid multiplications. Each word's row can be worked out without waiting for the others. The arXiv abstract of the paper that introduced the Transformer says the new design is more parallelizable and takes significantly less time to train, because it dispenses with recurrence and convolutions entirely.
Tip
There is a price. Wikipedia says standard self-attention has computational and memory costs that grow quadratically with sequence length. Doubling the length of the text makes the attention table about four times bigger, because every word is compared with every other.
The missing order
Here is a catch. Our table says how much each word matches the others. It does not say where any word is. If you shuffled the words, each word's row would still be made of the same scores, just in a new place. Wikipedia states it this way: self-attention alone is permutation-invariant, so transformers inject positional information, typically through positional encodings or learned positional embeddings.
Remember "The dog bit the man" and "The man bit the dog" from the Recurrent Neural Networks lesson. Same words, opposite news. A model that cannot tell the order cannot tell them apart. Alammar puts it as one thing that's missing from the model: a way to account for the order of the words in the input sequence.
His fix: the Transformer adds a vector to each input embedding. These vectors follow a pattern that helps the model work out the position of each word, or the distance between words. Think of numbering the seats in a cinema. The people are the same, but now each one carries a seat number. Later lessons on Transformers show how this fits into the full design.
Masking: no peeking at the future
One more tool. When a model writes text one word at a time, it must not look at words it has not written yet. Cheating would be too easy: while guessing word 3, it could just read word 3. Wikipedia says the decoder cannot attend to future outputs that have yet to be decoded, and the fix is called causal masking: the attention weight is forced to zero wherever a word would look at a later word.
Alammar describes how it is done. The self-attention layer in the decoder is only allowed to attend to earlier positions, by masking future positions, setting them to minus infinity, before the softmax step. A score of minus infinity becomes a share of exactly 0 after softmax. Wikipedia says the effect of adding minus infinity is to zero out the corresponding attention weight. Each word can still look at itself and everything before it.
Add this to our code. Keep the same scores, but replace every score to the right of the diagonal with minus infinity, then run softmax again. The table becomes a triangle:
masked = [[v if j <= i else float("-inf") for j, v in enumerate(row)]
for i, row in enumerate(scores)]
show("attention with causal mask", [softmax(row) for row in masked])Output
attention with causal mask
the cat sat down
the 1.00 0.00 0.00 0.00
cat 0.64 0.36 0.00 0.00
sat 0.13 0.74 0.13 0.00
down 0.15 0.27 0.09 0.49Read the triangle. The first word, "the", can only see itself, so its row is 1.00 and then zeros. "cat" splits its attention between "the" (0.64) and itself (0.36). "sat" now gives 0.74 to "cat" and 0.13 each to "the" and itself. The last word, "down", is unchanged: 0.15, 0.27, 0.09 and 0.49, because it was allowed to see everyone before it anyway.
Quick check
Which kind of job needs the causal mask?
Quick recap
Tip
Try it yourself: change one number in W_Q and run the code again. Watch which rows of the attention table move.
Next: multi-head attention, many ways of looking at once.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.