esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.39

Attention Mechanisms

How a model looks back at every word it has read, scores each one, and builds its answer from the ones that matter most.

13 min readFree

Looking back at what matters

The last lesson, GRU, ended with a promise: next, attention, letting a model look back at what matters. Here it is.

A small test. Read this, then cover it: "The cat that chased the mouse across the garden all afternoon finally sat on the mat." What sat on the mat? You say "the cat" at once. You jumped back to the one word that mattered.

The models in the last few lessons cannot jump back. A recurrent network reads word by word and keeps one set of notes. Attention gives it that skill: glance back at every earlier word and decide how much each one matters right now.

By the end you will know the problem attention fixes, what query, key and value mean, and the three steps it runs. You will also watch a tiny version work in plain Python.

Key words

Vector
A list of numbers. A model turns each word into a vector so it can do maths on it.
Bottleneck
A narrow spot that everything has to squeeze through, like the neck of a bottle.
Encoder and decoder
Two halves of a translation model: one reads the sentence, one writes the translation.
Score
A number for how well one word fits what the model wants right now. Bigger is a better fit.
Weight
A share between 0 and 1. All the weights together add up to 1, like slices of one pie.
Alignment
Which input word each output word lines up with, such as which French word a given English word comes from.

The one-vector bottleneck

Start with the job that attention was born for: translation. Jay Alammar describes the classic model like this. It has an encoder and a decoder. The encoder processes each word of the input and compiles what it captured into one vector, which he calls the context. After the whole input is read, the encoder sends the context to the decoder, which produces the translation word by word.

Picture a friend who reads a whole chapter, then has to retell it from one sticky note. For a short page, fine. For a long chapter, the note runs out of room.

That is the bottleneck. Alammar says the context vector turned out to be a bottleneck for these models and made it challenging for them to deal with long sentences. Lilian Weng names the same weakness: with one fixed-length vector, the model often has forgotten the first part once it finishes the whole input.

Wikipedia adds that recurrent networks favour the latest words, so earlier ones count for less.

The researchers behind the fix said it plainly. In their 2014 paper, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio write that they "conjecture that the use of a fixed-length vector is a bottleneck". A conjecture is an educated guess.

Tip

Alammar says real context vectors have sizes like 256, 512 or 1024. Five words or fifty, the sentence must fit.

The idea: look at everything, weigh each part

The fix is simple to say. Do not throw away the encoder's notes. Keep one note per input word, and let the decoder look at all of them whenever it writes a word.

Alammar names two differences from the classic model. First, the encoder passes a lot more data to the decoder: instead of only the last hidden state, it passes all of them. Second, the decoder does an extra step before each output word, to focus on the parts of the input that matter for that step.

Lilian Weng calls these "shortcuts between the context vector and the entire source input", with weights that are customised for each output element. Think of a shelf with one note per textbook page. The new model walks to the shelf for every answer.

Wikipedia defines attention as a method that determines the importance of, or "attends to", each component in a sequence relative to the other components. In language, that importance is represented by weights given to each word.

Query, key and value

Attention is usually described with three words: query, key and value. A library makes them easy.

You walk into a library with a question in your head: "I want something about gardens." That question is the query. Every book on the shelf has a label on its spine: "Gardening for Beginners", "Roman History". The labels are the keys. Inside each book is the content you actually want to read. That is the value.

You compare your question with each label. The labels that fit best are the books you pull down and read most closely. You do not take one book and ignore the rest. You take a bit of each, in proportion to how well its label matched.

  • Query

    What I am looking for right now. In translation, it comes from the decoder, which is about to write the next word.

  • Key

    A label on each input word that says what it is about. The query is compared with every key.

  • Value

    What each input word hands over if it is chosen. The final answer is made from the values.

Lilian Weng gives the same picture: the input is a set of key-value pairs, and in translation both are the encoder's hidden states. The output is a weighted sum of the values, where each weight comes from the dot product of the query with all the keys. A weighted sum is an average where some items count more than others. We compute exactly that soon.

Quick check

In the library picture, which part is the query?

Three steps: score, softmax, add up

Every attention calculation follows the same three steps.

  1. 1

    Score every key

    Compare the query with each key and get one number per input word. Bigger means a better match. A common way to compare is the dot product: multiply the two lists of numbers position by position, then add up the results.

  2. 2

    Turn scores into weights

    Raw scores can be any size, even negative. Softmax turns them into weights between 0 and 1 that add up to 1. Alammar describes the effect as amplifying the hidden states with high scores and drowning out those with low scores.

  3. 3

    Add up the values

    Multiply each value by its weight, then add them all together. The result is one vector: a blend that leans towards the best matches. The decoder uses it to write its next word.

Softmax came up in the Activation Functions lesson. Here is the whole thing as one formula. Q is the queries, K the keys, V the values, and d the length of each key.

Attention(Q,K,V)=softmax(QK⊤d)V\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{Q K^{\top}}{\sqrt{d}}\right) V

Say it in words: multiply the queries by the keys, which scores every pair. Divide by the square root of d, a scaling step that keeps the numbers from growing too large. Run softmax to turn each row of scores into weights. Finally multiply by the values to get the weighted sum. Weng gives this form, and Wikipedia gives the same formula with a scaling number in front.

Try it: attention in plain Python

Let us run it. We have four words, each with a made-up key and a made-up value. The numbers are invented: a key's three numbers mean "animal-ness", "place-ness" and "action-ness". The query is "I am looking for an animal", so it is large in the first position.

Python
import math

words = ["the", "cat", "sat", "mat"]
keys = [
    [0.1, 0.1, 0.1],
    [0.9, 0.1, 0.2],
    [0.1, 0.2, 0.9],
    [0.1, 0.9, 0.1],
]
values = [
    [0.0, 0.1],
    [1.0, 0.2],
    [0.3, 1.0],
    [0.5, 0.0],
]
query = [3.0, 0.0, 0.5]  # "I am looking for an animal"

scores = [sum(q * k for q, k in zip(query, key)) for key in keys]
print("scores :", [round(s, 2) for s in scores])

exps = [math.exp(s) for s in scores]
weights = [e / sum(exps) for e in exps]
for w, word in zip(weights, words):
    print(f"weight {word}: {w:.2f}")
print("sum of weights:", round(sum(weights), 2))

out = [sum(w * v[i] for w, v in zip(weights, values)) for i in range(2)]
print("output :", [round(x, 2) for x in out])

Output

scores : [0.35, 2.8, 0.75, 0.35]
weight the: 0.07
weight cat: 0.77
weight sat: 0.10
weight mat: 0.07
sum of weights: 1.0
output : [0.83, 0.26]

Read the output from the top. The key for "cat" scored 2.8, far above the others, because its first number (animal-ness) is large and the query's first number is large too. Softmax turned those scores into weights: "cat" gets 0.77, which is 77 percent of the pie. "the" and "mat" get 0.07 each. The weights add up to 1.0.

The final output is [0.83, 0.26]. It is mostly the value of "cat", [1.0, 0.2], with a little of the other three mixed in.

Change the question, change the focus

Now the important part. The words and keys stay the same. Only the query changes. Does the focus move?

This version wraps the three steps in a function called attend and runs three queries: an animal, a place and an action.

Python
import math

words = ["the", "cat", "sat", "mat"]
keys = [
    [0.1, 0.1, 0.1],
    [0.9, 0.1, 0.2],
    [0.1, 0.2, 0.9],
    [0.1, 0.9, 0.1],
]
values = [
    [0.0, 0.1],
    [1.0, 0.2],
    [0.3, 1.0],
    [0.5, 0.0],
]


def attend(query):
    scores = [sum(q * k for q, k in zip(query, key)) for key in keys]
    exps = [math.exp(s) for s in scores]
    weights = [e / sum(exps) for e in exps]
    out = [sum(w * v[i] for w, v in zip(weights, values)) for i in range(2)]
    return weights, out


queries = {
    "animal": [3.0, 0.0, 0.5],
    "place": [0.0, 3.0, 0.5],
    "action": [0.5, 0.0, 3.0],
}
for name, q in queries.items():
    weights, out = attend(q)
    best = words[weights.index(max(weights))]
    print(f"{name:7} -> {[round(w, 2) for w in weights]}  top: {best}  output: {[round(x, 2) for x in out]}")

Output

animal  -> [0.07, 0.77, 0.1, 0.07]  top: cat  output: [0.83, 0.26]
place   -> [0.07, 0.07, 0.13, 0.73]  top: mat  output: [0.48, 0.15]
action  -> [0.07, 0.13, 0.73, 0.07]  top: sat  output: [0.39, 0.77]

Look at the weights, in the order the, cat, sat, mat. With the animal query the biggest weight, 0.77, lands on "cat". With the place query it moves to "mat", at 0.73. With the action query it moves to "sat", also at 0.73. The outputs follow: [0.83, 0.26], [0.48, 0.15], [0.39, 0.77].

The words did not change. The question did, so the model listened to a different word. That is the heart of attention: the weights are worked out fresh every time, from the question being asked.

Quick check

In the run above, the query changed from "animal" to "place". What happened to the weights?

The 2014 idea: attention for translation

Attention in this form came from machine translation. The paper is Neural Machine Translation by Jointly Learning to Align and Translate, by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio. It was submitted to arXiv on 1 September 2014 and accepted at the ICLR 2015 conference as an oral presentation.

Its abstract says the basic encoder-decoder model squeezes a source sentence into a fixed-length vector. The authors propose letting the model automatically "(soft-)search" the source sentence for the parts that matter for predicting each target word, without cutting the sentence into hard segments first.

"Soft" means no hard choice: the model spreads its weights across all the input words, like the 0.77 and 0.07 shares above, instead of picking one. "Jointly" in the title means it learns to align and to translate together.

Did it work? The abstract reports that, on English-to-French translation, the approach achieved a translation performance comparable to the existing state-of-the-art phrase-based system.

Weights you can inspect

Attention has a gift: you can look at it. The weights are plain numbers, so you can print them and see which words the model leaned on.

Bahdanau's team did exactly that. The abstract says their closer look at the results shows that the (soft-)alignments the model found match human intuition well. In other words, the words the model leaned on made sense to human readers.

Alammar shows a good case. The model did not just line up the first output word with the first input word. It learned from training how to align words in that language pair. In an example from the attention papers, English "European Economic Area" has its words in a different order in French: "européenne économique zone". Alammar says you can see how the model paid attention correctly when producing "European Economic Area", following the reversed order, while every other word in the sentence is in a similar order.

A neural network is usually a black box. Here the box has a window: draw the weights as a grid, input words along one side, output words along the other, with a bright square wherever the model looked.

Additive and dot-product scoring

Step one of the recipe, scoring, can be done in more than one way. Two names come up all the time.

Additive scoring, used in Bahdanau's model, runs the query and the key together through a tiny neural network of its own and lets that network output the score. Weng's table writes it with a weight vector and a tanh bend: the score is that vector multiplied with tanh of a weight matrix applied to the query and the key joined side by side. In words, the model learns how to compare. Weng notes that the Transformer paper called this "additive attention".

Dot-product scoring, used in the Luong paper and the Transformer, has nothing to learn in the comparison. It multiplies the query and the key position by position and adds the products, just as our Python code did. The scaled version divides the result by the square root of the key's length, the step in the formula above. It is what our code used.

Either way, steps two and three do not change: softmax into weights, then a weighted sum of the values.

Why it mattered

Attention fixed the bottleneck, and it showed something bigger: a model can reach any part of its input directly.

Wikipedia says the transformer later removed the slower sequential recurrent network and relied on the faster parallel attention scheme, which has largely replaced recurrent networks in frontier models since BERT, T5 and GPT. Its paper, Attention Is All You Need, came out in 2017.

So far the query came from one side and the keys from the other. What if the words of one sentence looked at each other instead? That is the next step.

Quick check

What is the main problem attention was created to solve in translation models?

Next: self-attention, where every word looks at every other word.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.