esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.48

Embeddings

How words become lists of numbers where similar meanings sit close together, and how a computer measures that closeness.

14 min readFree

Words as numbers that carry meaning

The last lesson, Lemmatization, ended with a promise: next, embeddings, turning words into numbers that carry meaning. Here they are.

A computer cannot read the word cat. It can only work with numbers. So every language model has to answer one question first: which numbers should stand for each word?

Think of a map. Each city gets two numbers, its latitude and longitude. Cities that are close in the real world get close numbers, so you can tell that two cities are neighbours just by looking at the numbers. An embedding is a map for words. Each word gets a list of numbers, and words with similar meanings get lists that sit near each other.

Key words

Embedding
A list of numbers that stands for something, such as a word, so that similar things get similar lists.
Vector
Just a list of numbers, in a fixed order. Think of the two numbers of a map position, but longer.
Dimension
One slot in the list. A vector with 4 numbers has 4 dimensions.
One-hot vector
A long list of zeros with a single 1 in the slot that belongs to the word.
Dense vector
A short list where most slots hold real, in-between numbers, not zeros.
Cosine similarity
A score from -1 to 1 that tells how much two vectors point the same way.
Static embedding
One fixed vector per word, whatever sentence the word is in.
Contextual embedding
A vector built fresh for each use of a word, so it depends on the sentence around it.

The simplest choice: one-hot vectors

Start with the easy idea. Make a list of every word you know. Give each word a slot. To write down the word dog, put a 1 in its slot and a 0 everywhere else. That is a one-hot vector: one slot is hot, the rest are cold.

Here is the idea with just 8 words. Each word gets a list of 8 numbers.

Python
words = ["cat", "dog", "apple", "banana", "king", "queen", "man", "woman"]

def one_hot(word):
    return [1 if w == word else 0 for w in words]

def overlap(a, b):
    return sum(x * y for x, y in zip(a, b))

for w in ["cat", "dog", "apple"]:
    print(w, one_hot(w))

print("cat and dog overlap:", overlap(one_hot("cat"), one_hot("dog")))
print("cat and apple overlap:", overlap(one_hot("cat"), one_hot("apple")))
print("cat and cat overlap:", overlap(one_hot("cat"), one_hot("cat")))

Output

cat [1, 0, 0, 0, 0, 0, 0, 0]
dog [0, 1, 0, 0, 0, 0, 0, 0]
apple [0, 0, 1, 0, 0, 0, 0, 0]
cat and dog overlap: 0
cat and apple overlap: 0
cat and cat overlap: 1

Look at the last three lines. The overlap is found by multiplying slot by slot and adding up. Cat and dog overlap by 0. Cat and apple also overlap by 0. Only a word with itself scores 1.

That is the problem. In a one-hot world, cat is exactly as far from dog as it is from apple. The numbers say nothing about meaning. They are just name tags, like seat numbers in a cinema: seat 12 and seat 13 tell you nothing about the people sitting there.

There is a second problem, size. Each vector is as long as the whole vocabulary, so a real list of many thousands of words means vectors with many thousands of slots, almost all zeros. Wikipedia describes the same trouble for the simplest word-and-context models: they give a very sparse vector space of high dimensionality. Sparse means mostly empty.

Dense vectors: similar words sit close

The fix is to give each word a short list of real numbers instead, say 4 or 300 of them, with no zeros needed. These are dense vectors, and they are the embeddings.

Wikipedia puts it this way: typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning.

Picture a school hall at lunch. Friends sit at the same table, and people who barely know each other sit far apart. If you were told only where everyone sits, you could still guess who is friends with whom. Dense vectors work the same way. The position carries the meaning.

Measuring closeness: cosine similarity

If words are points, how do we say whether two are close? One popular way is cosine similarity. Picture each vector as an arrow from the centre of a map. Two arrows that point the same way are similar. Two arrows that point at right angles have nothing in common. Two arrows that point opposite ways are opposites.

Wikipedia defines it as the cosine of the angle between the vectors, which is the dot product of the vectors divided by the product of their lengths. The dot product is simple: multiply the two lists slot by slot, then add everything up.

cos⁡(θ)=A⋅B∥A∥ ∥B∥,A⋅B=∑i=1naibi\cos(\theta) = \frac{A \cdot B}{\lVert A \rVert \, \lVert B \rVert}, \qquad A \cdot B = \sum_{i=1}^{n} a_i b_i

In words: take two vectors A and B. Multiply them slot by slot and add up the results. That is the top of the fraction. Then find the length of each vector, which is the square root of its own dot product, and multiply the two lengths. That is the bottom. Top divided by bottom is the score.

Wikipedia notes that the value always falls between -1 and +1. Two proportional vectors score +1, two orthogonal vectors (at right angles) score 0, and two opposite vectors score -1. It also notes that the score does not depend on the lengths of the vectors, only on their angle. So a long arrow and a short arrow pointing the same way are a perfect match.

Where the idea comes from

How can a computer learn good numbers for words when nobody tells it what words mean? It uses a clue that linguists spotted long ago. Wikipedia says the idea that "a word is characterized by the company it keeps" was proposed in a 1957 article by John Rupert Firth.

Think about it with a made-up word. Suppose you read: "The glorp barked and chased the ball." You have never seen glorp, yet you picture a dog. The words around it told you. Words used in the same kinds of places, with the same neighbours, tend to mean similar things. Cat and dog both appear near feed, pet, and sleep. Cat and tax rarely do.

word2vec

The method that made this popular is word2vec. Wikipedia says it was developed by Tomas Mikolov, Kai Chen, Greg Corrado, Ilya Sutskever and Jeff Dean at Google, and presented at ICLR in 2013. The first paper, on arXiv as 1301.3781, has a plain goal in its title: Efficient Estimation of Word Representations in Vector Space.

The abstract of that paper makes a claim about speed. In the authors' words: "it takes less than a day to learn high quality word vectors from a 1.6 billion words data set". That mattered, because earlier methods based on other kinds of neural networks cost much more to train.

Wikipedia describes the models as shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. They take in a large body of text and give every unique word a vector, typically of several hundred dimensions. It names two designs.

  • Continuous bag of words

    Wikipedia calls this a fill in the blank task. The model sees the words around a gap and has to guess the missing word.

  • Skip-gram

    The opposite game. The model sees one word and has to guess the words that tend to sit around it.

  • Same result

    Either way, words that fit the same contexts end up with similar vectors. Wikipedia adds that why this works so well is still poorly understood.

Quick check

Why does a model trained on plain text, with no labels, end up with similar vectors for cat and dog?

Vector arithmetic: a word puzzle

Here is the famous surprise. Wikipedia reports that Mikolov et al. (2013) found semantic and syntactic patterns can be reproduced using vector arithmetic. Its example: the pattern "Man is to Woman as Brother is to Sister" can be produced by taking the vector for Brother, subtracting the vector for Man and adding the vector for Woman. The result is closest to the vector for Sister in the model.

Read it like a recipe on a map. Start at Brother. Walk the same way and distance that takes you from Man to Woman. You land near Sister. The article says such relations exist for country and capital, and for present tense and past tense, too.

Build a tiny embedding yourself

Now try it by hand. You will make up 4 numbers for each of 8 words, then measure closeness, find neighbours, and run a word puzzle. The vectors in this demo were hand-made by me so the idea is easy to see. They were not learned from any text, and they do not come from a real model. A real model such as word2vec learns hundreds of numbers per word, and the patterns are much messier.

  1. 1

    Write the vectors

    Give each word 4 numbers. Here the first slot loosely means royal, the second male versus female, the third animal, the fourth food. Real models have no such labels, but these help us read the output.

  2. 2

    Write the cosine function

    Dot product on top, the two lengths multiplied on the bottom, exactly as in the formula above.

  3. 3

    Compare some pairs

    Print the cosine for a few pairs, some close, some far apart.

  4. 4

    Find the nearest words

    For one word, score it against all the others and sort. The best score is the nearest neighbour.

  5. 5

    Do the puzzle

    Take king, subtract man, add woman, and see which word is nearest to the result.

Python
import math

# Hand-made vectors for this demo. Real ones are learned from text.
emb = {
    "cat":    [0.0, 0.0, 0.9, 0.1],
    "dog":    [0.0, 0.1, 0.9, 0.0],
    "apple":  [0.0, 0.0, 0.1, 0.9],
    "banana": [0.0, 0.0, 0.0, 0.9],
    "king":   [0.9, 0.5, 0.0, 0.0],
    "queen":  [0.9, -0.5, 0.0, 0.0],
    "man":    [0.1, 0.5, 0.1, 0.0],
    "woman":  [0.1, -0.5, 0.1, 0.0],
}

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def cosine(a, b):
    return dot(a, b) / (math.sqrt(dot(a, a)) * math.sqrt(dot(b, b)))

pairs = [("cat", "dog"), ("apple", "banana"), ("cat", "banana"),
         ("king", "queen"), ("king", "man"), ("dog", "queen")]
for a, b in pairs:
    print(f"{a} vs {b}: {cosine(emb[a], emb[b]):.3f}")

print()
def nearest(vec, skip, n=3):
    scored = [(cosine(vec, v), w) for w, v in emb.items() if w not in skip]
    scored.sort(reverse=True)
    return [(w, round(s, 3)) for s, w in scored[:n]]

for w in ["cat", "king", "banana"]:
    print("closest to", w, nearest(emb[w], {w}))

print()
target = [k - m + w for k, m, w in zip(emb["king"], emb["man"], emb["woman"])]
print("king - man + woman =", [round(t, 2) for t in target])
print("closest:", nearest(target, {"king", "man", "woman"}))

Output

cat vs dog: 0.988
apple vs banana: 0.994
cat vs banana: 0.110
king vs queen: 0.528
king vs man: 0.636
dog vs queen: -0.054

closest to cat [('dog', 0.988), ('apple', 0.22), ('woman', 0.191)]
closest to king [('man', 0.636), ('queen', 0.528), ('dog', 0.054)]
closest to banana [('apple', 0.994), ('cat', 0.11), ('woman', 0.0)]

king - man + woman = [0.9, -0.5, 0.0, 0.0]
closest: [('queen', 1.0), ('cat', 0.0), ('banana', 0.0)]
  • Pairs

    Cat and dog score 0.988, and apple and banana score 0.994, almost the top score of 1. Cat and banana score only 0.110. Dog and queen score -0.054, a hair below zero: no link at all. The score ranks closeness the way we hoped.

  • Neighbours

    The closest word to cat is dog (0.988). The closest to banana is apple (0.994). Notice the second place for cat: apple at 0.22, then woman at 0.191. Those are weak matches, which the small scores show.

  • The puzzle

    King minus man plus woman gives [0.9, -0.5, 0.0, 0.0], which is exactly the vector for queen, so queen comes first at 1.0. Be honest about why. I built the vectors so the male-female slot flips sign and the royal slot stays put. The demo shows how the arithmetic works. It proves nothing about real models.

Quick check

In the demo, cat vs dog scored 0.988 and cat vs banana scored 0.110. What does that tell you?

Same word, different vector

Static embeddings have one weakness. Each word gets a single vector, whatever sentence it sits in. Wikipedia points out that words with several meanings are squeezed into one vector. It gives the sentence "The club I tried yesterday was great!" and asks: is that a club sandwich, a clubhouse, or a golf club? One vector cannot tell.

The fix came with newer models. Wikipedia says that as of the late 2010s, contextually-meaningful embeddings such as ELMo and BERT have been developed. Unlike static word embeddings, these embeddings are at the token level: each occurrence of a word has its own embedding. It adds that occurrences of a word in similar contexts are situated in similar regions of BERT's embedding space.

Think of a name tag that is rewritten for every party. At a golf day, club gets one tag. At a sandwich shop, the same word gets another. The Transformers lesson showed the machinery: attention lets each word look at the words around it, so its final vector is shaped by the sentence. That is why transformers can give the same word different vectors in different places.

  • Static

    One vector per word. Fast and simple. Cannot tell a golf club from a club sandwich.

  • Contextual

    A new vector for each use, shaped by the sentence around it. Handles words with many meanings much better.

Sentences, documents and search

Words are only the start. Wikipedia says the idea has been extended to embeddings of entire sentences or even documents. One popular way it names is Sentence-BERT, which modifies a pre-trained BERT model.

Once a whole question and a whole page each have a vector, you can score them with cosine similarity. The Cosine similarity article gives the document use directly: each word gets a coordinate, a document is a vector of its word counts, and the score gives a useful measure of how similar two documents are likely to be in subject matter, whatever their length.

That is the heart of semantic search. Instead of matching exact words, the system turns your question into a vector and returns the pages whose vectors sit nearest. A search for cheap flights to Rome can then find a page about low-cost trips to Italy, even though they share almost no words. Wikipedia notes that word2vec embeddings can serve as inputs to systems for search and classification.

The AI engineering course covers retrieval-augmented generation (RAG), which uses this idea to hand a chat model the right pages before it answers.

A warning: embeddings copy our bias

Embeddings learn from human writing, and human writing is not always fair. Wikipedia says word embeddings may contain the biases and stereotypes in the data they were trained on. It points to a 2016 paper by Bolukbasi et al., which found that a popular word2vec embedding trained on Google News texts still showed disproportionate word associations reflecting gender and racial biases. One analogy it produced was "man is to computer programmer as woman is to homemaker".

What to remember

An embedding is a list of numbers that stands for a word, placed so that words with similar meanings sit near each other. One-hot vectors are the simple start, but they say nothing about meaning and waste space. Dense vectors fix both problems.

Cosine similarity measures how much two vectors point the same way, from -1 to 1. The idea that a word is known by the company it keeps led to word2vec in 2013, which learns vectors from plain text. Contextual embeddings, such as those from transformers, give a word a new vector for each use, and sentence embeddings extend the idea to whole texts and to search. And because they learn from human writing, they can copy human bias.

Quick check

Which job is a contextual embedding better at than a static one?

Next: recommendation systems, suggesting what people might like.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.