AI engineering / Beginner / 01.29
Embeddings for builders
How an embedding model turns text into a list of numbers so similar meanings sit close together, and how to build with that.
13 min readFree
Numbers that capture meaning
The last lesson, Context Evaluation, ended with a promise: next, embeddings, turning text into numbers that capture meaning. Here they are.
Think of a librarian. You ask for "something about cancelling a subscription". She does not hunt for that word on every cover. She knows what the books are about. An embedding gives your program a little of that skill.
This lesson is about using embeddings: what goes in, what comes out, how to compare two results and what to store. The Embeddings lesson in Machine Learning Foundations explains how a model learns them. You will run two small Python programs. Neither needs a key, a network or money.
Key words
- Embedding
- A list of numbers that stands for a piece of text, so that texts with similar meaning get similar lists.
- Vector
- Just a list of numbers in a fixed order, like [3, 0, 1]. An embedding is a vector.
- Embedding model
- The program that reads text and gives back its vector.
- Dimensions
- How many numbers are in the vector. A vector with 384 numbers has 384 dimensions.
- Cosine similarity
- A score that says how much two vectors point the same way. Higher means more alike.
Text in, numbers out
You hand an embedding model a piece of text. It hands back a vector. That is the whole job.
- 1
Send text
A word, a sentence or a paragraph, such as "reset your password".
- 2
Get a vector
The model returns a list of numbers.
- 3
Compare vectors
You never read the numbers yourself. You only measure how close two vectors are.
What are embeddings
The Hugging Face blog gives a plain definition: an embedding is "a numerical representation of a piece of information, for example, text, documents, images, audio, etc."
The OpenAI guide says it in builder's terms. An embedding is "a vector (list) of floating point numbers". A floating point number is just a number with a decimal point, like 0.84. It adds: "Small distances suggest high relatedness and large distances suggest low relatedness."
Picture a map of a town. Cafes cluster in one area and banks in another. Each place has two numbers, left-right and up-down, and alike places sit near each other. An embedding is a map with far more than two directions. The rule is the same: close means alike.
How long is the list? It depends on the model. The Hugging Face blog shows an example with 384 numbers. The OpenAI guide says that by default the length is 1536 for its text-embedding-3-small model and 3072 for text-embedding-3-large, and that a dimensions setting can make the list shorter. Read the number for your own model in its docs, because it sets how much you store.
Why not just count words? The Google crash course says one-hot lists, which give each item its own slot, fail to capture "semantic similarities between items". Its example: hot dogs and shawarmas are more alike than hot dogs and salads, yet a one-hot list cannot say so. The Machine Learning Foundations lesson goes deeper.
Comparing two vectors
You need one number that says how alike two vectors are. The usual choice is cosine similarity. The OpenAI guide says "We recommend cosine similarity".
Picture two arrows from the same point. Same direction means alike. Cosine similarity measures the angle between them. Wikipedia describes it as the dot product of the vectors divided by the product of their lengths.
In words. Multiply each pair of matching numbers and add them up: that is the dot product A · B. A list's length is the square root of the sum of its squares. Divide the dot product by the two lengths multiplied together.
A tiny case. Take A = [1, 1, 0] and B = [1, 0, 1]. The pairs give 1, 0 and 0, so the dot product is 1. Each list has length 1.414 (the square root of 2). The score is 1 divided by 1.414 times 1.414, which is 0.5.
Wikipedia gives the range: always between -1 and +1. Same direction scores +1, a right angle scores 0, opposite directions score -1. It depends only on the angle, not on how long the vectors are. For word counts, which cannot be negative, it stays between 0 and 1.
A toy you can run
Let us build the smallest thing that works like an embedding. It is a toy, not a real embedding model. A real model captures meaning. Our toy only captures shared words: it counts how often each word of a fixed word list appears in the text.
There is no model call, so nothing to pay for. The code keeps 5 short help notes. The word list is every word in those notes, 36 words, so each vector has 36 numbers. We write cosine similarity out by hand and rank the notes against two queries, scores rounded to 3 places.
import math
notes = [
"reset your password from the login page",
"invoices are emailed on the first day of each month",
"cancel your plan any time from billing settings",
"our team answers support emails within two days",
"export your data as a csv file from settings",
]
# The vocabulary is fixed: every word that appears in the notes.
vocab = sorted({word for note in notes for word in note.split()})
def toy_embed(text):
# One slot per vocabulary word. Each slot counts how often that word appears.
words = text.lower().split()
return [words.count(word) for word in vocab]
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
len_a = math.sqrt(sum(x * x for x in a))
len_b = math.sqrt(sum(y * y for y in b))
if len_a == 0 or len_b == 0:
return 0.0
return dot / (len_a * len_b)
def rank(query):
q = toy_embed(query)
scored = [(round(cosine(q, toy_embed(n)), 3), n) for n in notes]
return sorted(scored, reverse=True)
print(len(vocab), "slots per vector")
for query in ["reset my password", "change my secret code for logging in"]:
print()
print("Query:", query)
for score, note in rank(query):
print(score, note)Output
36 slots per vector Query: reset my password 0.535 reset your password from the login page 0.0 our team answers support emails within two days 0.0 invoices are emailed on the first day of each month 0.0 export your data as a csv file from settings 0.0 cancel your plan any time from billing settings Query: change my secret code for logging in 0.0 reset your password from the login page 0.0 our team answers support emails within two days 0.0 invoices are emailed on the first day of each month 0.0 export your data as a csv file from settings 0.0 cancel your plan any time from billing settings
"reset my password" scored 0.535 against the password note and 0.0 against the rest. Check the sum. The query has 2 words in the word list, so its length is the square root of 2. The note has 7 different words, so its length is the square root of 7. They share 2 words, so the dot product is 2. Then 2 divided by 1.414 times 2.646 gives 0.535.
The right note wins. Now the second query.
Where the toy breaks
The second query, "change my secret code for logging in", means the same thing as the first. A person would send you to the password note. The toy gave every note 0.0.
Why? None of the query's words is in the word list, and "login" is not "logging" to a program that matches letters. The toy cannot know that "secret code" and "password" mean nearly the same. It sees words as separate boxes, the same weakness the Google page names for one-hot lists.
Wikipedia describes the goal of a real model: words that are closer in the vector space are expected to be similar in meaning. With one, "secret code" and "password" should score high even though they share no word. That is the reason to use a real model.
Here is how a real call looks, from the OpenAI docs. It needs a key, goes over the network and costs money for every token. It is shown, not run.
# From the OpenAI Vector embeddings guide, shown without its output (not run here)
from openai import OpenAI
client = OpenAI()
response = client.embeddings.create(
input="Your text string goes here",
model="text-embedding-3-small"
)
print(response.data[0].embedding)The Tokens lesson explains what a token is. The guide says you can "save it in a vector database", which is the next lesson.
Quick check
Why did the toy score 0.0 for "change my secret code for logging in"?
Putting embeddings to work
Now the practical side: what are embeddings for, and how do you embed your own data?
Purpose and functionality
The OpenAI guide names six common uses. Here are the first three.
Search by meaning
Embed the query and each document, then rank by similarity. A real model does it by meaning, so "secret code" can find "password".
Grouping
The guide calls it clustering. Think of sorting a pile of support emails into piles by topic, without being told the topics.
Recommendations
Items with related text are recommended. If a reader liked one article, show the articles whose vectors sit nearby.
The other three. Anomaly detection finds outliers, items with little relatedness to the rest, like one odd message in a pile of normal ones. Classification labels a text by its most similar label: embed the labels too and pick the closest. Diversity measurement looks at how similarity scores are spread out.
The Hugging Face blog quotes Dale Markowitz of Google Cloud calling embeddings an "ML multitool", and says you do not need a huge labelled dataset to use them.
Quick check
You embed 500 support emails and want piles of similar topics, without naming the topics first. Which use is that?
Embedding your data
Embedding your data means running every text through the model once and keeping the results. The Hugging Face blog says this lets algorithms "search quickly, sort, group, and more", though it can be "expensive and technically complicated". Five habits help.
- 1
Batch it
Send many texts in one call, not one call per text. The OpenAI examples pass a list of texts as input.
- 2
Store three things plus one
Keep an id, the text and the vector, plus the name of the model that made it.
- 3
Embed the query with the same model
The OpenAI code search example embeds the query "using the same model" as the documents.
- 4
Re-embed when the model changes
A new model gives new vectors. Run every text through it again.
- 5
Never mix models
Compare vectors only when the same model made both.
The id
Links the vector back to your own database row, like a library card number.
The text
Shows the result with no second lookup, and lets you re-embed later.
The vector
The numbers you compare. The only part the score uses.
Why never mix models? Each model decides for itself what each slot means. Slot 7 in one model has nothing to do with slot 7 in another, and the lists may differ in length. It is like comparing Celsius with Fahrenheit with no label on either. You would get a number, and it would mean nothing.
The next program shows these habits. Its fake_embed_batch is a stand-in so the code runs free, with no key or network. It is not a real model. A real embedding service is slower, costs money for every token, and can fail or refuse. The code saves the model name with every row and refuses to search if a row came from a different model.
import math
notes = {
"n1": "reset your password from the login page",
"n2": "invoices are emailed on the first day of each month",
"n3": "cancel your plan any time from billing settings",
"n4": "our team answers support emails within two days",
"n5": "export your data as a csv file from settings",
}
# A stand-in for an embedding service, so this code runs free.
# A real one is slower, costs money for every token, and can fail.
def fake_embed_batch(model, texts):
vocab = ["password", "login", "invoices", "emails", "plan", "settings"]
print(" call: model", model, "items", len(texts))
return [[t.split().count(w) for w in vocab] for t in texts]
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
la = math.sqrt(sum(x * x for x in a))
lb = math.sqrt(sum(y * y for y in b))
return dot / (la * lb) if la and lb else 0.0
def embed_all(model, items, batch_size=2):
store = []
ids = list(items)
for start in range(0, len(ids), batch_size):
chunk = ids[start:start + batch_size]
vectors = fake_embed_batch(model, [items[i] for i in chunk])
for i, vec in zip(chunk, vectors):
store.append({"id": i, "text": items[i], "vector": vec, "model": model})
return store
def search(store, model, query):
if any(row["model"] != model for row in store):
raise ValueError("stored vectors come from a different model: re-embed first")
q = fake_embed_batch(model, [query])[0]
return max(store, key=lambda row: cosine(q, row["vector"]))["id"]
store = embed_all("toy-v1", notes)
print(len(store), "rows saved")
print("best match:", search(store, "toy-v1", "reset password"))
print("now the model changes:")
try:
search(store, "toy-v2", "reset password")
except ValueError as error:
print("stopped:", error)Output
call: model toy-v1 items 2 call: model toy-v1 items 2 call: model toy-v1 items 1 5 rows saved call: model toy-v1 items 1 best match: n1 now the model changes: stopped: stored vectors come from a different model: re-embed first
The 5 notes went out in 3 calls (2, 2 and 1 items), not 5. Each row saved the model name "toy-v1". The search for "reset password" returned n1, the password note. Then we pretended the model changed to "toy-v2", and the search stopped with a clear message instead of returning bad results.
Quick check
Your provider releases a newer embedding model. What should you do?
Dimensions and cost
Two things grow with your data: the bill and the storage.
The bill. The OpenAI guide says requests are "billed based on the number of tokens in the input". Long texts cost more, and every re-embed costs again. Prices change, so this lesson quotes none. Here is a sum with a made-up price: if a million tokens cost 2 cents, and 10,000 notes of 50 tokens each make 500,000 tokens, the bill is 1 cent. Check your provider's real prices.
The space. Each note stores one list. At the 1536 default for text-embedding-3-small, 10,000 notes mean 15,360,000 numbers. The guide says the dimensions setting shortens the list "without losing its concept-representing properties". Test that on your own data before you trust it.
Tip
Write down the model name, its dimensions and the date you embedded, next to your data. You will need them when a model changes.
Recap
What it is
An embedding model turns text into a vector. Similar meanings land close together, and you compare vectors with cosine similarity.
What it is for
Search by meaning, grouping, recommendations, spotting odd items and classification.
How to use it
Batch your data, store id, text, vector and model name, re-embed when the model changes, and never mix vectors from different models.
Next: vector databases, storing embeddings so you can search them fast.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.