esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.08

Choosing an embedding model

Hosted or open, how long a vector is, what it costs to store, and the few rules that keep your search from quietly breaking.

13 min readFree

Picking the model that makes the vectors

The last lesson, Hugging Face Tasks, ended with a promise: next, embedding models, choosing the model that turns text into vectors. Here it is.

An embedding model reads a piece of text and gives back a list of numbers. Texts with close meanings get lists that sit close together. Your whole search or answer system will stand on those lists, so the choice matters more than it looks. The Embeddings lesson in Machine Learning Foundations explains what an embedding is. This lesson is about choosing one for a product.

By the end you will know the two ways to get an embedding model, the five things worth comparing, how to work out storage cost on the back of an envelope, and three rules that stop silent bugs. You will run three small programs. None of them calls a real model.

Key words

Embedding model
A model whose only job is to turn text into a vector, a list of numbers.
Vector size (dimensions)
How many numbers are in each list. A vector of size 768 has 768 numbers.
Chunk
A small piece of a document, such as a paragraph, that you embed on its own.
Hosted model
A model that runs on someone else's computers. You send text over the internet and get the vector back.
Open model
A model whose files you can download and run on your own computer or server.
Normalise
Scale a vector so its length becomes exactly 1, without changing the way it points.
Re-embed
Run all your text through an embedding model again to make fresh vectors.

Two ways to get an embedding model

Think of buying bread. You can walk to a bakery and pay for each loaf, or you can bake at home. The first is quick to start and you pay as you go. The second takes some setup, but the ingredients are yours.

  • Hosted API

    You send text to a company and get vectors back. Nothing to install. OpenAI's guide says requests are "billed based on the number of tokens in the input", so every call has a cost, and your text leaves your system.

  • Open model you run

    You download the model and run it yourself. The Sentence Transformers docs say over 25,000 pre-trained models of this kind are available on Hugging Face. You pay for the computer, not per call, and you look after it.

  • A mix

    Many teams start with a hosted model to learn fast, then test an open one when the bill or the privacy rules start to matter. Both are fine choices. Test, then decide.

A token is a small piece of text. The Tokenization lesson in Machine Learning Foundations goes deeper.

Tip

Model facts in this lesson were checked on 16 November 2026. Model names, sizes and limits change often, so open the vendor's page again before you build.

What to compare

Pick by five plain questions, in this order.

  1. 1

    Does it speak your languages?

    If your users write in more than English, check this first. The Jina page says its jina-embeddings-v3 supports 89 languages. Google's docs say Gemini Embedding 2 works across over 100 languages. Each page lists its own numbers, so read the model's own page.

  2. 2

    How long can the input be?

    Every model has a maximum input length, counted in tokens. OpenAI's guide lists 8192 for its embedding models. Jina says jina-embeddings-v5-text-small supports up to 32,768 tokens and the nano model 8,192. Text longer than the limit has to be cut into chunks first.

  3. 3

    How big is the vector?

    Longer vectors can hold more detail, but they cost more to store and to compare. We work this out in a moment.

  4. 4

    How good is it on YOUR data?

    A public score is only a hint. Take 20 to 50 real questions from your users, write down which documents should answer each, and see which model finds them. This test is the one that counts.

  5. 5

    How fast is it?

    Time the model on your own text. A hosted model also adds network time. A model you run adds your own computer's speed, and a bigger model is slower.

Quick check

Your users write in German and Japanese. What do you check first?

What the vector size costs you

A vector is a list of decimal numbers. A common way to store each number takes 4 bytes. A byte is the smallest chunk of computer memory we count. Multiply the size of the vector by 4 and by the number of chunks, and you know how much space the vectors need.

bytes=chunks×numbers per vector×4\text{bytes} = \text{chunks} \times \text{numbers per vector} \times 4

In words: bytes equal the number of chunks, times the numbers in each vector, times four.

Here it is for one million chunks, at four vector sizes. A gigabyte (GB) here means one billion bytes. The sizes 384, 768, 1536 and 3072 are only examples to do maths with. The sums use no real model and no real prices.

Python
chunks = 1_000_000
bytes_per_number = 4

for size in [384, 768, 1536, 3072]:
    total_bytes = chunks * size * bytes_per_number
    print(size, "numbers per vector ->", total_bytes / 1_000_000_000, "GB")

Output

384 numbers per vector -> 1.536 GB
768 numbers per vector -> 3.072 GB
1536 numbers per vector -> 6.144 GB
3072 numbers per vector -> 12.288 GB

Each time the size doubles, the storage doubles too. Going from 384 to 3072 turns 1.536 GB into 12.288 GB, eight times as much. A database adds some extra on top, so treat these as a floor.

Tip

If you ever need a price in a sum, use a made-up number and label it as made up. Real prices change, so check the vendor's own page on the day you plan.

Shorter vectors

Some models let you ask for a shorter vector, so you can trade a little quality for a lot of storage. You do not need a second model to do it.

OpenAI's guide says its two newer embedding models can be shortened by passing in a `dimensions` parameter. By default the vector has 1536 numbers for text-embedding-3-small and 3072 for text-embedding-3-large. It also says that on the MTEB benchmark, a text-embedding-3-large embedding "can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536".

Google's docs say both gemini-embedding-001 and gemini-embedding-2 were trained with a technique called Matryoshka Representation Learning, so the first numbers of the vector still carry a useful version of the meaning. You set the size with `output_dimensionality`. The default is 3072, and Google recommends 768, 1536 or 3072. Think of nesting dolls: a small one sits inside the big one.

Normalise your vectors

A vector has a direction and a length. For comparing meaning, direction is what matters. Cosine similarity measures only direction, the angle between two vectors. The Embeddings for builders lesson shows it in use.

cos⁡(a,b)=a⋅b∥a∥ ∥b∥\cos(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}

In words: cosine is the dot product of a and b, divided by the length of a times the length of b. The dot product multiplies matching numbers and adds them up.

If every vector has length 1, the bottom of that fraction is 1 times 1, so the cosine is just the dot product. A dot product is simpler and faster, so many systems normalise first. To normalise, divide every number in the vector by the vector's length.

Here is a tiny check. The vectors below are made up by me, with 4 numbers each. They do not come from any model. Real ones have hundreds of numbers, but the rule is the same.

Python
import math

# A made-up vector with 4 numbers. Real ones have hundreds.
a = [3.0, 4.0, 0.0, 12.0]
b = [1.0, 2.0, 2.0, 2.0]

def length(v):
    return math.sqrt(sum(x * x for x in v))

def dot(u, v):
    return sum(x * y for x, y in zip(u, v))

def normalise(v):
    n = length(v)
    return [x / n for x in v]

print("length of a before:", length(a))
na = normalise(a)
nb = normalise(b)
print("length of a after:", round(length(na), 4))

cosine = dot(a, b) / (length(a) * length(b))
print("cosine of the raw vectors:", round(cosine, 4))
print("dot of the normalised vectors:", round(dot(na, nb), 4))

Output

length of a before: 13.0
length of a after: 1.0
cosine of the raw vectors: 0.7467
dot of the normalised vectors: 0.7467

The length of a was 13, and after normalising it is 1. The cosine of the raw vectors and the plain dot product of the normalised ones are both 0.7467. Same answer, simpler sum.

Do the models already do this for you? Some do. Google's docs say the default 3072-dimension Gemini embeddings are always normalised, and that Gemini Embedding 2 also normalises shorter ones. For gemini-embedding-001 you must normalise the shorter ones yourself. OpenAI's guide says that when you shorten an embedding by hand, you need to normalise it, and shows how. So read the docs for your model, and when in doubt, normalise. It does no harm.

Quick check

After you normalise two vectors, what does their dot product equal?

One model for documents and queries

This is the rule that breaks the most projects. A vector only makes sense inside the space of the model that made it. Think of two different maps of the same city. A pin at position (2, 5) on one map means nothing on the other.

So the question you type in and the chunks you stored must be embedded by the same model, with the same settings. OpenAI's code search example says it embeds the query "using the same model" as the documents. Google's docs add that if documents are embedded with a task prefix in the prompt, the query should follow the same format.

The code below shows a safe habit: store the model's name next to every vector, and refuse to compare when the names differ. The `fake_embed` function is a stand-in so the code runs free, with no key and no network. It returns fixed, made-up numbers and is not a real model. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse.

Python
# A stand-in, not a real model. It just makes fixed made-up vectors so the code runs free.
def fake_embed(text, model_name):
    base = {"model-a": [0.6, 0.8], "model-b": [0.8, 0.6]}[model_name]
    return {"model": model_name, "vector": base}

def similarity(x, y):
    if x["model"] != y["model"]:
        raise ValueError("different models: " + x["model"] + " vs " + y["model"])
    return sum(p * q for p, q in zip(x["vector"], y["vector"]))

doc = fake_embed("a stored chunk", "model-a")
query_ok = fake_embed("a question", "model-a")
query_bad = fake_embed("a question", "model-b")

print("same model:", round(similarity(doc, query_ok), 2))
try:
    similarity(doc, query_bad)
except ValueError as e:
    print("refused:", e)

Output

same model: 1.0
refused: different models: model-a vs model-b

If you switch models, re-embed everything

Moving to a new model, even from the same vendor, means every stored vector is now from a different space. Google's migration notes say it plainly: the embedding spaces of gemini-embedding-001 and gemini-embedding-2 are incompatible, so "you must re-embed all of your existing data".

That is why you keep the original text next to each vector. Without it you cannot re-embed.

  1. 1

    Build the new index beside the old one

    Do not delete anything yet. Search keeps working on the old vectors.

  2. 2

    Re-embed every chunk with the new model

    Store the new vectors with the new model's name.

  3. 3

    Test on your 20 to 50 real questions

    Compare the answers with the old index.

  4. 4

    Switch, then remove the old index

    Only when the new one is at least as good.

Four places to look

These show the range of choices. Each lists its facts on its own page, and nothing here ranks them.

  • OpenAI Embeddings API

    A hosted API. The guide describes an embedding as "a vector (list) of floating point numbers" and says the distance between two vectors measures how related they are.

  • Gemini Embedding

    A hosted API from Google. It supports text, images, video, audio and documents in one space, and works across over 100 languages.

  • Sentence Transformers

    A Python module for running embedding models yourself. The docs call it the go-to module for using and training embedding and reranker models.

  • Jina

    Has a hosted embeddings API. Its page lists models with different sizes and says context length varies by model.

Below are short snippets copied from those docs. They are shown without their output and were not run here, because they need a key or a download.

First, the OpenAI guide. You send your text and a model name, and read the vector from the answer.

Text
# From the OpenAI Vector embeddings guide, shown without its output (not run here)
from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    input="Your text string goes here", model="text-embedding-3-small"
)

print(response.data[0].embedding)

Second, the Gemini API docs. Same idea, different names.

Text
# From the Google Gemini API Embeddings docs, shown without its output (not run here)
from google import genai

client = genai.Client()

result = client.models.embed_content(
        model="gemini-embedding-2",
        contents="What is the meaning of life?"
)

print(result.embeddings)

Third, Sentence Transformers. This one runs on your own computer after you download the model, so there is no API key. The docs show the model producing vectors of size 1024 for this one.

Text
# From the Sentence Transformers documentation, shown without its output (not run here)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")

query = "Which planet is known as the Red Planet?"
documents = [
    "Jupiter is the largest planet in our solar system.",
    "Mars gets its reddish color from iron minerals in its soil.",
    "Photosynthesis lets plants turn sunlight into energy.",
]

query_embedding = model.encode_query(query)
document_embeddings = model.encode_document(documents)

In the last two lines, the query and the documents get their own methods, but one model does both. That is the same-model rule, built into the tool.

What about benchmark scores?

A benchmark is a shared test that many models take, so you can compare scores. The best known one for embeddings is MTEB, the Massive Text Embedding Benchmark. Its paper on arXiv (2210.07316) says it covers 8 embedding tasks over 58 datasets and 112 languages, and tests 33 models.

The paper's finding is a good warning: "no particular text embedding method dominates across all tasks". A model that wins at one job may be middling at another. A leaderboard can help you pick a few to try. It cannot replace trying them.

Quick check

A model tops a public leaderboard. What should you still do?

What to remember

  • Compare five things

    Languages, maximum input, vector size, quality on your own data, and speed. Test with 20 to 50 real questions.

  • Do the storage maths

    Chunks times numbers per vector times 4 bytes. One million chunks at 1536 numbers is 6.144 GB before database extras.

  • Three rules

    Normalise, use the same model for documents and queries, and re-embed everything if you switch.

Next: semantic search, finding by meaning instead of exact words.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.