esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.14

Performing Similarity Search

How to ask a vector store for the nearest pieces and trust the answer: the right distance, a sensible k, a minimum score, re-ranking and variety.

14 min readFree

Asking the store

The last lesson, Choosing a vector store, ended with a promise: next, performing similarity search, asking the store for the nearest pieces. Here it is.

You have a store full of pieces, each saved as a list of numbers, and a user has typed a question. Getting the few pieces that answer it takes a handful of small choices, and each one changes what the user sees.

Think of a librarian. A good one brings a few books, skips the ones that are not really about your topic, and does not hand you five copies of the same book. Similarity search is that job done by code.

Key words

Query
The question or search text the user typed.
Query vector
The query turned into a list of numbers by the same embedding model that made the stored vectors.
Similarity score
One number that says how close two vectors are. Higher can mean closer, depending on the metric.
k
How many results you ask for. "Top-k" means the k best-scoring pieces.
Filter
A rule on the saved details of a piece (its language, date or owner) that must be true before it can come back.
Threshold
A minimum score. A piece below it is not returned, even if it is the best one there is.
Re-ranking
A second, slower pass that puts the first results in a better order.
MMR
Maximal marginal relevance: a way to pick results that are relevant and also different from each other.
Recall@k
Of the pieces a person marked as right, the share that show up in your top k.

You will run three small Python programs. None needs a key, a network or money. The vectors are made by hand with three numbers each, so you can check every sum. Real embeddings are much longer, but the steps are the same. The Embeddings lesson in Machine Learning Foundations explains where the numbers come from.

Embed the query with the same model

This rule is easy to break. The stored pieces were turned into vectors by one embedding model. The query must be turned into a vector by that same model, with the same settings.

Why? Each model draws its own map. Comparing a point from one map with a point from another is like giving directions in miles to a friend who reads kilometres. The numbers look fine, but they mean nothing. If you switch models or settings, embed every stored piece again, and save the model name next to the store.

Choose the metric

A metric is the rule for "how close are these two vectors?". Three are common: cosine, dot product and Euclidean distance. Your store asks you to pick one, and your embedding model's docs say which one it expects. Match it.

  • Cosine similarity

    Compares the direction two vectors point in and ignores how long they are. Wikipedia calls it most appropriate where frequency matters more than absolute values, such as word counts in documents.

  • Dot product

    Multiplies the numbers pair by pair and adds them up. Longer vectors score higher, so length counts.

  • Euclidean distance

    The straight-line gap between the two points. Smaller means closer, which is the opposite of the other two.

Here is what Wikipedia says about each. On cosine: "Cosine similarity is the cosine of the angle between the vectors; that is, it is the dot product of the vectors divided by the product of their lengths." It adds that "the cosine similarity does not depend on the magnitudes of the vectors, but only on their angle." For the other one, its page on Euclidean distance says the distance between two points "is the length of the line segment between them."

The cosine page also ties the metrics together. You can scale each vector to unit length (L2 normalisation) and then use ordinary Euclidean distance, which gives "the same ordering as the cosine distance". At length 1, the dot product also equals the cosine, because the lengths you would divide by are both 1.

So: if every vector has length 1, the three metrics rank results in the same order. If not, they can disagree. Check your vectors, do not assume.

Tip

Higher is better for cosine and dot product. Lower is better for Euclidean distance. Mixing up the two sorts your results upside down.

See the three metrics agree

Here is a tiny store of four pieces. The three numbers stand for (refunds, shipping, recipes). They are made up by hand, not made by a real model. One piece, "refund FAQ", has much bigger numbers than the rest, as if it were a longer text. The query is "how do I get my money back?". The program ranks the pieces with each metric, first on the raw vectors, then after scaling every vector to length 1. Its third step, a minimum score, comes back later in this lesson.

Python
import math

def dot(a, b): return sum(x * y for x, y in zip(a, b))
def length(a): return math.sqrt(dot(a, a))
def cosine(a, b): return dot(a, b) / (length(a) * length(b))
def euclid(a, b): return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))
def unit(a): return [x / length(a) for x in a]

# Hand-made 3-number vectors. Think of them as (refunds, shipping, recipes).
docs = {
    "refund rules": [0.9, 0.1, 0.0],
    "refund FAQ": [3.6, 0.8, 0.2],
    "shipping times": [0.1, 0.9, 0.0],
    "pasta recipe": [0.0, 0.1, 0.9],
}
query = [0.8, 0.2, 0.0]  # "how do I get my money back?"

def show(title, scores, high_is_best=True):
    ranked = sorted(scores, key=scores.get, reverse=high_is_best)
    print(title, "->", ", ".join(f"{n} {scores[n]:.2f}" for n in ranked))

print("Step 1: raw vectors")
show("cosine   ", {n: cosine(query, v) for n, v in docs.items()})
show("dot      ", {n: dot(query, v) for n, v in docs.items()})
show("euclidean", {n: euclid(query, v) for n, v in docs.items()}, False)

print("Step 2: after scaling every vector to length 1")
q1 = unit(query)
d1 = {n: unit(v) for n, v in docs.items()}
show("cosine   ", {n: cosine(q1, v) for n, v in d1.items()})
show("dot      ", {n: dot(q1, v) for n, v in d1.items()})
show("euclidean", {n: euclid(q1, v) for n, v in d1.items()}, False)

print("Step 3: minimum score 0.75")
MIN_SCORE = 0.75
def search(q, k=3):
    scores = {n: cosine(q, v) for n, v in docs.items()}
    ranked = sorted(scores, key=scores.get, reverse=True)[:k]
    return [(n, scores[n]) for n in ranked if scores[n] >= MIN_SCORE]

# The second query points at nothing in particular, like a question about something we never stored.
for label, q in [("on topic ", query), ("off topic", [0.3, 0.3, 0.3])]:
    hits = search(q)
    print(label, "->", [(n, round(s, 2)) for n, s in hits] if hits else "nothing relevant found")

Output

Step 1: raw vectors
cosine    -> refund FAQ 1.00, refund rules 0.99, shipping times 0.35, pasta recipe 0.03
dot       -> refund FAQ 3.04, refund rules 0.74, shipping times 0.26, pasta recipe 0.02
euclidean -> refund rules 0.14, shipping times 0.99, pasta recipe 1.21, refund FAQ 2.87
Step 2: after scaling every vector to length 1
cosine    -> refund FAQ 1.00, refund rules 0.99, shipping times 0.35, pasta recipe 0.03
dot       -> refund FAQ 1.00, refund rules 0.99, shipping times 0.35, pasta recipe 0.03
euclidean -> refund FAQ 0.06, refund rules 0.13, shipping times 1.14, pasta recipe 1.40
Step 3: minimum score 0.75
on topic  -> [('refund FAQ', 1.0), ('refund rules', 0.99)]
off topic -> nothing relevant found

In step 1 the metrics disagree. Cosine puts "refund FAQ" and "refund rules" on top, both near 1.00. The dot product puts "refund FAQ" first by size alone, 3.04 against 0.74. Euclidean distance (lower is better) puts "refund rules" first and "refund FAQ" last at 2.87, because its big vector sits far away. In step 2, after scaling to length 1, all three agree: refund FAQ, refund rules, shipping times, recipe. That is the Wikipedia fact at work.

Pick k and apply filters

k is how many pieces you ask for. Too small and the right piece may be left out. Too large and you fill the model's context with noise, which costs money and can confuse the answer.

A filter is different. It is a hard rule, not a closeness score. "Only English pieces", "only this customer's files", "only pieces from this year". The store checks the saved details of each piece, and drops those that do not match.

Filters also protect people. If a store holds many customers' files, a customer-id filter stops one customer seeing another's text. Put it in the search call, every time.

Set a minimum score, and handle "nothing found"

A vector store always returns something. Ask for the top 3 and you get 3, even if the question is about cooking and the store holds only refund rules. They are just the three least-far pieces.

If you hand those pieces to a model, it may build a confident answer from text that has nothing to do with the question. So set a minimum score. If nothing passes it, say "nothing relevant found" and take a safer path: ask the user to rephrase, or let a human take over.

Where should the line go? Scores depend on the model and the metric, so there is no universal number. Collect a few questions that should find something and a few that should not, look at their scores, and put the line between the two groups.

The program in the metric section also ran this idea. Its step 3 asks for the top 3 with a minimum score of 0.75. The first query, the refund question, keeps two pieces. The second query, a vague one that points at nothing in particular, keeps none, so the program says so instead of guessing.

Tip

"Nothing found" is a good result. A search that can say it has no answer is far more trustworthy than one that always has one.

Re-rank with a stronger model

The first pass has to be fast, because it looks at the whole store. It compares two vectors that were made separately, so it cannot see how the query and the piece fit together.

A second pass can fix some of that. A cross-encoder is a model that reads the query and one piece together, in one go, and gives a relevance score. The Sentence Transformers page says it plainly: "A re-ranker based on a Cross-Encoder can substantially improve the final results for the user." It adds that "Scoring thousands or millions of (query, document)-pairs would be rather slow", so "we use the retriever to create a set of e.g. 100 possible candidates which are then re-ranked by the Cross-Encoder."

  1. 1

    Retrieve wide

    Ask the store for a larger k, for example 50 or 100 candidates.

  2. 2

    Re-rank

    Score each candidate against the query with the stronger model.

  3. 3

    Keep a few

    Keep the best 3 to 5 and pass those on.

A real re-ranker is a model: slower than vector search, possibly paid for every call, and it can still be wrong. Measure it on your own data before you keep it.

Variety with maximal marginal relevance

A help centre may hold five near-copies of one refund article. The top 5 by score are those copies, and the user sees one idea five times.

MMR fixes this. Qdrant's article describes it as reranking results by two things: "Relevance to your query" and "Diversity from already selected items". Elastic lists the steps: start with the most relevant item, score each remaining item on relevance and difference from the chosen ones, take the best, and repeat.

The Elastic article gives the formula: MMR score = λ × relevance − (1 − λ) × max_similarity_to_selected. The symbol λ (lambda) sets the balance. Qdrant says "λ = 1.0" is pure relevance, "λ = 0.5" is a balanced approach and "λ = 0.0" is pure diversity.

MMR(d)=λ⋅sim(d,q)−(1−λ)⋅max⁡s∈pickedsim(d,s)\text{MMR}(d) = \lambda \cdot \text{sim}(d, q) - (1 - \lambda) \cdot \max_{s \in \text{picked}} \text{sim}(d, s)

In words: reward a piece for matching the query, and take points off if it is very close to a piece you already picked. The top score goes next.

Below, six pieces are about refunds. Five are near-copies. One, "refund time limit", points a little differently. The program picks 3 with plain scoring, then with MMR at lambda 0.5, and prints every round.

Python
import math

def dot(a, b): return sum(x * y for x, y in zip(a, b))
def length(a): return math.sqrt(dot(a, a))
def cosine(a, b): return dot(a, b) / (length(a) * length(b))
def euclid(a, b): return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))
def unit(a): return [x / length(a) for x in a]

print("MMR, lambda 0.5, pick 3 of 6")
LAM = 0.5
pool = {
    "refund A": [0.95, 0.30, 0.00],
    "refund B": [0.94, 0.32, 0.00],
    "refund C": [0.96, 0.28, 0.00],
    "refund D": [0.93, 0.34, 0.00],
    "refund E": [0.97, 0.25, 0.00],
    "refund time limit": [0.60, 0.20, 0.50],
}
q = [1.0, 0.2, 0.2]
rel = {n: cosine(q, v) for n, v in pool.items()}
print("plain top 3:", sorted(rel, key=rel.get, reverse=True)[:3])
chosen = []
while len(chosen) < 3:
    best, best_score = None, -9
    round_scores = []
    for n in pool:
        if n in chosen: continue
        redundancy = max((cosine(pool[n], pool[c]) for c in chosen), default=0.0)
        score = LAM * rel[n] - (1 - LAM) * redundancy
        round_scores.append(f"{n} {score:.3f}")
        if score > best_score: best, best_score = n, score
    chosen.append(best)
    print(f"round {len(chosen)}:", ", ".join(round_scores), "-> picked", best)
print("MMR top 3:", chosen)

Output

MMR, lambda 0.5, pick 3 of 6
plain top 3: ['refund E', 'refund C', 'refund A']
round 1: refund A 0.488, refund B 0.486, refund C 0.489, refund D 0.485, refund E 0.490, refund time limit 0.442 -> picked refund E
round 2: refund A -0.012, refund B -0.012, refund C -0.011, refund D -0.013, refund time limit 0.050 -> picked refund time limit
round 3: refund A -0.012, refund B -0.012, refund C -0.011, refund D -0.013 -> picked refund C
MMR top 3: ['refund E', 'refund time limit', 'refund C']

Plain scoring picks refund E, refund C and refund A: three near-copies. MMR picks refund E first, because it scores highest. In round 2 every near-copy drops to about -0.01, since each is almost identical (redundancy near 1) to the piece already taken. "refund time limit" scores 0.050 and wins. In round 3 only near-copies are left, so it takes the least bad one; a bigger pool would give another different piece.

Measure with recall@k

How do you know k, the score line or a re-ranker is any good? You measure. Build a small labelled set: a list of real questions, and for each one, the pieces a person says are the right answers. Then run your search and count.

Recall@k asks: of the pieces marked right, how many appear in the top k? If a question has two right pieces and your top 3 holds one, recall@3 for that question is one half. Average over all the questions.

recall@k=right pieces found in the top kright pieces there are\text{recall@}k = \frac{\text{right pieces found in the top } k}{\text{right pieces there are}}

In words: found, divided by should-have-found. Here is a labelled set of three questions, with what the search returned (made up for this lesson), and the score for k = 1, 2 and 3.

Python
# For each test question: the pieces a person marked as right, and what the search returned (best first).
labelled = {
    "how do I get my money back?": ({"refund rules", "refund FAQ"}, ["refund FAQ", "shipping times", "refund rules"]),
    "when does my parcel arrive?": ({"shipping times"}, ["shipping times", "refund rules", "pasta recipe"]),
    "can I return a gift?": ({"refund rules", "gift returns"}, ["refund FAQ", "shipping times", "pasta recipe"]),
}

def recall_at_k(k):
    total = 0
    for right, returned in labelled.values():
        found = right & set(returned[:k])
        total += len(found) / len(right)
    return total / len(labelled)

for k in (1, 2, 3):
    print(f"recall@{k} = {recall_at_k(k):.2f}")

Output

recall@1 = 0.50
recall@2 = 0.50
recall@3 = 0.67

Recall@1 and recall@2 are both 0.50, then recall@3 rises to 0.67. The third question never finds its right pieces, whatever k is: "gift returns" is not returned at all, and "refund rules" is missing too. Raising k cannot fix that.

Change one thing at a time, rerun the same set, and keep what raises the number. Recall says whether the right piece was found, not whether the final answer was good.

Quick check

A user asks about something your store has no pieces on. What should the search do?

Quick check

When do cosine, dot product and Euclidean distance rank results in the same order?

Recap

Next: the retrieval process, from question to the right context.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.