AI engineering / Core / 02.09
Semantic Search
Find things by meaning, not by exact words. Keyword search, embedding search, why each one fails, and how to combine them.
14 min readFree
Finding by meaning
The last lesson, Embedding Models, ended with a promise: next, semantic search, finding by meaning instead of exact words. Here it is.
Picture a help page for an app. A customer types "I want my money back". The page that answers it is titled "Refund policy". The two share no words at all. A plain search box finds nothing. A search that understands meaning finds the page at once.
By the end you will know how both kinds of search work, how each one fails, how to mix them, and how to check that your search is any good. You will run one small Python program that does all of it, with no keys and no network.
Key words
- Keyword search
- Search that looks for the exact words you typed. Also called lexical search.
- Semantic search
- Search that looks for things with the same meaning, even when the words differ.
- Embedding
- A list of numbers that stands for the meaning of a piece of text. The Embedding Models lesson covers how a model makes one.
- Cosine similarity
- A score for how close two embeddings point. Near 1 means very similar. Near 0 means unrelated.
- BM25
- A well-known formula that scores how well a document matches the words in a query.
- Hybrid search
- Running keyword search and semantic search together and merging the two result lists.
- Rank
- A result's place in a list. Rank 1 is the top.
- Hit rate
- Out of your test questions, how many got the right answer back in the top few results.
Search by meaning
Wikipedia describes semantic search as "search with meaning", set against lexical search, which looks for literal matches of the query words or variants of them. It adds that modern systems use vector embeddings, which turn words, phrases or whole documents into numbers, so the engine can find results by meaning and not only by exact keywords.
The Embedding Models lesson showed how text becomes numbers. Search is the plain use of that idea. You do it in two moments.
- 1
Prepare once
Turn every document into an embedding and keep the numbers. Do this ahead of time, not on every search.
- 2
Embed the question
When someone searches, turn their query into an embedding with the same model.
- 3
Compare
Score the query against every document with cosine similarity.
- 4
Sort
Put the highest scores first and show the top few.
Two small cautions. The query and the documents must use the same embedding model, or the numbers do not line up. And with a huge pile of documents, comparing against every one is slow, so real systems keep embeddings in a vector database. The Vector DBs lesson covers that.
Search by words
Keyword search is older and simpler. It checks which documents contain your words. The best-known scoring formula is BM25. The name is short for "best matching", and it first came out of a search system called Okapi, built at City University in London in the 1980s and 1990s, according to Wikipedia.
Wikipedia calls BM25 a bag-of-words function. That means it counts the query words that show up in each document and ignores where they sit. Three ideas sit inside it, and each is plain common sense.
Rare words count more
A word that appears in almost every document tells you little. A word that appears in one document tells you a lot. This is the IDF part, short for inverse document frequency.
More mentions help, with a limit
A document that says the word three times is a better match than one that says it once. But the gain slows down. Ten mentions are not ten times better.
Long documents are marked down
A long page contains many words by chance. So its score is scaled by its length compared with the average length.
Here is the formula Wikipedia gives. The second line is how it computes the rarity part. In words: for each query word, multiply its rarity by how often it appears in the document, softened by the limit and the length scaling, then add the results for all query words.
Here f is how many times the word appears in the document, |D| is the document's length in words, avgdl is the average length across all documents, N is the number of documents, and n is how many of them contain the word. The two free settings are k1 and b. Wikipedia says they are usually chosen as k1 between 1.2 and 2.0 and b equal to 0.75. We use k1 = 1.5 and b = 0.75.
How each one fails
Neither method is better in every case. They fail in opposite directions.
Keywords miss synonyms
"Money back" and "refund" mean the same thing, but they share no letters. BM25 scores that page at zero.
Embeddings can blur exact things
A product code, a person's name or a number carries little meaning of its own. A model may treat ERR-4012 and ERR-4015 as nearly the same. Test this on your own data before you trust it.
Both can bury the answer
Keywords reward pages that repeat your words. Embeddings reward pages on a similar topic. Either can put a near miss above the real answer.
Try it yourself
The program below is a tiny help centre with 8 short documents. It has three parts. First, BM25 written out with the formula above. Second, a toy "meaning" score. Third, the fusion step you will meet in the next section. It also runs a small test at the end.
Be clear about the toy meaning score. It is not an embedding model. It fakes meaning with a hand-made map that puts words with similar meaning into a group, such as refund, money and back. A real model learns that from huge amounts of text, with no hand-made list. We use the fake so the code is free and runs anywhere. Words outside the map are ignored, which also mimics how a real model can blur rare codes.
A real embedding model is slower, costs money for every token it reads, can give a different answer from another model or version, and can fail. Nothing here calls one.
import math
import re
DOCS = {
"d1": "How to reset your password if you forgot it",
"d2": "Error ERR-4012 means your payment card was declined",
"d3": "Error ERR-4015 means the file is too large to upload",
"d4": "Refund policy: we return what you paid within 30 days",
"d5": "How to cancel your subscription at any time",
"d6": "Change your email address in account settings",
"d7": "Invoices and receipts are sent after every payment",
"d8": "Upload photos and files from your phone",
}
STOP = {"a", "an", "the", "to", "of", "in", "if", "it", "is", "are", "was", "my", "your",
"you", "for", "and", "from", "at", "i", "what", "does", "we", "how", "after"}
def words(text):
found = re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())
return [w for w in found if w not in STOP]
# Keyword search: BM25 with the standard formula, k1 = 1.5 and b = 0.75.
K1, B = 1.5, 0.75
tokens = {d: words(t) for d, t in DOCS.items()}
avgdl = sum(len(t) for t in tokens.values()) / len(tokens)
def bm25(query):
scores = {}
for d, toks in tokens.items():
total = 0.0
for q in set(words(query)):
n = sum(1 for t in tokens.values() if q in t)
idf = math.log((len(tokens) - n + 0.5) / (n + 0.5) + 1)
f = toks.count(q)
total += idf * f * (K1 + 1) / (f + K1 * (1 - B + B * len(toks) / avgdl))
scores[d] = total
return scores
# "Meaning" search: a hand-made synonym map. This FAKES meaning. A real
# embedding model learns it from huge amounts of text. Words not in the
# map are ignored, just as a real model blurs rare codes.
GROUPS = {
"REFUND": ["refund", "money", "back", "return", "paid", "reimburse"],
"PAY": ["payment", "card", "pay", "invoices", "receipts"],
"TROUBLE": ["error", "problem", "issue", "declined", "fail", "broken"],
"UPLOAD": ["upload", "photos", "files", "file", "attach"],
}
CONCEPT = {w: c for c, ws in GROUPS.items() for w in ws}
def vector(text):
counts = {}
for w in words(text):
if w in CONCEPT:
counts[CONCEPT[w]] = counts.get(CONCEPT[w], 0) + 1
return counts
def cosine(a, b):
dot = sum(a[k] * b.get(k, 0) for k in a)
na = math.sqrt(sum(v * v for v in a.values()))
nb = math.sqrt(sum(v * v for v in b.values()))
return dot / (na * nb) if na and nb else 0.0
def meaning(query):
q = vector(query)
return {d: cosine(q, vector(t)) for d, t in DOCS.items()}
def ranked(scores):
pairs = sorted(scores.items(), key=lambda p: (-p[1], p[0]))
return [d for d, s in pairs if s > 0]
# Reciprocal rank fusion with k = 60: add 1 / (60 + rank) for each list.
def rrf(*lists, k=60):
fused = {}
for lst in lists:
for rank, d in enumerate(lst, start=1):
fused[d] = fused.get(d, 0) + 1 / (k + rank)
return fused
def show(name, scores, order):
print(" ", name)
for d in order[:3]:
print(" ", d, round(scores[d], 4), DOCS[d])
for query in ["ERR-4015 problem", "I want my money back"]:
print("Query:", query)
kw, mn = bm25(query), meaning(query)
fused = rrf(ranked(kw), ranked(mn))
show("keywords (BM25):", kw, ranked(kw))
show("meaning (toy):", mn, ranked(mn))
show("fused (RRF):", fused, ranked(fused))
print()
# A small labelled set: each query with the one document that should come first.
LABELLED = [
("ERR-4015 problem", "d3"),
("ERR-4012 trouble", "d2"),
("I want my money back", "d4"),
("reimburse me", "d4"),
("forgot password", "d1"),
("pay problem", "d2"),
]
def hit_rate(search, k):
hits = sum(1 for q, want in LABELLED if want in search(q)[:k])
return hits / len(LABELLED)
def by_keywords(q): return ranked(bm25(q))
def by_meaning(q): return ranked(meaning(q))
def by_fusion(q): return ranked(rrf(by_keywords(q), by_meaning(q)))
for name, fn in [("keywords", by_keywords), ("meaning", by_meaning), ("fused", by_fusion)]:
print(f"{name:9} hit@1 = {hit_rate(fn, 1):.2f} hit@3 = {hit_rate(fn, 3):.2f}")Output
Query: ERR-4015 problem
keywords (BM25):
d3 1.5385 Error ERR-4015 means the file is too large to upload
meaning (toy):
d2 0.7071 Error ERR-4012 means your payment card was declined
d3 0.4472 Error ERR-4015 means the file is too large to upload
fused (RRF):
d3 0.0325 Error ERR-4015 means the file is too large to upload
d2 0.0164 Error ERR-4012 means your payment card was declined
Query: I want my money back
keywords (BM25):
meaning (toy):
d4 1.0 Refund policy: we return what you paid within 30 days
fused (RRF):
d4 0.0164 Refund policy: we return what you paid within 30 days
keywords hit@1 = 0.50 hit@3 = 0.50
meaning hit@1 = 0.50 hit@3 = 0.67
fused hit@1 = 1.00 hit@3 = 1.00Reading the output
Start with the first query, ERR-4015 problem. BM25 returns only document d3, the one that contains the code ERR-4015. The word "problem" appears nowhere, so it adds nothing, and the rare code does the work.
The toy meaning search sees it differently. It ignores the code, because the code is not in its map. It only knows "problem", which belongs to the trouble group. Document d2 has two trouble words (error and declined), so it scores 0.7071 and takes first place. Document d3 has one, so it scores 0.4472. The meaning search put the wrong error page on top. That is the exact-code failure from the last section, made visible.
Now the second query, I want my money back. BM25 returns nothing. The words in the query do not appear in the refund page. The meaning search finds d4 with a perfect score of 1.0, because money, back, return and paid all sit in the same group. That is the synonym failure, and the fix.
Hybrid search
Hybrid search runs both searches and merges the two lists into one. The merge raises a puzzle. BM25 scores run on one scale (1.5385 above), and cosine scores on another (0.7071). You cannot just add numbers that mean different things.
Reciprocal rank fusion, or RRF, avoids the puzzle by ignoring the scores and using only the ranks. The Elasticsearch reference describes it as a method for combining several result sets with different relevance indicators into a single result set, and says it needs no tuning.
Here is the recipe. For each list a document appears in, add one divided by (k plus its rank). Add up across the lists. A document near the top of several lists wins.
The Elasticsearch page writes the same sum as a short loop, where a document that is missing from a list adds nothing. It says the constant k decides how much influence lower-ranked documents have, that a higher value gives them more, and that its default is 60. We use k = 60 too.
- 1
Take the two lists
One from BM25, one from the meaning search. Keep only documents with a score above zero.
- 2
Give each rank a small prize
Rank 1 earns 1/61, rank 2 earns 1/62, and so on. The prizes shrink slowly.
- 3
Add the prizes per document
A document in both lists collects two prizes.
- 4
Sort by the total
Highest total first. That is the fused list.
Check it against the output. For the first query, d3 was rank 1 in BM25 and rank 2 in the meaning list: 1/61 plus 1/62 is about 0.0325. Document d2 was rank 1 only in the meaning list: 1/61, about 0.0164. So d3 wins, and the error code is rescued. For the second query only the meaning list had a result, so d4 simply keeps first place with 0.0164.
Notice what RRF rewards: agreement. A document both lists like beats one that only a single list loves. That is usually what you want, but it means one strong list can be outvoted. If you ever need one list to count more, check what your tool allows. The Elasticsearch page says each retriever carries equal weight in its formula.
Is your search any good?
Never judge a search by trying one query and nodding. Build a small labelled set instead. It is a list of real questions, each paired with the document that should come back. You write it once and reuse it every time you change something.
Then count. Hit rate at k asks: for what share of the questions did the right document show up in the top k results? Hit rate at 1 means first place. Hit rate at 3 means anywhere in the top three. A close cousin, recall at k, asks what share of all the right documents were found in the top k. With only one right answer per question, the two numbers are the same.
The last three lines of the output do exactly this on six questions. Keywords alone get half of them right. The toy meaning search gets half at first place, and two thirds in the top three. The fused search gets all six. That looks lovely, but treat it with care: six questions is a tiny set, we picked questions that show the point, and the meaning score is fake. A real test set has dozens or hundreds of questions, written from what real users ask.
Tip
Keep every question that ever broke your search. Add it to the labelled set, fix the search, and the set stops it from breaking again.
Quick check
A user searches for the exact code ERR-4015. Which method is most likely to find it reliably?
Quick check
In reciprocal rank fusion, what does the formula use from each list?
Quick check
Your labelled set has 10 questions, and the right document is in the top 3 for 8 of them. What is hit rate at 3?
What to remember
Two ways to match
Keywords match exact words. Meaning search matches ideas through embeddings and cosine similarity.
Two ways to fail
Keywords miss synonyms. Embeddings can blur codes, names and numbers.
One fix, one check
Merge the lists with reciprocal rank fusion, and measure the result with a labelled set and hit rate.
Next: classify, recommend and spot odd ones out with embeddings.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.