esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.10

Classify, recommend, spot odd ones

Three more jobs for embeddings: sort items into labels, suggest what a person may like, and flag the ones that do not fit.

14 min readFree

Three more jobs for the same numbers

The last lesson, Semantic Search, ended with a promise: next, classify, recommend and spot odd ones out with embeddings. Here it is.

You know the key idea. An embedding turns text into a short list of numbers, and texts with similar meaning get lists that sit close together. Search used that closeness to match questions. The same closeness does three more jobs.

Picture a sorting office. One worker drops each parcel in the right bin: classifying. Another, who knows what you bought before, says "you may like this too": recommending. A third spots the one parcel that looks nothing like the rest: finding odd ones out.

You will run four small Python programs. They use made-up numbers and the standard library, so they cost nothing.

Key words

Label
The name of a group, such as "billing" or "bug". Classifying means choosing a label.
Vector
A list of numbers. Here each has three, so you can follow by eye. Real embeddings have hundreds or thousands.
Distance
How far apart two vectors are. Small means similar.
Centre
The average of many vectors: add them up, divide by how many there are.
Standard deviation
How spread out a set of values is. Small means they hug the average. Large means they scatter.
Anomaly
An item that does not fit with the rest. Also called an outlier.

Data classification: choosing a label

A help desk gets hundreds of messages a day: some about money, some about things that broke, some plain questions. Someone has to sort them. Software can do it.

The Wikipedia page on statistical classification says: "An algorithm that implements classification, especially in a concrete implementation, is known as a classifier." It also notes that "Other classifiers work by comparing observations to previous observations by means of a similarity or distance function." That second sentence is our first method.

  • Nearest labelled examples

    Keep a pile of messages a person already labelled. For a new message, find the closest ones and copy their label. No model has to answer a question.

  • Ask a model

    Give a language model the message and a fixed list of allowed labels. Ask for a short structured reply, such as JSON. Then check the reply.

  • Mix them

    Use nearest examples for the easy cases. Send only the unclear ones to a model, to keep cost and delay down.

Method one: the nearest neighbours vote

This method is called k-nearest neighbours. The Wikipedia page says that in classification "a new example is assigned a label based on the labels of its k nearest training examples". The k is a small whole number you pick.

  1. 1

    Keep labelled examples

    Each is a vector plus the label a person gave it.

  2. 2

    Measure distances

    Work out how far the new item is from every labelled example.

  3. 3

    Pick the k closest

    Sort by distance and keep the first k. With k = 3, you keep three.

  4. 4

    Let them vote

    Count the labels among those three. The most common one wins.

We have 9 labelled tickets and 3 labels. Each ticket has a made-up vector: how much it is about money, how much about something broken, how much a how-to question. Real embeddings have no such tidy meaning, but the method is the same. Four new tickets arrive, and k is 3. Each line shows the answer, the votes, and the distances to the three closest examples.

Python
import math
from collections import Counter

# Made-up vectors: (money, broken, how-to). Bigger = more of that.
labelled = [
    ("I was charged twice",        (0.9, 0.2, 0.1), "billing"),
    ("Refund my last payment",     (0.8, 0.1, 0.2), "billing"),
    ("Wrong amount on invoice",    (0.9, 0.3, 0.1), "billing"),
    ("App crashes on start",       (0.1, 0.9, 0.2), "bug"),
    ("Button does nothing",        (0.1, 0.8, 0.3), "bug"),
    ("Page shows an error",        (0.2, 0.9, 0.1), "bug"),
    ("How do I export my data?",   (0.1, 0.2, 0.9), "how-to"),
    ("Where is the dark mode?",    (0.1, 0.1, 0.9), "how-to"),
    ("Can I invite a teammate?",   (0.2, 0.1, 0.8), "how-to"),
]
new_tickets = [
    ("Why was my card billed?",         (0.85, 0.15, 0.3)),
    ("Checkout freezes when I pay",     (0.5, 0.6, 0.2)),
    ("How do I change my password?",    (0.1, 0.2, 0.85)),
    ("Export fails with an error",      (0.1, 0.5, 0.5)),
]

def distance(a, b):
    return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))

K = 3
for text, vec in new_tickets:
    nearest = sorted(labelled, key=lambda row: distance(vec, row[1]))[:K]
    votes = Counter(label for _, _, label in nearest)
    winner, count = votes.most_common(1)[0]
    gaps = [f"{distance(vec, v):.2f}" for _, v, _ in nearest]
    print(text, "->", winner, f"({count} of {K})", gaps)

Output

Why was my card billed? -> billing (3 of 3) ['0.12', '0.21', '0.25']
Checkout freezes when I pay -> bug (3 of 3) ['0.44', '0.46', '0.50']
How do I change my password? -> how-to (3 of 3) ['0.05', '0.11', '0.15']
Export fails with an error -> bug (2 of 3) ['0.36', '0.50', '0.50']

The first ticket has three billing neighbours, so it is billing, 3 of 3. The second is about paying, yet its three closest examples are all bugs, so it gets "bug", 3 of 3. The third is a how-to, 3 of 3.

The fourth, "Export fails with an error", sits between two groups. Two of its three neighbours are bugs and one is a how-to, so it becomes "bug" with only 2 of 3. A split vote says the answer is shaky, so a person might look.

Choosing k, and what can go wrong

The same page says: "In binary (two class) classification problems, it is helpful to choose k to be an odd number as this avoids tied votes." With three labels, ties can still happen, so decide in advance what to do, for example send a tied item to a person.

It adds that "generally, larger values of k reduces effect of the noise on the classification, but make boundaries between classes less distinct." In plain words, a bigger k is calmer but blurrier.

And when one class is much bigger, "examples of a more frequent class tend to dominate the prediction of the new example". If 90 of 100 labelled tickets are billing, new tickets lean billing. Keep the pile fairly balanced.

Tip

Try a few values of k on tickets whose answer you already know, and keep the one that gets the most right.

Method two: ask a model with a fixed list

Two habits make a model dependable here. First, give a fixed list of allowed labels, so it cannot invent new ones. Second, ask for structured output, a reply in a fixed shape such as JSON, so your code can read it without guessing.

Even then, check the reply. If it is not valid JSON, or the label is not on the list, call it "unsure" and let a person look.

The code uses a stand-in. The function fake_model is a tiny pretend model that always returns the same fixed text, so the program runs free with no key and no internet. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. That is why the checking matters. The last two lines feed in bad replies on purpose.

Python
import json

LABELS = ["billing", "bug", "how-to"]

def fake_model(messages):
    # Stand-in: always returns the same fixed text, so this code runs free.
    return '{"label": "bug", "confidence": 0.8}'

def classify(ticket, model):
    prompt = f"Pick exactly one label from {LABELS} for this ticket. Reply as JSON with label and confidence.\nTicket: {ticket}"
    reply = model([{"role": "user", "content": prompt}])
    try:
        data = json.loads(reply)
    except json.JSONDecodeError:
        return "unsure"
    return data["label"] if data.get("label") in LABELS else "unsure"

print(classify("Export fails with an error", fake_model))
print(classify("Hello", lambda messages: "I think it is probably billing!"))
print(classify("Hello", lambda messages: '{"label": "spam"}'))

Output

bug
unsure
unsure

The first line is the good reply, "bug". The second reply was chatty text, not JSON, so it became "unsure". The third was valid JSON but used "spam", which is not on our list, so it also became "unsure".

How to check a classifier works

Test it on tickets whose right label a person already wrote, and count the matches. Use tickets the classifier has never seen, never the labelled pile itself. Then read the misses: they show what to fix, such as a missing label or a poor example. Repeat after every change.

Quick check

Your classifier gets 9 of 10 right on the very tickets it was built from. What now?

Recommending: more of what you liked

The Wikipedia page on recommender systems defines one as "a type of information filtering system that aims to suggest items most relevant for some input or to a particular user." Playlists and "you may also like" rows are everyday examples.

It describes content-based filtering, whose algorithms "try to recommend items similar to those that a user liked in the past, spends the most time on, or is examining in the present." That is a good match for embeddings, which give every item a vector.

  1. 1

    Embed every item

    Each film, article or product gets a vector.

  2. 2

    Average the liked ones

    Average the vectors of liked items. That is the user's taste as one vector.

  3. 3

    Score the rest

    Compare the taste vector to every other item.

  4. 4

    Show the top few

    Leave out items already liked, then show the closest.

For closeness we use cosine similarity, which compares the direction two vectors point in. A score of 1 means the same direction. A score near 0 means little in common.

cos⁡(a,b)=a⋅b∥a∥ ∥b∥\cos(a,b)=\frac{a\cdot b}{\lVert a\rVert\,\lVert b\rVert}

In words: multiply the two vectors number by number, add the results, and divide by the length of each vector. Our seven films have made-up vectors of action, comedy and romance. The user liked two.

Python
import math

# Made-up vectors: (action, comedy, romance). Bigger = more of that.
films = {
    "Night Chase":    (0.9, 0.1, 0.1),
    "Iron Fist 2":    (0.8, 0.2, 0.2),
    "Laugh Track":    (0.1, 0.9, 0.2),
    "Wedding Mix-up": (0.1, 0.7, 0.7),
    "Love Letters":   (0.1, 0.2, 0.9),
    "Action Comedy":  (0.6, 0.7, 0.1),
    "Quiet Garden":   (0.0, 0.1, 0.6),
}
liked = ["Night Chase", "Laugh Track"]

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

profile = tuple(sum(films[f][i] for f in liked) / len(liked) for i in range(3))
print("profile", tuple(round(x, 2) for x in profile))

scores = {f: cosine(profile, v) for f, v in films.items() if f not in liked}
for f, s in sorted(scores.items(), key=lambda kv: -kv[1]):
    print(f"{s:.2f}", f)
top2 = sorted(scores, key=scores.get, reverse=True)[:2]
print("recommend:", top2)

Output

profile (0.5, 0.5, 0.15)
0.99 Action Comedy
0.86 Iron Fist 2
0.70 Wedding Mix-up
0.43 Love Letters
0.32 Quiet Garden
recommend: ['Action Comedy', 'Iron Fist 2']

The user liked "Night Chase" (all action) and "Laugh Track" (all comedy). Their average, (0.5, 0.5, 0.15), leans to both. The top two are "Action Comedy" at 0.99 and "Iron Fist 2" at 0.86. "Love Letters" scores only 0.43.

This is a first taste. The Recommendation Systems lesson in Machine Learning Foundations goes deeper.

When it fits, and how to check it

It fits when you know something about each item, such as a description, and only a little about the user. The page says content-based methods suit situations "where there is known data on an item (name, location, description, etc.), but not on the user." It struggles with a brand new user who liked nothing, because there is nothing to average. Show popular items first.

To check it, hide one item a user liked, build their taste from the rest, and see whether the hidden item shows up in the top picks. Do it for many users and count. Also read the lists yourself: if every pick is a copy of what they already have, they may never see anything new.

Tip

Add a little variety on purpose, such as one pick from outside the top group.

Spotting the odd ones out

The Wikipedia page on anomaly detection describes it as "the identification of rare items, events or observations which deviate significantly from the majority of the data and do not conform to a well defined notion of normal behavior." It lists uses including "cyber-security, intrusion detection, fraud detection, fault detection, system health monitoring".

With embeddings, the simple version is: find the centre of a group of vectors, measure how far each item sits from it, and flag the far ones. One common rule flags any item whose distance is more than two standard deviations above the average distance.

di>dˉ+2sd_i > \bar{d} + 2s

In words: d_i is one item's distance from the centre, d-bar is the average of all the distances, and s is their standard deviation. Bigger than average plus twice the spread means flagged. We run it on 12 made-up tickets. Eleven sit close together and one was placed far away on purpose.

Python
import math

# Made-up vectors for 12 support tickets in one group.
vectors = {
    "t1": (0.50, 0.50, 0.52), "t2": (0.55, 0.48, 0.50), "t3": (0.48, 0.53, 0.47),
    "t4": (0.52, 0.47, 0.55), "t5": (0.47, 0.52, 0.50), "t6": (0.53, 0.55, 0.49),
    "t7": (0.49, 0.49, 0.53), "t8": (0.51, 0.51, 0.45), "t9": (0.54, 0.46, 0.48),
    "t10": (0.46, 0.50, 0.51), "t11": (0.50, 0.54, 0.54),
    "odd": (0.95, 0.05, 0.90),
}

n = len(vectors)
centre = tuple(sum(v[i] for v in vectors.values()) / n for i in range(3))
dist = {k: math.dist(v, centre) for k, v in vectors.items()}
mean = sum(dist.values()) / n
std = math.sqrt(sum((d - mean) ** 2 for d in dist.values()) / n)
limit = mean + 2 * std
print("centre", tuple(round(x, 2) for x in centre))
print(f"mean distance {mean:.2f}, std {std:.2f}, limit {limit:.2f}")
for k, d in sorted(dist.items(), key=lambda kv: -kv[1])[:3]:
    print(k, f"{d:.2f}", "FLAG" if d > limit else "ok")
print("flagged:", [k for k, d in dist.items() if d > limit])

Output

centre (0.54, 0.47, 0.54)
mean distance 0.13, std 0.17, limit 0.47
odd 0.69 FLAG
t3 0.11 ok
t8 0.10 ok
flagged: ['odd']

The centre is (0.54, 0.47, 0.54). The average distance is 0.13 and the spread is 0.17, so the limit is 0.47. The item called "odd" sits at 0.69, well past it, so it is flagged.

The odd item also pulled the centre and the spread towards itself. With only a few items, one extreme point can hide itself that way, so this rule works better with more data.

When it fits, and how to check it

It fits when you have plenty of normal items, few strange ones, and no labels for the strange ones. The page says "Unsupervised anomaly detection techniques assume the data is unlabelled and are by far the most commonly used". Our distance rule is of that kind. It also notes that "in many applications anomalies themselves are of interest". A flag only says look here, not why.

To check it, mix a few known odd items into a set and see how many get flagged. Then count false alarms, normal items flagged by mistake. Too many and people stop reading the alerts, so tune the limit until a flag is worth a person's time.

Quick check

Why does a flag from this method not prove something is wrong?

Putting it together

In short: classify by nearest labelled examples or a checked model reply, recommend by averaging what a user liked, and spot odd ones by distance from the centre. Test each on examples it has never seen.

All three rest on one idea: close vectors mean similar things. The real work is choosing good examples, deciding what to do with unsure answers, and testing fairly.

Next: chunking, cutting documents into pieces worth retrieving.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.