AI engineering / Advanced / 03.19
LLM Evaluations
Evals are tests for AI behaviour. Learn the parts of an eval, offline versus live checks, and build a small harness that scores answers.
12 min readFree
How do you know it works?
The last lesson, Production monitoring, ended with a promise: next, LLM evaluations, testing quality before and after you ship. Monitoring tells you what your app is doing right now. Evaluations tell you whether what it does is any good.
Picture a baker who changes the bread recipe. She does not hand it to customers and hope. She bakes a small batch, tastes it, and compares it with the old loaf. An eval is that taste test for an AI app, written down so anyone can repeat it.
By the end you will know why ordinary tests fall short, the three parts of an eval, the difference between checking before release and checking live, and how to start small. You will also run a tiny eval harness of your own.
Key words
- Eval
- Short for evaluation. A repeatable test that measures how well an AI app does its job.
- Test case
- One example in your eval: an input, plus what a good answer should or should not contain.
- Dataset
- The full set of test cases, kept together.
- Grader
- Whatever decides if an answer passes: a piece of code, another model, or a person.
- Pass rate
- The share of test cases that passed, such as 6 out of 8.
- Offline eval
- An eval you run on saved examples before real users see a change.
- Online eval
- A check made on live traffic, using what real users actually do.
- Regression
- Something that used to work and now does not, after a change.
Why normal tests are not enough
Ordinary software is predictable. Ask a calculator for 2 + 2 and you get 4 every time, so a test can say "the answer must be exactly 4". Wikipedia describes software testing as "the act of checking whether software meets its intended objectives and satisfies expectations." The idea carries over to AI apps. The method needs care.
Two things break the old method.
Answers vary
OpenAI's guide says: "Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures." Ask twice and the wording can change.
Many answers are right
"Returns are free for 30 days" and "You have a month to send it back at no cost" can both be correct. A test that wants one exact sentence marks the second one wrong.
Some answers are fuzzy
Is a summary good? Is a reply friendly? Nobody can write an exact-match rule for that. You need a score, a rubric or a person.
Wikipedia also names an old puzzle that gets bigger here: the test oracle problem, which it describes as "the difficulty of determining the expected outcomes of selected test cases or to determine whether the actual outputs agree with the expected outcomes." With AI apps, deciding what the right answer looks like is a large part of the job.
The way out is not to give up on tests. It is to test the behaviour you care about, such as "mentions the 30-day window" or "never shows a password", instead of one exact string. OpenAI's guide defines evals as "structured tests for measuring a model's performance" that help ensure accuracy, performance and reliability "despite the nondeterministic nature of AI systems". Nondeterministic means the output is not fixed in advance.
Tip
Do not skip this because the app "seems fine". OpenAI lists "Vibe-based evals" as an anti-pattern: using "it seems like it's working" as an evaluation strategy, or waiting until you ship before implementing any evals. Hamel Husain adds that vibe checks are useful, but not enough.
The three parts of an eval
Every eval, big or small, has the same three parts. Think of a driving test: a set of situations, an examiner who scores them, and a result sheet.
- 1
A dataset
A list of inputs, each with the behaviour you expect. For a shop bot: the input "Can I get a refund?" and the note "must mention 30 days".
- 2
A way to score
Something that looks at each answer and says pass or fail, or gives a number. This is the grader.
- 3
A report
A summary you can read in ten seconds: which cases failed, why, and the overall pass rate.
OpenAI frames the full workflow the same way: define the objective, collect a dataset, define metrics, run and compare, then keep evaluating. Their list for the dataset step includes production data, historical data and human-curated data. They also say your data should include typical cases, edge cases and adversarial cases. An edge case is a rare or odd input. An adversarial case is one built to trick the app.
Three ways to score an answer
The Claude docs tell you to pick the grading method that is the fastest, most reliable and most scalable. They list three, and the ones in your code come first.
Code
A program checks the answer, such as "does it contain the phrase?". The docs call code-based grading "Fastest and most reliable, extremely scalable", but it "lacks nuance for more complex judgments".
A model as judge
A second model reads the answer against a rubric, which is a short list of what counts as good. The docs say it is "Fast and flexible, scalable and suitable for complex judgment", and advise: "Test to ensure reliability first, then scale."
Humans
People read and mark. The docs call it "Most flexible and high quality, but slow and expensive. Avoid if possible." OpenAI also says to use human feedback to calibrate automated scoring, so you know the machine marks agree with people.
Hamel Husain's post walks through the same ladder. He starts with unit tests, which are quick assertions on the output, then moves to human and model review, then to A/B tests. The next three lessons take each grader in turn: deterministic evals, model-based evals and human evals. For now, you will use the first kind.
Before release and after
Evals live in two places.
Offline
You run the eval on a saved dataset before release. It is like a rehearsal. It is safe, fast and repeatable, but it only covers the cases you thought to write.
Online
You watch real traffic after release. It is like opening night. You see real inputs you never imagined, but you also affect real people.
Together
Failures you spot live become new offline cases. Offline checks then protect you from the same failure next time.
OpenAI's guide describes this loop. It says to set up continuous evaluation "to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time". Another of its tips: "Log everything: Log as you develop so you can mine your logs for good eval cases." This is where the Production monitoring lesson pays off.
Hamel Husain's third level, A/B testing, is an online check: you show two versions to different users and see which one gets the behaviour you want. He says it is okay to put this stage off until the product is ready for real users.
Tip
Each change is a new risk. Re-running the same offline eval after every prompt or model edit is how you catch a regression early. The next lesson after the three graders, regression testing, goes deeper.
Start small, grow from failures
A beginner's trap is to wait until the dataset is perfect. Do the opposite.
A common starting point, which is advice from practice and not from one of this lesson's sources, is 20 to 50 real cases. Take real questions from users, logs or your own testing. Write down what a good answer must contain. Run them. When the app fails on something new, add that failure as a case. The set grows exactly where your app is weak.
The Claude docs support the spirit of this. Among their design principles: "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." Make cases that mirror your real task, edge cases included, and let a program score them.
Hamel Husain also says something that surprises people: unlike traditional unit tests, "you don't necessarily need a 100% pass rate. Your pass rate is a product decision, depending on the failures you are willing to tolerate." A rude reply and a leaked secret are not the same size of problem.
Build a tiny eval harness
Time to build one. The program below is a harness, a small frame that runs every case through the app and scores it. It tests a shop's support bot with 8 cases. Each case has an input, a list of keywords the answer must contain, and a list of words it must not contain.
Important: fake_app is a stand-in. It follows fixed rules, with no real model inside, so this code runs free, offline and gives the same answer every time. A real model is slower, costs money for every token (the small pieces of text it reads and writes), can give a different answer each run, and can fail or refuse. Your eval would call the real app in the same place.
def fake_app(question):
# A stand-in for a real AI app: fixed rules, so this code runs free and gives the same answer every time.
q = question.lower()
if "refund" in q:
return "You can ask for a refund within 30 days of buying."
if "password" in q:
return "Click Forgot password on the sign-in page. Your password is 'hunter2'."
if "ship" in q:
return "Standard delivery takes 3 to 5 days."
if "cancel" in q:
return "Go to Orders and press Cancel. Orders already shipped cannot be cancelled."
if "invoice" in q:
return "Your invoice is in the Billing page."
if "price" in q:
return "The Pro plan costs 49 dollars a month."
if "human" in q:
return "Sure, I can pass you to a person on the support team."
return "Sorry, I do not know."
cases = [
{"id": "refund", "input": "Can I get a refund?", "must_have": ["refund", "30 days"], "must_not": ["guarantee"]},
{"id": "password", "input": "I forgot my password", "must_have": ["forgot password"], "must_not": ["your password is"]},
{"id": "shipping", "input": "How long does shipping take?", "must_have": ["3 to 5 days"], "must_not": ["free"]},
{"id": "cancel", "input": "How do I cancel my order?", "must_have": ["orders", "cancel"], "must_not": ["call us"]},
{"id": "invoice", "input": "Where is my invoice?", "must_have": ["billing", "pdf"], "must_not": []},
{"id": "price", "input": "What is the price of Pro?", "must_have": ["49"], "must_not": ["free"]},
{"id": "human", "input": "Let me talk to a human", "must_have": ["person"], "must_not": ["sorry"]},
{"id": "unknown", "input": "Do you sell gift cards?", "must_have": ["do not know"], "must_not": ["yes"]},
]
def grade(case, answer):
low = answer.lower()
reasons = []
for word in case["must_have"]:
if word not in low:
reasons.append("missing '" + word + "'")
for word in case["must_not"]:
if word in low:
reasons.append("contains forbidden '" + word + "'")
return (len(reasons) == 0, "; ".join(reasons) or "ok")
passed = 0
print(f"{'case':<10}{'result':<8}reason")
print("-" * 44)
for case in cases:
ok, why = grade(case, fake_app(case["input"]))
passed += ok
print(f"{case['id']:<10}{'PASS' if ok else 'FAIL':<8}{why}")
print("-" * 44)
print(f"pass rate: {passed}/{len(cases)} = {passed / len(cases):.0%}")Output
case result reason -------------------------------------------- refund PASS ok password FAIL contains forbidden 'your password is' shipping PASS ok cancel PASS ok invoice FAIL missing 'pdf' price PASS ok human PASS ok unknown PASS ok -------------------------------------------- pass rate: 6/8 = 75%
Read the table. Six cases passed and two failed, so the pass rate is 6 out of 8, which is 75 percent. The reasons column is the useful part: a score alone tells you something is wrong, and the reason tells you where to look.
The two failures are different. The password case failed because the answer contained "your password is", a leak the case was written to catch. That is a real bug in the app. The invoice case failed because the answer lacks the word "pdf". Maybe the app is wrong. Maybe the case is too strict, since nothing says the invoice has to be a PDF. When a case fails, check the case as well as the app.
This is a keyword check, the simplest grader. It cannot tell a friendly answer from a cold one, and it can be fooled by a wrong answer that happens to use the right words. That is why the next lessons add other graders.
Reading the harness line by line
- 1
cases
The dataset. Each case is a small record with an id, the input, must_have and must_not.
- 2
grade
The grader. It lowercases the answer, checks each must_have word is present and each must_not word is absent, and collects the reasons. No reasons means pass.
- 3
The loop
Runs every case through fake_app, grades it, and prints one table row. It also counts the passes.
- 4
The last line
The report: passes divided by cases, shown as a percentage.
Tip
Swap fake_app for a function that calls your real app, and nothing else changes. That is the point: the harness does not care what is behind the door.
Quick check
Quick check
Why is a test like "the answer must equal this exact sentence" a poor fit for most AI apps?
Quick check
In the harness, case "invoice" failed for a missing "pdf". What is the best next step?
What to remember
Test behaviour
Normal tests want one exact answer. AI answers vary, so check what must be there and what must not.
Three parts
A dataset of inputs with expected behaviour, a grader, and a report with reasons.
Start small, then grow
Begin with a few real cases, run them often, and add each new failure as a case.
Next: deterministic evals, checks code can score exactly.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.