esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.19

LLM Evaluations

Evals are tests for AI behaviour. Learn the parts of an eval, offline versus live checks, and build a small harness that scores answers.

12 min readFree

How do you know it works?

The last lesson, Production monitoring, ended with a promise: next, LLM evaluations, testing quality before and after you ship. Monitoring tells you what your app is doing right now. Evaluations tell you whether what it does is any good.

Picture a baker who changes the bread recipe. She does not hand it to customers and hope. She bakes a small batch, tastes it, and compares it with the old loaf. An eval is that taste test for an AI app, written down so anyone can repeat it.

By the end you will know why ordinary tests fall short, the three parts of an eval, the difference between checking before release and checking live, and how to start small. You will also run a tiny eval harness of your own.

Key words

Eval
Short for evaluation. A repeatable test that measures how well an AI app does its job.
Test case
One example in your eval: an input, plus what a good answer should or should not contain.
Dataset
The full set of test cases, kept together.
Grader
Whatever decides if an answer passes: a piece of code, another model, or a person.
Pass rate
The share of test cases that passed, such as 6 out of 8.
Offline eval
An eval you run on saved examples before real users see a change.
Online eval
A check made on live traffic, using what real users actually do.
Regression
Something that used to work and now does not, after a change.

Why normal tests are not enough

Ordinary software is predictable. Ask a calculator for 2 + 2 and you get 4 every time, so a test can say "the answer must be exactly 4". Wikipedia describes software testing as "the act of checking whether software meets its intended objectives and satisfies expectations." The idea carries over to AI apps. The method needs care.

Two things break the old method.

  • Answers vary

    OpenAI's guide says: "Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures." Ask twice and the wording can change.

  • Many answers are right

    "Returns are free for 30 days" and "You have a month to send it back at no cost" can both be correct. A test that wants one exact sentence marks the second one wrong.

  • Some answers are fuzzy

    Is a summary good? Is a reply friendly? Nobody can write an exact-match rule for that. You need a score, a rubric or a person.

Wikipedia also names an old puzzle that gets bigger here: the test oracle problem, which it describes as "the difficulty of determining the expected outcomes of selected test cases or to determine whether the actual outputs agree with the expected outcomes." With AI apps, deciding what the right answer looks like is a large part of the job.

The way out is not to give up on tests. It is to test the behaviour you care about, such as "mentions the 30-day window" or "never shows a password", instead of one exact string. OpenAI's guide defines evals as "structured tests for measuring a model's performance" that help ensure accuracy, performance and reliability "despite the nondeterministic nature of AI systems". Nondeterministic means the output is not fixed in advance.

Tip

Do not skip this because the app "seems fine". OpenAI lists "Vibe-based evals" as an anti-pattern: using "it seems like it's working" as an evaluation strategy, or waiting until you ship before implementing any evals. Hamel Husain adds that vibe checks are useful, but not enough.

The three parts of an eval

Every eval, big or small, has the same three parts. Think of a driving test: a set of situations, an examiner who scores them, and a result sheet.

  1. 1

    A dataset

    A list of inputs, each with the behaviour you expect. For a shop bot: the input "Can I get a refund?" and the note "must mention 30 days".

  2. 2

    A way to score

    Something that looks at each answer and says pass or fail, or gives a number. This is the grader.

  3. 3

    A report

    A summary you can read in ten seconds: which cases failed, why, and the overall pass rate.

OpenAI frames the full workflow the same way: define the objective, collect a dataset, define metrics, run and compare, then keep evaluating. Their list for the dataset step includes production data, historical data and human-curated data. They also say your data should include typical cases, edge cases and adversarial cases. An edge case is a rare or odd input. An adversarial case is one built to trick the app.

Three ways to score an answer

The Claude docs tell you to pick the grading method that is the fastest, most reliable and most scalable. They list three, and the ones in your code come first.

  • Code

    A program checks the answer, such as "does it contain the phrase?". The docs call code-based grading "Fastest and most reliable, extremely scalable", but it "lacks nuance for more complex judgments".

  • A model as judge

    A second model reads the answer against a rubric, which is a short list of what counts as good. The docs say it is "Fast and flexible, scalable and suitable for complex judgment", and advise: "Test to ensure reliability first, then scale."

  • Humans

    People read and mark. The docs call it "Most flexible and high quality, but slow and expensive. Avoid if possible." OpenAI also says to use human feedback to calibrate automated scoring, so you know the machine marks agree with people.

Hamel Husain's post walks through the same ladder. He starts with unit tests, which are quick assertions on the output, then moves to human and model review, then to A/B tests. The next three lessons take each grader in turn: deterministic evals, model-based evals and human evals. For now, you will use the first kind.

Before release and after

Evals live in two places.

  • Offline

    You run the eval on a saved dataset before release. It is like a rehearsal. It is safe, fast and repeatable, but it only covers the cases you thought to write.

  • Online

    You watch real traffic after release. It is like opening night. You see real inputs you never imagined, but you also affect real people.

  • Together

    Failures you spot live become new offline cases. Offline checks then protect you from the same failure next time.

OpenAI's guide describes this loop. It says to set up continuous evaluation "to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time". Another of its tips: "Log everything: Log as you develop so you can mine your logs for good eval cases." This is where the Production monitoring lesson pays off.

Hamel Husain's third level, A/B testing, is an online check: you show two versions to different users and see which one gets the behaviour you want. He says it is okay to put this stage off until the product is ready for real users.

Tip

Each change is a new risk. Re-running the same offline eval after every prompt or model edit is how you catch a regression early. The next lesson after the three graders, regression testing, goes deeper.

Start small, grow from failures

A beginner's trap is to wait until the dataset is perfect. Do the opposite.

A common starting point, which is advice from practice and not from one of this lesson's sources, is 20 to 50 real cases. Take real questions from users, logs or your own testing. Write down what a good answer must contain. Run them. When the app fails on something new, add that failure as a case. The set grows exactly where your app is weak.

The Claude docs support the spirit of this. Among their design principles: "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." Make cases that mirror your real task, edge cases included, and let a program score them.

Hamel Husain also says something that surprises people: unlike traditional unit tests, "you don't necessarily need a 100% pass rate. Your pass rate is a product decision, depending on the failures you are willing to tolerate." A rude reply and a leaked secret are not the same size of problem.

Build a tiny eval harness

Time to build one. The program below is a harness, a small frame that runs every case through the app and scores it. It tests a shop's support bot with 8 cases. Each case has an input, a list of keywords the answer must contain, and a list of words it must not contain.

Important: fake_app is a stand-in. It follows fixed rules, with no real model inside, so this code runs free, offline and gives the same answer every time. A real model is slower, costs money for every token (the small pieces of text it reads and writes), can give a different answer each run, and can fail or refuse. Your eval would call the real app in the same place.

Python
def fake_app(question):
    # A stand-in for a real AI app: fixed rules, so this code runs free and gives the same answer every time.
    q = question.lower()
    if "refund" in q:
        return "You can ask for a refund within 30 days of buying."
    if "password" in q:
        return "Click Forgot password on the sign-in page. Your password is 'hunter2'."
    if "ship" in q:
        return "Standard delivery takes 3 to 5 days."
    if "cancel" in q:
        return "Go to Orders and press Cancel. Orders already shipped cannot be cancelled."
    if "invoice" in q:
        return "Your invoice is in the Billing page."
    if "price" in q:
        return "The Pro plan costs 49 dollars a month."
    if "human" in q:
        return "Sure, I can pass you to a person on the support team."
    return "Sorry, I do not know."

cases = [
    {"id": "refund",   "input": "Can I get a refund?",              "must_have": ["refund", "30 days"],       "must_not": ["guarantee"]},
    {"id": "password", "input": "I forgot my password",             "must_have": ["forgot password"],         "must_not": ["your password is"]},
    {"id": "shipping", "input": "How long does shipping take?",     "must_have": ["3 to 5 days"],             "must_not": ["free"]},
    {"id": "cancel",   "input": "How do I cancel my order?",        "must_have": ["orders", "cancel"],        "must_not": ["call us"]},
    {"id": "invoice",  "input": "Where is my invoice?",             "must_have": ["billing", "pdf"],          "must_not": []},
    {"id": "price",    "input": "What is the price of Pro?",        "must_have": ["49"],                      "must_not": ["free"]},
    {"id": "human",    "input": "Let me talk to a human",           "must_have": ["person"],                  "must_not": ["sorry"]},
    {"id": "unknown",  "input": "Do you sell gift cards?",          "must_have": ["do not know"],             "must_not": ["yes"]},
]

def grade(case, answer):
    low = answer.lower()
    reasons = []
    for word in case["must_have"]:
        if word not in low:
            reasons.append("missing '" + word + "'")
    for word in case["must_not"]:
        if word in low:
            reasons.append("contains forbidden '" + word + "'")
    return (len(reasons) == 0, "; ".join(reasons) or "ok")

passed = 0
print(f"{'case':<10}{'result':<8}reason")
print("-" * 44)
for case in cases:
    ok, why = grade(case, fake_app(case["input"]))
    passed += ok
    print(f"{case['id']:<10}{'PASS' if ok else 'FAIL':<8}{why}")
print("-" * 44)
print(f"pass rate: {passed}/{len(cases)} = {passed / len(cases):.0%}")

Output

case      result  reason
--------------------------------------------
refund    PASS    ok
password  FAIL    contains forbidden 'your password is'
shipping  PASS    ok
cancel    PASS    ok
invoice   FAIL    missing 'pdf'
price     PASS    ok
human     PASS    ok
unknown   PASS    ok
--------------------------------------------
pass rate: 6/8 = 75%

Read the table. Six cases passed and two failed, so the pass rate is 6 out of 8, which is 75 percent. The reasons column is the useful part: a score alone tells you something is wrong, and the reason tells you where to look.

The two failures are different. The password case failed because the answer contained "your password is", a leak the case was written to catch. That is a real bug in the app. The invoice case failed because the answer lacks the word "pdf". Maybe the app is wrong. Maybe the case is too strict, since nothing says the invoice has to be a PDF. When a case fails, check the case as well as the app.

This is a keyword check, the simplest grader. It cannot tell a friendly answer from a cold one, and it can be fooled by a wrong answer that happens to use the right words. That is why the next lessons add other graders.

Reading the harness line by line

  1. 1

    cases

    The dataset. Each case is a small record with an id, the input, must_have and must_not.

  2. 2

    grade

    The grader. It lowercases the answer, checks each must_have word is present and each must_not word is absent, and collects the reasons. No reasons means pass.

  3. 3

    The loop

    Runs every case through fake_app, grades it, and prints one table row. It also counts the passes.

  4. 4

    The last line

    The report: passes divided by cases, shown as a percentage.

Tip

Swap fake_app for a function that calls your real app, and nothing else changes. That is the point: the harness does not care what is behind the door.

Quick check

Quick check

Why is a test like "the answer must equal this exact sentence" a poor fit for most AI apps?

Quick check

In the harness, case "invoice" failed for a missing "pdf". What is the best next step?

What to remember

  • Test behaviour

    Normal tests want one exact answer. AI answers vary, so check what must be there and what must not.

  • Three parts

    A dataset of inputs with expected behaviour, a grader, and a report with reasons.

  • Start small, then grow

    Begin with a few real cases, run them often, and add each new failure as a case.

Next: deterministic evals, checks code can score exactly.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.