AI engineering / Advanced / 03.20
Deterministic Evals
Checks your own code can mark with no doubt: exact words, a pattern, a valid shape, a number, a tool call. Cheap, fast and the same every run.
13 min readFree
Checks code can score exactly
The last lesson, LLM Evaluations, ended with a promise: next, deterministic evals, checks code can score exactly. Here they are.
Think of a spelling test. The teacher has an answer key. Your word either matches the key or it does not. Now think of an essay. Two teachers may give two different marks. Deterministic evals are the spelling test of AI engineering.
By the end you will know the main kinds of exact check, why they are so useful, where they stop working, and how to run six of them in plain Python. The code needs no model, no key and no internet.
Key words
- Eval
- A test for an AI feature. You give it fixed questions, look at the answers and score them.
- Deterministic
- Always gives the same result for the same input. Run the check twice on the same text and you get the same mark twice.
- Check (or assertion)
- One small rule an answer must follow, such as "must contain the word Paris". It says pass or fail.
- Golden answer
- The correct answer you wrote down ahead of time. Checks compare the model's answer with it.
- Regular expression (regex)
- A short pattern that describes text, like "four digits, a dash, two digits". It finds text that fits the pattern.
- Schema
- A description of the shape a piece of data must have, such as "an object with a name that is text and an age that is a whole number".
- Tolerance
- How far a number may be from the right one and still count. A tolerance of 0.05 means off by up to five hundredths is fine.
What makes a check deterministic
A model gives a different answer each time you ask. Your check must not. If the check itself wobbles, you cannot tell whether the feature got worse or the test did.
A deterministic check is a small piece of ordinary code. It takes the model's answer, applies a fixed rule, and returns pass or fail. No model is used to judge. Promptfoo, a tool for testing AI features, lists these under the heading "Deterministic eval metrics" and describes them as "programmatic tests that are run on LLM output".
The Anthropic guide on building evaluations says to choose "the fastest, most reliable, most scalable method" for grading. It names code-based grading first and describes it as "Fastest and most reliable, extremely scalable, but also lacks nuance for more complex judgments that require less rule-based rigidity."
Cheap
It is a few lines of code. No second model to pay for, and nothing to wait for.
Fast
A thousand checks finish in the blink of an eye, so you can run them after every small change.
Repeatable
The same answer gets the same mark today, tomorrow and on a teammate's laptop.
The kinds of exact check
Most checks are one of a handful of types. The Promptfoo page lists many of them in a table with the plain meaning of each. A few of its rows, in its own words: equals, "output matches exactly"; contains, "output contains a string or number as text"; regex, "output matches regex"; is-json, "output is valid json (optional json schema validation)".
Here are the ones you will use most, with a daily-life picture for each.
Exact match
The answer must equal the golden answer, letter for letter. The Anthropic page writes it as output == golden_answer. It adds that exact match evals measure whether the output matches a predefined correct answer, "typically after normalizing whitespace and case".
Normalizing means tidying first: cut stray spaces and make everything lower case. Use exact match when the answer is one of a few labels, like "positive", "negative" or "neutral". It is like a multiple-choice sheet: the box is ticked or it is not.
Contains and does not contain
The answer must include a key phrase. The Anthropic page writes it as key_phrase in output. This is gentler than exact match. A model may wrap the answer in a polite sentence, and the check still passes if the key fact is inside.
Flip it around and you can check what must not appear: a competitor's name, a swear word, a leaked secret code. "Does not contain" is a very common guard.
Regex: matching a pattern
Sometimes you do not know the exact words, only the shape. A date looks like four digits, a dash, two digits, a dash, two digits. A regex describes that shape.
Python's re module has two functions you will meet. The Python docs say re.search scans through the string "looking for the first location where the regular expression pattern produces a match". And re.fullmatch returns a match only "If the whole string matches the regular expression pattern". Search finds the pattern anywhere. Full match demands the whole text fits.
Valid JSON and the right shape
Apps often ask a model for JSON, a text format for structured data, so the next piece of code can read it. A check can try to parse the text. If parsing fails, the answer is broken, however nice it sounds.
Then check the shape: are the keys there, and is each value the right kind? Promptfoo's is-json check offers optional schema validation. It also lists checks that a function call matches the function's JSON schema. The Structured Output lesson covers asking for this shape in the first place.
Numbers within a tolerance
If the right answer is 12.5, is 12.49 wrong? For a shop total, close is probably fine. A check that pulls the number out of the text and compares it with the target, allowing a small gap, handles this.
Tool calls with the right arguments
When a model can use tools (see the Function Calling lesson), the answer is not words but a request: "call get_weather with city Pune". That request is data, so a check can compare it with the right one: same tool name, same arguments. Wrong city, wrong call.
Length, citations and more
Once you see the pattern, you can write a check for almost any rule that has a clear yes or no. Three more favourites:
Length limit
A summary must be under 100 characters, or a reply under 3 sentences. In Promptfoo's short syntax this is a line such as javascript:output.length < 100, and python:len(output) > 10 is shown the same way.
Citations exist
If the answer says "see source [3]", check that source 3 is really in the list you gave the model. It catches made-up sources with no model involved.
Speed or similarity
Promptfoo lists latency (below a threshold in milliseconds) and levenshtein (a count of letter edits between two texts, below a threshold). Both are numbers a program can compute.
Tip
Start with the checks that guard against disasters: broken JSON, a missing citation, a forbidden word. They are the easiest to write and the most painful to miss.
Six checks you can run
Let's build a small library of six checks, one for each kind we just met (exact, contains, regex, JSON shape, number tolerance, tool arguments). Then we run them on six sample answers.
Important: these six answers are stand-ins. We typed them by hand so the code runs free, with no key and no network. No model wrote them. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. That is exactly why your checks must be steady even when the model is not.
Each case lists which checks apply to it. The grid prints pass, FAIL, or a dash when a check was not asked for.
import json
import re
def exact(output, expected):
return output == expected
def contains(output, phrase):
return phrase in output
def regex(output, pattern):
return re.search(pattern, output) is not None
def json_schema_lite(output, schema):
try:
data = json.loads(output)
except json.JSONDecodeError:
return False
if not isinstance(data, dict):
return False
return all(key in data and type(data[key]) is kind for key, kind in schema.items())
def number_within(output, target, tolerance):
match = re.search(r"-?\d+(?:\.\d+)?", output)
return match is not None and abs(float(match.group()) - target) <= tolerance
def tool_args_equal(call, name, args):
return call.get("tool") == name and call.get("args") == args
# Six stand-in outputs, typed in by hand. No model is running here.
cases = [
("capital", "Paris.",
{"exact": ("Paris",), "contains": ("Paris",)}),
("returns", "You can return it within 30 days.",
{"contains": ("30 days",), "regex": (r"\b\d+ days\b",)}),
("ship date", "Shipped on 3 November.",
{"regex": (r"\d{4}-\d{2}-\d{2}",)}),
("profile", '{"name": "Asha", "age": 31}',
{"json_schema": ({"name": str, "age": int},)}),
("total", "12.49",
{"number": (12.5, 0.05,)}),
("weather call", {"tool": "get_weather", "args": {"city": "Pune", "unit": "c"}},
{"tool_args": ("get_weather", {"city": "Pune", "unit": "celsius"})}),
]
checks = {
"exact": exact, "contains": contains, "regex": regex,
"json_schema": json_schema_lite, "number": number_within,
"tool_args": tool_args_equal,
}
print((f"{'case':<13}" + "".join(f"{name:<13}" for name in checks)).rstrip())
passed = total = 0
for label, output, wanted in cases:
row = f"{label:<13}"
for name, fn in checks.items():
if name in wanted:
ok = fn(output, *wanted[name])
passed += ok
total += 1
row += f"{'pass' if ok else 'FAIL':<13}"
else:
row += f"{'-':<13}"
print(row.rstrip())
print(f"{passed} of {total} checks passed")Output
case exact contains regex json_schema number tool_args capital FAIL pass - - - - returns - pass pass - - - ship date - - FAIL - - - profile - - - pass - - total - - - - pass - weather call - - - - - FAIL 5 of 8 checks passed
Look at the grid line by line.
capital
The answer was "Paris." with a full stop. Exact match against "Paris" fails, but contains passes. The model was right. The check was too strict.
returns
Both checks pass: the text holds "30 days", and the pattern for "some number, then days" matches too.
ship date
The pattern wants 2026-11-03 style. The answer said "3 November". It is a fair sentence, but not the format the app needs, so it fails.
profile, total, weather call
The JSON has the right keys and kinds. 12.49 is within 0.05 of 12.5. The weather call has the right tool and city but says "c" instead of "celsius", so it fails.
Overall, 5 of 8 checks passed. Three checks failed, and they are not the same kind of failure. Two point at real problems: the date format and the unit. One is an over-strict check: the full stop. A deterministic check only reports a difference. A person decides which side is wrong.
How the checks work inside
Each function is tiny, and that is the point. A few details worth knowing:
- 1
Parse before you judge
json_schema_lite first tries json.loads. Broken JSON returns False at once, with no crash.
- 2
Pull the number out
number_within finds the first number in the text with a regex, turns it into a float, then compares the gap with the tolerance.
- 3
Compare whole structures
tool_args_equal checks the tool name, then compares the whole arguments dictionary. Python's == on dictionaries means one wrong value fails the call.
- 4
Return only True or False
Every check gives one plain yes or no. That keeps the grid simple and lets you count passes.
The JSON check is called "lite" for a reason. It only checks that each key is there and its value is the right kind. Real schema tools do far more, such as nested objects and value ranges.
Run them on every change
Because the checks are cheap and steady, you can run them after every edit, the way programmers run unit tests. Change a prompt, swap a model, tweak a setting, then run the checks and compare with the last good score. The Anthropic page shows the cycle: build test cases, try the prompt, test against the cases, refine, and repeat.
Here is a tiny version. Again the model is a stand-in, a function that returns fixed text depending on a version number. A real model would be slower, would cost money per token, could answer differently each run and could refuse. We keep it fake so you can run it free.
import re
def fake_model(messages, version):
# A stand-in. It returns fixed text so this runs free.
question = messages[-1]["content"]
if "return" in question:
return "Returns are accepted within 30 days." if version == 1 else "Returns are accepted for a while."
return "Order 1042 ships on 2026-11-20."
tests = [
("How long do I have to return it?", lambda out: "30 days" in out),
("When does order 1042 ship?", lambda out: re.search(r"\d{4}-\d{2}-\d{2}", out) is not None),
]
def score(version):
results = [check(fake_model([{"role": "user", "content": q}], version)) for q, check in tests]
return sum(results), len(results)
baseline = score(1)
latest = score(2)
print("saved score:", baseline)
print("new score: ", latest)
print("ship it" if latest[0] >= baseline[0] else "STOP: the score dropped")Output
saved score: (2, 2) new score: (1, 2) STOP: the score dropped
Version 1 passed both tests. Version 2 gave a vague answer about returns, so one test failed and the score fell from 2 to 1. The last line stops the change. You did not read a single answer by hand.
Tip
Save your last good score in a file and compare against it. A drop is a warning even if the new number still looks fine.
Where exact checks run out
Now the honest part. Exact checks only see what you thought to write down. Ask a model to "explain refunds kindly" and there is no single right answer to compare with. Is the reply friendly? Clear? Accurate? A rule cannot tell.
That is the weakness the Anthropic page names: code-based grading "lacks nuance for more complex judgments". It describes model-based grading as "Fast and flexible, scalable and suitable for complex judgment", and tells you to "Test to ensure reliability first, then scale."
Too strict
Exact match marks a correct answer wrong because of a full stop or a different word order. Loosen the check, or normalize first.
Too loose
A contains check passes "Paris is not the capital" because it holds the word Paris. Check what you mean, not just a word.
Blind to quality
A reply can pass every rule and still be rude, confusing or unhelpful. Those need a judge with some sense.
So use both. Run the deterministic checks first, since they are quick and catch the clear breaks. Then use model-based evals, and sometimes people, for the open-ended qualities. The Anthropic page also says it can be better to have "More questions with slightly lower signal automated grading" than "fewer questions with high-quality human hand-graded evals".
Quick check
A model's answer is "The capital is Paris, of course!" and the golden answer is "Paris". Which check passes it?
Quick check
Why must a deterministic check give the same result every time?
Putting it together
Here is the lesson in one place.
Pick checks with one right answer
Exact match, contains, regex, JSON shape, number within tolerance, valid citations, length and tool arguments.
Run them all the time
They are cheap, fast and repeatable, so run them after every change and compare with your last good score.
Know their limit
They cannot judge tone, clarity or helpfulness. Add model-based evals and human review for those.
Next: model-based evals, using a model to grade another model.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.