AI engineering / Advanced / 03.25
Eval tools
Libraries that run your evals for you. See what DeepEval and RAGAS do, what to look for in any eval tool, and build a tiny runner yourself.
14 min readFree
Let a library do the boring part
The last lesson, Evaluation Metrics, ended with a promise: next, eval tools, libraries that run all of this for you. Here they are.
Think of a school exam. The questions, the marking rule and the pass mark are the real work. Handing out papers, adding up marks and printing the results is just routine. An eval tool does the routine part, so you can spend your time on the questions and the marking rule.
By the end you will know what two open-source eval libraries, DeepEval and RAGAS, do. You will have six things to check before you pick any eval tool. And you will have written a small runner of your own, in plain Python, that works the same way.
Key words
- Eval tool
- A library that runs your evals: it holds the test cases, scores each one and shows the results.
- Test case
- One example to check: the question you asked, the answer your app gave, and often the answer you hoped for.
- Metric
- A rule that turns an answer into a score, usually from 0 to 1. It is the ruler you measure with.
- Threshold
- The lowest score that still counts as a pass.
- Judge model
- A second model that reads an answer and scores it. The Model-Based Evals lesson covers how that works.
- CI
- Continuous integration: a robot that runs your checks every time someone changes the code.
- Open source
- The code is public. You can read it, run it and change it for free.
What an eval tool does for you
You already know the parts of an eval from the LLM Evaluations lesson: examples, scoring and a report. An eval tool packs those parts into a library. The jobs are always the same.
- 1
Hold the test cases
Each case is one example, written as code so it can sit in your project and be reviewed like any other file.
- 2
Score each case
A metric reads the case and gives a score. Some metrics are plain code. Others ask a judge model.
- 3
Compare with the pass mark
Each case has a threshold. Score at or above it, and the case passes.
- 4
Report
Print or save a summary: what passed, what failed and by how much.
- 5
Set the exit signal
Tell the robot in CI whether the whole run passed, so a bad change can be stopped.
None of this is magic. To show that, let us build the small version first. Then the real libraries will look familiar.
Worked example: a mini eval runner
Pytest is a popular Python tool that finds your test functions and runs them. We will not use it. We will write the same idea in about forty lines.
A decorator is a line starting with @ placed above a function. It adds something to the function. Ours will add each test function to a list, together with its pass mark. Then a runner loops over the list, collects the scores and prints a table.
Two parts of this code are stand-ins. The model, `fake_model`, just looks up a fixed answer, so the code runs free with no key and no network. The metric, `word_overlap`, only counts how many of the expected words appear in the answer. It is not a real quality score. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. A real metric is often a judge model, which has the same problems.
import re
from dataclasses import dataclass
# Stand-in app so the code runs free. It is NOT a real model.
def fake_model(question):
answers = {
"How long do refunds take?": "Refunds take 5 business days.",
"Can I change my plan?": "Yes, you can change your plan any time.",
"Do you ship abroad?": "We love helping customers.",
}
return answers[question]
@dataclass
class TestCase: # same field names as the tools' test cases
input: str
actual_output: str
expected_output: str
# Stand-in metric: share of the expected words found in the answer (0 to 1).
def word_overlap(case):
words = lambda text: set(re.findall(r"[a-z0-9]+", text.lower()))
expected = words(case.expected_output)
return len(expected & words(case.actual_output)) / len(expected)
TESTS = [] # the registry: every decorated function lands here
def eval_case(threshold):
def register(fn):
TESTS.append((fn.__name__, fn, threshold))
return fn
return register
@eval_case(threshold=0.8)
def test_refund_time():
q = "How long do refunds take?"
return word_overlap(TestCase(q, fake_model(q), "Refunds take 5 business days"))
@eval_case(threshold=0.8)
def test_change_plan():
q = "Can I change my plan?"
return word_overlap(TestCase(q, fake_model(q), "You can change your plan any time"))
@eval_case(threshold=0.5)
def test_shipping():
q = "Do you ship abroad?"
return word_overlap(TestCase(q, fake_model(q), "We ship to most countries"))
def run_all():
rows = []
for name, fn, threshold in TESTS:
score = fn()
rows.append((name, score, threshold, score >= threshold))
print(f"{'test':<20}{'score':>6}{'need':>6} result")
for name, score, threshold, ok in rows:
print(f"{name:<20}{score:>6.2f}{threshold:>6.2f} {'PASS' if ok else 'FAIL'}")
passed = sum(ok for *_, ok in rows)
print(f"{passed} of {len(rows)} passed")
return passed == len(rows)
raise SystemExit(0 if run_all() else 1) # a CI job reads this exit code
Output
test score need result test_refund_time 1.00 0.80 PASS test_change_plan 1.00 0.80 PASS test_shipping 0.20 0.50 FAIL 2 of 3 passed
Read the table. The first two cases score 1.00 because every expected word appears in the fake answer. Both beat their pass mark of 0.80, so they pass.
The shipping case scores 0.20. The expected answer has five words, and the fake answer shares only one of them, "we". That is 1 out of 5. The pass mark was 0.50, so it fails. The summary says 2 of 3 passed.
The last line is the CI part. `raise SystemExit(1)` ends the program with exit code 1, which means failure. A robot in CI sees that code and marks the whole change as failed. That is how a bad prompt gets stopped before it ships.
Tip
Different cases can have different pass marks. A refund policy must be exact, so it gets a high mark. A friendly greeting can have a lower one. Pick each mark on purpose.
DeepEval
DeepEval is an open-source library for testing AI apps in Python. Its footer calls it an open-source LLM evaluation framework with an Apache 2.0 licence. It plugs into pytest, the tool we just imitated: the quickstart says that `deepeval test run` collects and runs your eval files the same way `pytest` would.
Its docs split the work into two things. The test case holds the example. The metric is the ruler: the Metrics page says that while a test case represents the thing you are trying to measure, the metric acts as the ruler for specific criteria of interest.
The test case
The quickstart says an LLM test case represents a single unit of LLM app interaction. It must have an input and an actual_output (what your app answered). An expected_output, the answer you hoped for, is optional.
The metric
The Metrics page lists 50+ ready-to-use metrics. All of them output a score between 0 and 1. It also says almost all of the predefined ones use LLM-as-a-judge, with the exception of JevEval, which is scored by a different kind of model.
The threshold
A metric succeeds only when its score is equal to or greater than the threshold, which is 0.5 by default. You can set your own, as the snippet below does.
The metrics come in families. For retrieval apps (the RAG lessons covered those), the docs list retriever metrics such as Contextual Relevancy, Contextual Precision and Contextual Recall, and generator metrics such as Answer Relevancy and Faithfulness. There are also metrics for agents, chatbots and safety.
Here is how the docs write one test. The first line says where it comes from and that we did not run it. Running it needs a judge model and its key, because the quickstart tells you to set an OPENAI_API_KEY environment variable or configure another model.
# From the DeepEval 5-min Quickstart, shown without its output (not run here)
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, SingleTurnParams
from deepeval.metrics import GEval
def test_correctness():
correctness_metric = GEval(
name="Correctness",
criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
threshold=0.5
)
test_case = LLMTestCase(
input="I have a persistent cough and fever. Should I be worried?",
# Replace this with the actual output from your LLM application
actual_output="A persistent cough and fever could be a viral infection or something more serious. See a doctor if symptoms worsen or don't improve in a few days.",
expected_output="A persistent cough and fever could indicate a range of illnesses, from a mild viral infection to more serious conditions like pneumonia or COVID-19. You should seek medical attention if your symptoms worsen, persist for more than a few days, or are accompanied by difficulty breathing, chest pain, or other concerning signs."
)
assert_test(test_case, [correctness_metric])Compare it with our runner. It is the same shape. A function named test_something builds a test case, picks a metric with a threshold, and asserts that the case passes. The criteria is written in everyday language: the judge model reads it and scores the answer.
Then you run `deepeval test run test_example.py` in your terminal. The docs say the same command can run in CI, and that results are cached locally.
RAGAS
Ragas (the docs spell it Ragas) is another open-source Python library. Its README describes it as objective metrics, intelligent test generation and data-driven insights for LLM apps. Many of its metrics check RAG apps, which you met in the retrieval lessons. A RAG app first fetches notes, then writes an answer from them.
Two things stand out in its docs. First, test data generation: if you have no test set yet, the README says it can create one for you. Second, its metric list. The Available Metrics page groups metrics by job: retrieval augmented generation, agents and tool use, comparing text, SQL and general purpose. It also says LLM-based metrics might use one or more LLM calls to arrive at the score, and the README mentions traditional metrics too.
One RAG metric is easy to understand, and it uses a formula. Faithfulness asks: is every claim in the answer backed up by the notes the app fetched? The docs say to find all the claims in the response, check each one against the retrieved context, and then divide.
In words: if an answer makes four claims and the notes back three of them, faithfulness is 3 divided by 4, which is 0.75. The docs say the score runs from 0 to 1, and higher means more consistent. A low score warns you that the answer may contain things the notes never said.
Here is the docs' own example. As before, the first line shows where it is from, and we did not run it. It needs a real model behind it, with a key.
# From the Ragas docs (Faithfulness), shown without its output (not run here)
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness
# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)
# Create metric
scorer = Faithfulness(llm=llm)
# Evaluate
result = await scorer.ascore(
user_input="When was the first super bowl?",
response="The first superbowl was held on Jan 15, 1967",
retrieved_contexts=[
"The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
],
)
print(f"Faithfulness Score: {result.value}")The shape is again familiar, with different names. The user_input is our input. The response is our actual_output. The retrieved_contexts are the notes the app fetched, which a RAG check needs and a plain question-and-answer check does not. The scorer is the metric.
Notice one difference from DeepEval's example. This one is not a pytest test. It scores one example and prints the value. You would put it in a loop, or in a test, yourself. Different tools give you different amounts of ready-made structure.
The same idea in every tool
Put our runner next to the two libraries and you can see one idea in three costumes.
Our runner
TestCase with input, actual_output and expected_output. A word_overlap function as the metric. A threshold in the decorator. A table at the end.
DeepEval
LLMTestCase with the same field names. A metric such as GEval with a threshold. assert_test as the check. deepeval test run as the runner.
Ragas
A sample with user_input, response and retrieved_contexts. A metric object such as Faithfulness. A score you read from result.value.
That is why a hand-made harness is a fine way to start. When you outgrow it, moving to a library is mostly renaming things.
What to look for in any eval tool
New eval tools appear often, and docs change. Do not memorise tool names. Memorise what to ask. Here are six questions, in two groups of three.
Test cases as code
Can you write cases in files, keep them in git and review them like any code? If cases only live in a web page, changes are harder to track.
Ready metrics
Are there metrics for your kind of app, such as RAG or agents? Can you also write your own? DeepEval's Metrics page, for one, includes custom metrics, even fully self-coded ones.
Judge-model support
Can you pick which model judges the answers? Remember the Model-Based Evals lesson: a judge is itself a model, so it can be wrong and it costs tokens.
CI integration
Does it return a pass or fail signal a robot can read? DeepEval's CI page describes running the same evals on every push or every pull request.
Reports
Can you see what changed since last time? You want to compare runs side by side and catch a case that used to pass.
Data privacy
Where does your data go? Judging calls send your questions and answers to whichever model you set up as the judge. Also check what the tool itself reports back.
On privacy, both libraries are open about it. DeepEval's Data Privacy page says it uses PostHog to track only very basic telemetry data (event names, metric names, an anonymous random ID) and excludes personal information. You can switch it off with the environment variable DEEPEVAL_TELEMETRY_OPT_OUT set to 1. The Ragas README has an Open Analytics section that collects minimal, anonymised usage data and says you can opt out by setting RAGAS_DO_NOT_TRACK to true.
Telemetry is the tool's own small report about how it is used. The bigger question is the judging calls. If your test cases hold real customer messages, those messages travel to the judge's provider. Test with made-up data when you can, and read the provider's data terms before you use real text.
Tip
Tool facts in this lesson were checked on 16 November 2026. Versions and features change, so open the docs before you choose.
Build or borrow?
Build a small runner when you have a handful of cases and one simple metric. It teaches you the most and has nothing to learn or update.
Borrow a library when you need ready metrics for retrieval or agents, test data generation, saved reports or a CI setup you do not want to write. In both cases the hard part stays the same: choosing good questions and a fair marking rule. No library does that for you.
Quick checks
Quick check
In an eval tool, what does the threshold do?
Quick check
Your test cases hold real customer messages and use a judge model. Where do those messages go?
Quick check
Why does a CI robot need an exit code from your eval runner?
What to remember
Tools hold the routine
They keep test cases, score them, compare with a pass mark, report and signal CI. The questions and the marking rule are still yours.
One shape everywhere
An input, an answer, maybe an expected answer, a metric and a threshold. DeepEval and Ragas dress it differently, and so did our mini runner.
Ask the six questions
Cases as code, ready metrics, judge support, CI, reports and data privacy. Check them on any tool, today or in two years.
Next: multimodal AI, models that work with images, audio and video.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.