esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.23

Regression Testing

How to change a prompt, a model or a setting without breaking what already worked: a fixed set of cases, run on every change, with a gate that stops bad releases.

14 min readFree

Making sure changes don't break what worked

The last lesson, Human Evals, ended with a promise: next, regression testing, making sure changes don't break what worked. Here it is.

Think of a restaurant that changes its tomatoes to save money. The soup tastes a little different, and nobody tastes the old soup beside the new one. A month later the regulars stop coming. An AI app does the same, only faster: one prompt tweak fixes one answer and quietly spoils three others.

By the end you will know what to keep in a test set, how to run it before every release, how to compare it with the last good run, how to deal with answers that change each time, and how a small gate stops a bad change from shipping. You will also run two small programs.

Key words

Regression
Something that used to work and now does not, after a change. The word means going backwards.
Regression testing
Running the same checks again after every change, to see whether anything went backwards.
Golden set
A fixed list of test cases, each with a clear rule for what a good answer looks like. "Golden" because you treat it as the trusted reference.
Baseline
The results of the last run you were happy with. New results are compared against it.
CI (continuous integration)
A robot that runs your tests automatically every time someone changes the code, so nobody has to remember to.
Gate
A rule that blocks a release when a number gets too bad, like a barrier at a car park.
Flaky
A test that passes on some runs and fails on others, with nothing changed.

What a regression is

This idea is much older than AI. The Wikipedia page on regression testing describes it as "re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change." If it does not, that is called a regression.

The same page says why this keeps happening: "Frequently, a fix for a problem in one area inadvertently causes a software bug in another area." That sentence is about ordinary software. For an AI app it is even more true, because one prompt touches every answer at once.

  • Your words

    A new prompt, a new system message, a new example, a new rule. One edit can change the tone or format of every reply.

  • Your settings

    The number of documents retrieved, the chunk size, the search method, the temperature. These look small and change a lot.

  • Your parts

    A new tool, a new tool description, a new model, a new version of the same model. The provider can change things too, even when you did not.

Notice the last item. You do not control every change. A provider can retire an older model and ask you to move to a newer one. That is a change to your app, even though you typed nothing.

Build a golden set

A golden set is the heart of regression testing. It is a list of cases. Each case has an input (what the user sends), and a rule for what counts as a good answer. The rules can be the exact checks from the Deterministic Evals lesson, the judged checks from Model-based Evals, or a mix.

  1. 1

    Start from real use

    Pull real questions from your logs. OpenAI's evaluation guide says: "Log as you develop so you can mine your logs for good eval cases."

  2. 2

    Cover three kinds

    Easy everyday cases, edge cases (odd spelling, a very long question, another language) and tricky or hostile cases. Mix them so one kind cannot hide the others.

  3. 3

    Write the rule first

    For each case, write down what a pass looks like before you run anything. If you cannot say what good means, the case is not ready.

  4. 4

    Give each case a short name

    Names like refund-policy or pii-refusal. When a case fails, the name tells you at a glance what broke.

  5. 5

    Freeze it

    Do not quietly edit a case to make a failing run pass. Change a case only on purpose, and write down why.

Tip

Every real bug becomes a new case. Wikipedia says the suite grows as bugs are found: "As regression test suites tend to grow with each found defect, test automation is frequently involved." It also calls it good practice, when a bug is found and fixed, "to record a test that exposes the bug and re-run that test regularly after subsequent changes to the program." A user complaint is the best test case you will ever get.

Run it before every release

A test set that nobody runs protects nobody. The fix is to make a robot run it.

That robot is CI, short for continuous integration. Wikipedia describes it as "the practice of integrating source code changes frequently and ensuring that the integrated codebase is in a workable state." It adds: "Typically, developers merge changes to an integration branch, and an automated system builds and tests the software system." It also says the automated process often "runs on each commit or runs on a schedule such as once a day."

A commit is one saved change to your code. So the plan is simple. You change a prompt and save it. The robot wakes up, runs your golden set, and reports. If the report is bad, the change does not go live.

Tools exist for this. The Promptfoo docs, in their CI/CD guide, list why you would run evals in CI. Two of the reasons are "Catch regressions early - Test prompt changes before they reach production" and "Quality gates - Enforce minimum performance thresholds." OpenAI's guide says the same in other words: "Set up continuous evaluation (CE) to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time."

  1. 1

    Change

    Someone edits a prompt, a setting, a tool or the model name.

  2. 2

    Run

    CI runs the golden set on the new version, with no human needed.

  3. 3

    Compare

    The results are lined up against the baseline, the last good run.

  4. 4

    Gate

    If the score dropped too far, CI fails and the release stops. If not, it moves on.

  5. 5

    Update the baseline

    When a change is released on purpose, its results become the new baseline.

When answers change every time

Here is the part that makes AI different from ordinary software. Ordinary software gives the same answer to the same input. A model may not. OpenAI's guide puts it plainly: "Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures."

Three habits keep this under control.

  • Lower the randomness

    Many providers let you set a temperature, a dial for how adventurous the wording is. For tests, set it low so answers vary less. The Sampling Parameters lesson explains it. Check your provider's docs first, because some newer models no longer accept every setting.

  • Repeat the shaky cases

    Run each flaky case several times, say five or ten, and count how many pass. One run is a coin toss. Many runs are a measurement.

  • Judge by rates

    Compare pass rates, not single results. "Eight in ten passed" is a fair way to talk about a case that is sometimes wrong.

Below is a tiny program that shows why one run misleads. The function fake_model is a stand-in so the code runs free, with no key and no internet. It is not a real model. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. This stand-in is built to be right about 8 times in 10, so you can watch the numbers wobble.

Python
import random

random.seed(7)

def fake_model(question):
    # A stand-in, not a real model. It gets this one question right
    # about 8 times in 10, to mimic an answer that changes between runs.
    if random.random() < 0.8:
        return "Refunds take 5 days."
    return "Refunds are instant."

def passes(answer):
    return "5 days" in answer

for run in range(1, 4):
    ok = sum(passes(fake_model("How long do refunds take?")) for _ in range(10))
    print("Run", run, "->", ok, "of 10 passed")

total = sum(passes(fake_model("How long do refunds take?")) for _ in range(50))
print("Over 50 tries:", total / 50 * 100, "percent passed")

Output

Run 1 -> 10 of 10 passed
Run 2 -> 8 of 10 passed
Run 3 -> 7 of 10 passed
Over 50 tries: 86.0 percent passed

Look at the three runs. The same stand-in, the same question, and the score went 10, then 8, then 7. If you had run it once and seen 10 of 10, you would have thought it perfect. If you had seen 7 of 10, you would have thought it broken. Fifty tries gave 86 percent, a steadier picture of the same stand-in.

Quick check

A flaky case passes 3 times out of 5 runs after your change, and passed 4 out of 5 before. What is the best next step?

Compare, then gate

Now put the pieces together. You have a baseline from the last good run, and a new run from the changed version. Two jobs remain.

Job two is the gate. Compute the pass rate for both runs, subtract, and stop the release if the drop is bigger than a margin you chose. The pass rate is simply the cases that passed, divided by all cases, times 100.

pass rate=cases passedall cases×100\text{pass rate} = \frac{\text{cases passed}}{\text{all cases}} \times 100

In words: if 7 of 10 cases pass, the pass rate is 70 percent. The margin gives you room for harmless wobble, so a one-in-a-hundred blip does not stop every release. The Promptfoo CI/CD guide shows the same idea. Under the heading "Quality Gates" it says: "Fail the build when quality thresholds aren't met." Its own example computes a pass rate from the results and exits with an error if the rate is under 95 percent.

The program below does the whole job for ten golden cases. It holds a baseline and a new run, gives the pass rates, lists the cases that newly fail and newly pass, and runs a gate that exits with a non-zero status when the drop is more than 5 percentage points. A percentage point is one step on the 0 to 100 scale, so going from 80 to 75 is a drop of 5 points. The results are fixed true or false values, so no model is involved.

Python
baseline = {
    "refund-policy": True, "greeting": True, "order-status": True,
    "cancel-order": True, "tone-polite": True, "pii-refusal": True,
    "json-shape": True, "long-question": False, "typo-input": True,
    "two-languages": False,
}
new_run = {
    "refund-policy": True, "greeting": True, "order-status": False,
    "cancel-order": True, "tone-polite": True, "pii-refusal": False,
    "json-shape": True, "long-question": True, "typo-input": True,
    "two-languages": False,
}
MAX_DROP = 5  # percentage points

def pass_rate(results):
    return sum(results.values()) / len(results) * 100

def compare(old, new):
    broke = [c for c in old if old[c] and not new[c]]
    fixed = [c for c in old if not old[c] and new[c]]
    return broke, fixed

def gate(old, new):
    drop = pass_rate(old) - pass_rate(new)
    if drop > MAX_DROP:
        raise SystemExit(1)
    return drop

old_rate, new_rate = pass_rate(baseline), pass_rate(new_run)
broke, fixed = compare(baseline, new_run)
print("Baseline pass rate:", old_rate, "percent")
print("New pass rate:     ", new_rate, "percent")
print("Newly failing:", broke)
print("Newly passing:", fixed)
try:
    gate(baseline, new_run)
    print("Gate: OK to release")
except SystemExit as e:
    print("Gate: FAILED, exit code", e.code)

Output

Baseline pass rate: 80.0 percent
New pass rate:      70.0 percent
Newly failing: ['order-status', 'pii-refusal']
Newly passing: ['long-question']
Gate: FAILED, exit code 1

Read the output. The baseline passed 8 of 10 cases, 80 percent. The new run passed 7 of 10, 70 percent. That is a drop of 10 points, bigger than the margin of 5, so the gate fails with exit code 1. In a real CI job, that code is what turns the build red.

The report holds a second lesson. One case got fixed (long-question) and two broke (order-status and pii-refusal). If you only looked at the story "we fixed a bug", you would feel good. The list of newly failing cases tells you what the change really cost. One of them, pii-refusal, is about protecting private data, which is far worse to lose than a long question.

Tip

A pass-rate gate can hide a serious break behind a few small wins. For cases that must never fail, such as safety or privacy, add a stricter rule: fail the build if any of those cases newly fails, whatever the overall rate says.

Quick check

Baseline pass rate is 90 percent, the margin is 5 points, and the new run scores 87 percent. What does the gate do?

Model upgrades are changes too

It is easy to forget that the model itself is part of your app. When the provider ships a new version, or when an old one is switched off, your app behaves differently even though your code did not change.

Anthropic's model deprecations page is a clear example. It says: "As safer and more capable models launch, Anthropic regularly retires older ones. Applications relying on Anthropic models may need occasional updates to keep working." For the move to a new model it advises: "consider thorough testing of your applications with the new models well before the retirement date."

That advice is regression testing. Your golden set is how you do the thorough testing. The plan is short.

  1. 1

    Know the dates

    Watch your provider's notices. Retirement dates are the deadline for your testing, not the start of it.

  2. 2

    Run the set on the new model

    Keep the prompt, the settings and the cases the same. Change one thing only, the model name.

  3. 3

    Read the diff

    Look at the newly failing list. A newer model is often better overall and still worse on a few of your cases.

  4. 4

    Fix, then repeat

    Adjust the prompt for the cases that broke, and run again. Do not touch the model and the prompt in the same step, or you will not know which one moved the score.

  5. 5

    Switch over on purpose

    When the gate passes, release it, and save the new results as the baseline.

Habits that keep it working

  • Fast and always on

    A suite people skip is worth nothing. Keep a small quick set for every change and the full set before each release.

  • One change at a time

    Change the prompt, or the model, or a setting. When two things move together, a drop could be either, and you cannot tell.

  • Never hide a failure

    Do not loosen a rule just to turn the build green. If a case is truly out of date, retire it on purpose and write down why.

Tip

Wikipedia notes that failures in a regression run might mean a real regression "or an out-of-date test". When a case fails, ask both questions before you panic: did the app break, or did the world change and the case stay behind?

Quick check

A user reports a wrong answer. After you fix it, what should you also do?

To sum up: keep a golden set, run it on every change, compare with the last good run, judge by rates, and let a gate stop what is worse. Then every prompt tweak and model upgrade becomes something you can do without holding your breath.

Next: evaluation metrics, the numbers that sum up quality.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.