esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.18

Production monitoring

Keep a live AI feature healthy: what to watch, when to raise an alarm, and how to switch to a backup when something breaks.

13 min readFree

Shipping is the start, not the end

Last lesson, LLM Observability, gave you a way to see inside each request. Production monitoring uses that view to keep a live AI feature healthy. Think of a restaurant. Cooking one good meal is the easy part. Keeping the kitchen running all night, with a broken oven, a late supplier and a Saturday rush, is the real job.

An AI feature is the same. It works on your laptop. Then real people use it, the model provider changes something, and nobody tells you. Monitoring is how you find out before your users do.

Key words

Monitoring
Counting what your feature does all the time, and showing it as numbers and charts.
Metric
One number that you track over time, like how many requests failed this hour.
Baseline
What "normal" looks like for a metric. You compare today against it.
Alert
A message sent to a person when a metric goes past a limit.
Feature flag
A switch in your settings that turns a feature on or off without shipping new code.
Fallback
A backup plan, such as a second model that takes over when the first one fails.

The four golden signals

Engineers at Google wrote down a short list of numbers that every live service should watch. The Google SRE book says it like this: "The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four."

  • Latency

    The book defines it as "The time it takes to service a request." For an AI feature, this is how long the user waits for an answer.

  • Traffic

    The book calls it "A measure of how much demand is being placed on your system". For you: requests per minute, and tokens per request.

  • Errors

    The book counts requests that fail "explicitly", "implicitly" (for example, "an HTTP 200 success response, but coupled with the wrong content") or "by policy".

The fourth signal is saturation, which the book describes as how "full" your service is. For an AI feature this often means how close you are to your provider's rate limit. A rate limit is a cap on how many requests you may send in a minute.

Look at that errors line again. A request can succeed on paper and still be wrong. A model can answer 200 OK with an empty reply or a polite refusal. AI features hide many failures inside "successful" requests, so you need extra numbers.

error rate=failed requestsall requests\text{error rate} = \frac{\text{failed requests}}{\text{all requests}}

In words: divide the requests that failed by all the requests, and you get the share that went wrong. If 4 in 200 fail, the error rate is 2%.

Tip

The book also warns: "a slow error is even worse than a fast error!" A request that hangs for 30 seconds and then fails is the worst of both. Track the latency of failures too.

What to watch on an AI feature

Start with the four signals, then add the numbers that only an AI feature has. Here they are, in three groups.

  • Is it working?

    Error rate, rate-limit errors (the provider saying "slow down"), timeouts, and latency. Watch the slowest requests, not only the average. One slow request in twenty still ruins the day for those users.

  • Is it good?

    Refusals ("I can't help with that"), empty answers, user feedback such as thumbs up and thumbs down, and eval scores. An eval is a small test that grades an answer. Run one on a sample of live traffic, say 1 in 50 requests.

  • Is it safe and affordable?

    Cost per request, tokens used, and moderation flags, which are the times a safety filter caught something. Cost matters because a real model charges for every token.

You can read your cost the same way. One provider's production guide suggests setting a notification threshold: you "receive an email alert once you pass a certain usage threshold". Do this on day one. A loop bug can burn a month of budget over a weekend.

Drift: the ground moves under you

Drift means things change after you shipped. Wikipedia says concept drift "is an evolution of data that invalidates the data model", and adds: "This causes problems because the predictions become less accurate as time passes." Picture a map. The roads were right on the day it was printed. A year later, new roads exist and the map quietly gets worse.

For an AI feature, drift comes from two places.

Your users drift

People ask new kinds of questions. A new product launches, a new language shows up, a news event changes what everyone wants to know. Your prompt was tuned for last month's questions. Watch simple facts about the inputs, like average length and the topics people ask about.

The provider drifts

The company behind the model can change it. Even without a big announcement, answers can shift. The OpenAI docs say it plainly: "Even different snapshots of models within the same family could produce different results." A snapshot is one fixed version of a model, like a numbered edition of a book.

So where the provider allows it, pin your version. The same docs recommend "Pinning your production applications to specific model snapshots" to "ensure consistent behavior". Pinning means you write the exact version name in your settings, so it cannot change under you. You then upgrade on purpose, after testing, not by surprise.

Tip

The same page also recommends "Building tests and evaluation suites that measure prompt behavior so you can monitor performance as you iterate, or when you change and upgrade model versions". Keep a fixed set of test questions. Run it before every upgrade and compare.

Alerts that help, not nag

An alert should mean "a person needs to act now". The SRE book is blunt about the cost: "Paging a human is a quite expensive use of an employee's time." It also warns that "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts". This is the boy who cried wolf, with pagers.

Its advice for the rules: "Rules that generate alerts for humans should be simple to understand and represent a clear failure." So keep each rule to one sentence, with one number.

Here is a made-up starter set. Your own numbers come from your own baseline, so treat these as examples only.

Text
ALERT RULES (made-up numbers, for illustration)

errors   more than 10% of requests fail in one hour     -> page a person, turn on fallback
errors   daily rate above 2x the baseline               -> ticket for tomorrow
refusals daily rate above 2x the baseline               -> ticket, check the prompt and model
latency  slowest 5% of requests take over 20 seconds    -> page a person
cost     spend today above 150% of a normal day          -> email the owner

Notice two levels. A big, fast problem pages a person right away. A slow, small problem becomes a ticket for the next working day. If everything pages, nothing gets read.

Dashboards: one screen, honest numbers

The SRE book says a dashboard "provides a summary view of a service's core metrics", prebuilt to show what matters most. Keep it to one screen.

  1. 1

    Top row: the health lights

    Error rate, refusal rate, latency of the slowest requests, and cost today. Each one shows its baseline next to it.

  2. 2

    Middle: the last 24 hours

    A line for each number by hour, so a bad afternoon stands out.

  3. 3

    Bottom: the humans

    User thumbs up and down, eval score on live samples, and a list of recent alerts.

Always draw the baseline on the chart. A number alone, such as "6% refusals", tells you nothing. "6% against a normal 3%" tells you a lot.

When it breaks: a playbook

An incident is a time when your feature is hurting users. The best moment to decide what to do is before it happens. Write the steps down and keep them near the dashboard.

  1. 1

    Check the symptom

    The SRE book says to ask two questions, what is broken and why. "The 'what's broken' indicates the symptom; the 'why' indicates a (possibly intermediate) cause." Start with the symptom: errors up, or refusals up?

  2. 2

    Switch to the fallback

    Send traffic to a second model, or to a simpler non-AI answer, using a feature flag. Wikipedia describes a feature toggle this way: "A condition within the code enables or disables a feature during runtime." No new release is needed.

  3. 3

    Or turn the feature off

    Wikipedia also names the kill switch: it "lets operators disable a feature at runtime, for example to mitigate an incident or reduce load". Show users a friendly message and a normal way to do the task.

  4. 4

    Find the why, then fix it

    Now, with users safe, check the provider's status, your last deploy, and your prompt. Flip the flag back only when you know it is fine.

  5. 5

    Write down what you learned

    Add a new alert or a new test, so the same surprise cannot catch you twice.

Rolling out changes safely

Most outages start with a change: a new prompt, a new model version, a new tool. Three ways to try a change without betting everything on it.

  • Shadow test

    The new version runs on real requests, but its answers are thrown away or only stored. Users still see the old one. You compare quietly, with no risk.

  • Gradual rollout

    Give the change to a few users first, then more. Wikipedia calls this a canary release, which "allows developers to have features incrementally tested by a small set of users".

  • A/B test

    Half your users get version A and half get version B at the same time, and you compare their numbers: feedback, refusals, cost. Wikipedia notes that feature toggles make "canary roll out and A/B testing" easier to handle.

The same flag you use for incidents also drives rollouts. You turn the new version on for 5% of users, watch the dashboard, then 25%, then everyone. If a number jumps, you flip it back.

Worked example: a tiny monitor

Time to build one. This script watches three made-up days of hourly numbers. It works out the error rate and refusal rate per day, compares them to a baseline, and raises alerts. It also flips a feature flag to a fallback model when the hourly error alert fires.

Important: there is no model in this code. It is a stand-in. The numbers come from a random generator with a fixed seed, so the code runs free and gives the same result every time. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. We simulate those failures so you can see how a monitor reacts.

Python
import random

random.seed(12)

BASE_ERROR = 0.02      # normal share of failed requests
BASE_REFUSAL = 0.03    # normal share of "I can't help with that"
DAILY_LIMIT = 2        # daily alert: rate above 2 x baseline
HOURLY_ERROR_LIMIT = 0.10  # hourly alert: more than 1 in 10 fail

def true_rates(day, hour):
    """Made-up truth: the chance that one request fails or is refused."""
    error, refusal = BASE_ERROR, BASE_REFUSAL
    if day == 2 and 14 <= hour < 18:
        error = 0.30       # the provider has a bad afternoon
    if day == 3 and hour >= 9:
        refusal = 0.12     # a model update makes it say "no" more often
    return error, refusal

use_fallback = False       # the feature flag
timeline = []
days = {}

for day in (1, 2, 3):
    total = errors = refusals = 0
    for hour in range(24):
        n = random.randint(150, 250)          # requests this hour
        p_error, p_refusal = true_rates(day, hour)
        if use_fallback:                      # the fallback model is healthy
            p_error, p_refusal = BASE_ERROR, BASE_REFUSAL
        e = sum(random.random() < p_error for _ in range(n))
        r = sum(random.random() < p_refusal for _ in range(n))
        total, errors, refusals = total + n, errors + e, refusals + r

        if not use_fallback and e / n > HOURLY_ERROR_LIMIT:
            use_fallback = True
            timeline.append(f"day {day} {hour:02d}:00  ALERT errors {e / n:.0%} this hour -> flag ON, fallback model")
        elif use_fallback and day == 2 and hour == 19:
            use_fallback = False              # a person checks the provider, then flips it back
            timeline.append(f"day {day} {hour:02d}:00  provider looks fine -> flag OFF, main model")
    days[day] = (errors / total, refusals / total)
    for name, rate, base in (("error", days[day][0], BASE_ERROR), ("refusal", days[day][1], BASE_REFUSAL)):
        if rate > DAILY_LIMIT * base:
            timeline.append(f"day {day} 23:59  ALERT daily {name} rate {rate:.1%} vs baseline {base:.0%}")

print("TIMELINE")
for line in timeline:
    print(line)
print()
print("day  error  refusal")
for day, (e, r) in days.items():
    print(f"{day}    {e:.1%}  {r:.1%}")

Output

TIMELINE
day 2 14:00  ALERT errors 25% this hour -> flag ON, fallback model
day 2 19:00  provider looks fine -> flag OFF, main model
day 3 23:59  ALERT daily refusal rate 8.8% vs baseline 3%

day  error  refusal
1    1.7%  3.3%
2    3.1%  3.2%
3    2.2%  8.8%

Read the timeline. On day 2 at 14:00 the hourly error rate hit 25%. That is far above the 10% limit, so the alert fired and the flag switched to the fallback. A person checked the provider at 19:00 and switched the main model back on.

Now look at the table. Day 2's error rate for the whole day was only 3.1%, which is under the daily limit of 2 times the 2% baseline. A daily check would have missed it. The hourly check caught the bad afternoon.

Day 3 shows the other kind of problem. Nothing crashed, and errors stayed near 2.2%. But refusals rose to 8.8% against a baseline of 3%, and the daily alert fired at the end of the day. In our story a model update started refusing more. Waiting a whole day to learn that is slow, so a real team would also check refusals hourly.

Quick checks

Quick check

Your model starts answering many questions with "Sorry, I can't help with that." The request status is still "success". What catches this?

Quick check

Why pin your production app to a specific model snapshot?

Quick check

The error alert fires at 2 a.m. What is the best first move?

Recap

A live AI feature needs a baseline, a short list of numbers, a few honest alerts, one clear dashboard and a switch you can flip. Start with the four golden signals, then add refusals, empty answers, cost and eval scores. Pin model versions where you can, test before you upgrade, and roll out changes in small steps.

Next: LLM evaluations, testing quality before and after you ship.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.