AI engineering / Advanced / 03.18
Production monitoring
Keep a live AI feature healthy: what to watch, when to raise an alarm, and how to switch to a backup when something breaks.
13 min readFree
Shipping is the start, not the end
Last lesson, LLM Observability, gave you a way to see inside each request. Production monitoring uses that view to keep a live AI feature healthy. Think of a restaurant. Cooking one good meal is the easy part. Keeping the kitchen running all night, with a broken oven, a late supplier and a Saturday rush, is the real job.
An AI feature is the same. It works on your laptop. Then real people use it, the model provider changes something, and nobody tells you. Monitoring is how you find out before your users do.
Key words
- Monitoring
- Counting what your feature does all the time, and showing it as numbers and charts.
- Metric
- One number that you track over time, like how many requests failed this hour.
- Baseline
- What "normal" looks like for a metric. You compare today against it.
- Alert
- A message sent to a person when a metric goes past a limit.
- Feature flag
- A switch in your settings that turns a feature on or off without shipping new code.
- Fallback
- A backup plan, such as a second model that takes over when the first one fails.
The four golden signals
Engineers at Google wrote down a short list of numbers that every live service should watch. The Google SRE book says it like this: "The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four."
Latency
The book defines it as "The time it takes to service a request." For an AI feature, this is how long the user waits for an answer.
Traffic
The book calls it "A measure of how much demand is being placed on your system". For you: requests per minute, and tokens per request.
Errors
The book counts requests that fail "explicitly", "implicitly" (for example, "an HTTP 200 success response, but coupled with the wrong content") or "by policy".
The fourth signal is saturation, which the book describes as how "full" your service is. For an AI feature this often means how close you are to your provider's rate limit. A rate limit is a cap on how many requests you may send in a minute.
Look at that errors line again. A request can succeed on paper and still be wrong. A model can answer 200 OK with an empty reply or a polite refusal. AI features hide many failures inside "successful" requests, so you need extra numbers.
In words: divide the requests that failed by all the requests, and you get the share that went wrong. If 4 in 200 fail, the error rate is 2%.
Tip
The book also warns: "a slow error is even worse than a fast error!" A request that hangs for 30 seconds and then fails is the worst of both. Track the latency of failures too.
What to watch on an AI feature
Start with the four signals, then add the numbers that only an AI feature has. Here they are, in three groups.
Is it working?
Error rate, rate-limit errors (the provider saying "slow down"), timeouts, and latency. Watch the slowest requests, not only the average. One slow request in twenty still ruins the day for those users.
Is it good?
Refusals ("I can't help with that"), empty answers, user feedback such as thumbs up and thumbs down, and eval scores. An eval is a small test that grades an answer. Run one on a sample of live traffic, say 1 in 50 requests.
Is it safe and affordable?
Cost per request, tokens used, and moderation flags, which are the times a safety filter caught something. Cost matters because a real model charges for every token.
You can read your cost the same way. One provider's production guide suggests setting a notification threshold: you "receive an email alert once you pass a certain usage threshold". Do this on day one. A loop bug can burn a month of budget over a weekend.
Drift: the ground moves under you
Drift means things change after you shipped. Wikipedia says concept drift "is an evolution of data that invalidates the data model", and adds: "This causes problems because the predictions become less accurate as time passes." Picture a map. The roads were right on the day it was printed. A year later, new roads exist and the map quietly gets worse.
For an AI feature, drift comes from two places.
Your users drift
People ask new kinds of questions. A new product launches, a new language shows up, a news event changes what everyone wants to know. Your prompt was tuned for last month's questions. Watch simple facts about the inputs, like average length and the topics people ask about.
The provider drifts
The company behind the model can change it. Even without a big announcement, answers can shift. The OpenAI docs say it plainly: "Even different snapshots of models within the same family could produce different results." A snapshot is one fixed version of a model, like a numbered edition of a book.
So where the provider allows it, pin your version. The same docs recommend "Pinning your production applications to specific model snapshots" to "ensure consistent behavior". Pinning means you write the exact version name in your settings, so it cannot change under you. You then upgrade on purpose, after testing, not by surprise.
Tip
The same page also recommends "Building tests and evaluation suites that measure prompt behavior so you can monitor performance as you iterate, or when you change and upgrade model versions". Keep a fixed set of test questions. Run it before every upgrade and compare.
Alerts that help, not nag
An alert should mean "a person needs to act now". The SRE book is blunt about the cost: "Paging a human is a quite expensive use of an employee's time." It also warns that "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts". This is the boy who cried wolf, with pagers.
Its advice for the rules: "Rules that generate alerts for humans should be simple to understand and represent a clear failure." So keep each rule to one sentence, with one number.
Here is a made-up starter set. Your own numbers come from your own baseline, so treat these as examples only.
ALERT RULES (made-up numbers, for illustration)
errors more than 10% of requests fail in one hour -> page a person, turn on fallback
errors daily rate above 2x the baseline -> ticket for tomorrow
refusals daily rate above 2x the baseline -> ticket, check the prompt and model
latency slowest 5% of requests take over 20 seconds -> page a person
cost spend today above 150% of a normal day -> email the ownerNotice two levels. A big, fast problem pages a person right away. A slow, small problem becomes a ticket for the next working day. If everything pages, nothing gets read.
Dashboards: one screen, honest numbers
The SRE book says a dashboard "provides a summary view of a service's core metrics", prebuilt to show what matters most. Keep it to one screen.
- 1
Top row: the health lights
Error rate, refusal rate, latency of the slowest requests, and cost today. Each one shows its baseline next to it.
- 2
Middle: the last 24 hours
A line for each number by hour, so a bad afternoon stands out.
- 3
Bottom: the humans
User thumbs up and down, eval score on live samples, and a list of recent alerts.
Always draw the baseline on the chart. A number alone, such as "6% refusals", tells you nothing. "6% against a normal 3%" tells you a lot.
When it breaks: a playbook
An incident is a time when your feature is hurting users. The best moment to decide what to do is before it happens. Write the steps down and keep them near the dashboard.
- 1
Check the symptom
The SRE book says to ask two questions, what is broken and why. "The 'what's broken' indicates the symptom; the 'why' indicates a (possibly intermediate) cause." Start with the symptom: errors up, or refusals up?
- 2
Switch to the fallback
Send traffic to a second model, or to a simpler non-AI answer, using a feature flag. Wikipedia describes a feature toggle this way: "A condition within the code enables or disables a feature during runtime." No new release is needed.
- 3
Or turn the feature off
Wikipedia also names the kill switch: it "lets operators disable a feature at runtime, for example to mitigate an incident or reduce load". Show users a friendly message and a normal way to do the task.
- 4
Find the why, then fix it
Now, with users safe, check the provider's status, your last deploy, and your prompt. Flip the flag back only when you know it is fine.
- 5
Write down what you learned
Add a new alert or a new test, so the same surprise cannot catch you twice.
Rolling out changes safely
Most outages start with a change: a new prompt, a new model version, a new tool. Three ways to try a change without betting everything on it.
Shadow test
The new version runs on real requests, but its answers are thrown away or only stored. Users still see the old one. You compare quietly, with no risk.
Gradual rollout
Give the change to a few users first, then more. Wikipedia calls this a canary release, which "allows developers to have features incrementally tested by a small set of users".
A/B test
Half your users get version A and half get version B at the same time, and you compare their numbers: feedback, refusals, cost. Wikipedia notes that feature toggles make "canary roll out and A/B testing" easier to handle.
The same flag you use for incidents also drives rollouts. You turn the new version on for 5% of users, watch the dashboard, then 25%, then everyone. If a number jumps, you flip it back.
Worked example: a tiny monitor
Time to build one. This script watches three made-up days of hourly numbers. It works out the error rate and refusal rate per day, compares them to a baseline, and raises alerts. It also flips a feature flag to a fallback model when the hourly error alert fires.
Important: there is no model in this code. It is a stand-in. The numbers come from a random generator with a fixed seed, so the code runs free and gives the same result every time. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. We simulate those failures so you can see how a monitor reacts.
import random
random.seed(12)
BASE_ERROR = 0.02 # normal share of failed requests
BASE_REFUSAL = 0.03 # normal share of "I can't help with that"
DAILY_LIMIT = 2 # daily alert: rate above 2 x baseline
HOURLY_ERROR_LIMIT = 0.10 # hourly alert: more than 1 in 10 fail
def true_rates(day, hour):
"""Made-up truth: the chance that one request fails or is refused."""
error, refusal = BASE_ERROR, BASE_REFUSAL
if day == 2 and 14 <= hour < 18:
error = 0.30 # the provider has a bad afternoon
if day == 3 and hour >= 9:
refusal = 0.12 # a model update makes it say "no" more often
return error, refusal
use_fallback = False # the feature flag
timeline = []
days = {}
for day in (1, 2, 3):
total = errors = refusals = 0
for hour in range(24):
n = random.randint(150, 250) # requests this hour
p_error, p_refusal = true_rates(day, hour)
if use_fallback: # the fallback model is healthy
p_error, p_refusal = BASE_ERROR, BASE_REFUSAL
e = sum(random.random() < p_error for _ in range(n))
r = sum(random.random() < p_refusal for _ in range(n))
total, errors, refusals = total + n, errors + e, refusals + r
if not use_fallback and e / n > HOURLY_ERROR_LIMIT:
use_fallback = True
timeline.append(f"day {day} {hour:02d}:00 ALERT errors {e / n:.0%} this hour -> flag ON, fallback model")
elif use_fallback and day == 2 and hour == 19:
use_fallback = False # a person checks the provider, then flips it back
timeline.append(f"day {day} {hour:02d}:00 provider looks fine -> flag OFF, main model")
days[day] = (errors / total, refusals / total)
for name, rate, base in (("error", days[day][0], BASE_ERROR), ("refusal", days[day][1], BASE_REFUSAL)):
if rate > DAILY_LIMIT * base:
timeline.append(f"day {day} 23:59 ALERT daily {name} rate {rate:.1%} vs baseline {base:.0%}")
print("TIMELINE")
for line in timeline:
print(line)
print()
print("day error refusal")
for day, (e, r) in days.items():
print(f"{day} {e:.1%} {r:.1%}")Output
TIMELINE day 2 14:00 ALERT errors 25% this hour -> flag ON, fallback model day 2 19:00 provider looks fine -> flag OFF, main model day 3 23:59 ALERT daily refusal rate 8.8% vs baseline 3% day error refusal 1 1.7% 3.3% 2 3.1% 3.2% 3 2.2% 8.8%
Read the timeline. On day 2 at 14:00 the hourly error rate hit 25%. That is far above the 10% limit, so the alert fired and the flag switched to the fallback. A person checked the provider at 19:00 and switched the main model back on.
Now look at the table. Day 2's error rate for the whole day was only 3.1%, which is under the daily limit of 2 times the 2% baseline. A daily check would have missed it. The hourly check caught the bad afternoon.
Day 3 shows the other kind of problem. Nothing crashed, and errors stayed near 2.2%. But refusals rose to 8.8% against a baseline of 3%, and the daily alert fired at the end of the day. In our story a model update started refusing more. Waiting a whole day to learn that is slow, so a real team would also check refusals hourly.
Quick checks
Quick check
Your model starts answering many questions with "Sorry, I can't help with that." The request status is still "success". What catches this?
Quick check
Why pin your production app to a specific model snapshot?
Quick check
The error alert fires at 2 a.m. What is the best first move?
Recap
A live AI feature needs a baseline, a short list of numbers, a few honest alerts, one clear dashboard and a switch you can flip. Start with the four golden signals, then add refusals, empty answers, cost and eval scores. Pin model versions where you can, test before you upgrade, and roll out changes in small steps.
Next: LLM evaluations, testing quality before and after you ship.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.