AI guardrails & red teaming / Core / 02.14
Continuous monitoring
Watch your live AI app for attacks: what to write down, what to keep out, which signs matter, and what to do when an alert fires.
13 min readFree
Testing is a photo. Monitoring is a camera.
The last lesson, Automated vs manual testing, ended with a promise: next, monitoring live AI apps for attacks. Here it is.
A test is like a photo. It shows how your app looked on the day you took it. But once real people use the app, things change every hour. New users arrive. Some of them poke at the app on purpose. A model update changes how it answers. You need something that keeps watching after the tests are done. That is monitoring.
One rule first: test only systems you own, or have written permission to test. Everything below is about watching your own app.
By the end you will know what to write down in your logs, what to keep out of them, which signs point to an attack, how an alert and a dashboard help, and what to do when an alert fires. You will also run a small program that raises an alert from a made-up log.
Key words
- Log
- A running diary your app writes. Each line says what happened, when, and to whom.
- Monitoring
- Reading that diary all the time, by machine, so you notice trouble early.
- Signal
- A small clue in the logs that something may be wrong, like many refusals in a row.
- Alert
- A message sent to a person when a signal crosses a line you set, like a smoke alarm.
- Dashboard
- One screen of charts that shows the health of your app at a glance.
- Filter hit
- A moment when a safety filter caught something and blocked or changed it.
- Incident
- A real problem that needs a person to act, such as a suspected attack.
Why watching never stops
Think of a shop. Locking the door at night is a test you pass once. A camera that someone watches catches the thief who found a window you forgot.
AI apps need the camera for three reasons. Real users only show up once you are live. Models can answer differently after an update. And attackers try again and again, because trying is cheap.
The NIST AI Risk Management Framework has a function called Manage. One of its goals reads: "Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management." In plain words: once the app is live, have a plan to watch it, and a plan to act when something goes wrong.
The Production monitoring and LLM observability lesson in AI engineering goes deeper on running AI apps day to day. Here we look at the logs through a security lens.
What to write down
You can only spot what you recorded. So the first job is deciding what goes in the diary.
The OWASP Logging Cheat Sheet says each log entry must record "when, where, who and what". For an AI app, that turns into a short list.
Prompts and replies
What users asked and what the model answered, or at least a safe summary of it, such as its length.
Tool calls
Every time the model uses a tool, like searching, sending an email or reading a file. Note which tool, with what input, and who asked.
Safety events
Refusals, filter hits, and anything blocked. These are the early smoke.
Add the cost side too: how many tokens a request used, and how long it took. A token is one small piece of text a model counts. Cost matters for security because attackers can burn your money. More on that soon.
The same cheat sheet also says to log failed sign-ins, failed access checks and input that did not pass validation. It adds a careful note: unexpected input can come from misuse or plain bugs, so treat it as suspicious, not as proof of an attack.
Here is one made-up log entry. It is structured data: a record with named fields, so a machine can sort and count it.
{
"time": "2026-11-16T09:00:20Z",
"user": "ola",
"event": "filter_hit",
"filter": "secret-fruit-words",
"input_chars": 212,
"prompt_hash": "a41c9e07",
"tokens_used": 0
}Notice prompt_hash. A hash is a short fingerprint made from text. Identical prompts give the same fingerprint, so you can spot repeats without keeping the words.
What to keep out of the logs
A diary full of secrets is its own danger. If someone steals your logs, they steal everything in them. So logging has a second job: leave things out.
The OWASP cheat sheet is firm. It says: "Never log data unless it is legally sanctioned." It lists things that should usually not be recorded directly, but instead removed, masked, sanitized, hashed, or encrypted. The list includes access tokens, authentication passwords, encryption keys and other primary secrets, bank account or payment card holder data, and sensitive personal data. It also lists information a user has opted out of collection or not consented to.
Users type private details into chat boxes, so treat every raw prompt as possibly personal.
Tip
Ask three questions about each field before you log it: do I need it, who can read it, and how long do I keep it? If you cannot answer, store a hash or a length instead of the words.
Logs can also be attacked. The cheat sheet warns about log injection: an attacker puts a fake line break into their input so the log shows a made-up entry that looks like it came from someone else. One defence it names is to avoid unstructured text log formats. That is why the example above uses named fields.
Signs of an attack
Attackers are noisy in ways normal users are not. Here are four signals worth watching. Each one is a clue, not proof. A clue becomes useful when it repeats.
A spike in refusals or filter hits
One refusal is normal. Many from one user in a short time looks like someone testing the walls to see which ones give.
Many similar prompts
The same request reworded again and again. A fingerprint like prompt_hash, or a simple similarity score, helps you notice the pattern.
Very long or very many inputs
OWASP LLM10 (Unbounded Consumption) names a variable-length input flood, and also continuous input overflow: sending inputs that exceed the model's context window, which means the amount of text it can read at once.
Strange tool calls
A tool used at an odd time, in an odd order, or on something the user never asked about. This matters most for apps where the model can act, as the Agentic AI security lesson explains.
Cost is a signal too. The same OWASP page describes "Denial of Wallet": attackers start a high volume of operations to exploit the cost-per-use model of cloud AI services, which can make the bill unsustainable for the provider. A sudden jump in spending on one account is worth an alert.
The Protecting AI APIs lesson covers rate limiting, which caps how many requests one user can make. Monitoring is its partner: the limit slows the attacker down, and the log tells you who it was.
Quick check
One user's chat shows 40 refusals in one minute. What is the best first reading?
Counting in a sliding window
How do you turn a clue into an alert? You count, and you count over a short stretch of time. A total for the whole day hides bursts. Ten filter hits spread over eight hours is a user having a rough day. Ten in one minute is something else.
The trick is called a sliding window. Picture a ruler that covers the last 60 seconds and slides forward with the clock. Each time a new event arrives, you drop events that fell off the back of the ruler and count what is left. If the count is at or above your limit, you raise the alert.
The SIEM page on Wikipedia gives a similar rule as an example of an alert: "Alert on 3 or more failed logins in 1 minute from a single host." Same idea: a small number, a short window, one source.
- 1
Read each line
Split it into the time, the user and the event.
- 2
Keep only filter hits
Other events, like plain prompts, do not count here.
- 3
Slide the window
Add the new hit to that user's list, then drop hits older than 60 seconds.
- 4
Compare to the limit
If the user has 3 or more hits left, print an alert, once per user.
A small alert you can run
Here is the whole thing in plain Python, with 15 made-up log lines. The users (mira, ola, kai) are invented, and so are the events. The program uses only the standard library, so it runs free and offline. There is no real model here. Real logs are messier and arrive all day.
from collections import deque
LOG = """\
09:00:05 mira prompt
09:00:12 ola prompt
09:00:20 ola filter_hit
09:00:31 ola filter_hit
09:00:36 mira filter_hit
09:00:44 ola filter_hit
09:00:51 ola refusal
09:01:02 kai prompt
09:01:10 ola filter_hit
09:01:25 kai filter_hit
09:01:50 mira filter_hit
09:02:08 kai filter_hit
09:02:30 mira prompt
09:03:15 kai filter_hit
09:03:40 kai filter_hit"""
WINDOW = 60 # seconds
THRESHOLD = 3 # filter hits inside one window
def to_seconds(clock):
h, m, s = clock.split(":")
return int(h) * 3600 + int(m) * 60 + int(s)
recent = {} # user -> times of recent filter hits
totals = {} # user -> all filter hits, for comparison
alerted = set()
for line in LOG.splitlines():
clock, user, event = line.split()
if event != "filter_hit":
continue
now = to_seconds(clock)
hits = recent.setdefault(user, deque())
hits.append(now)
totals[user] = totals.get(user, 0) + 1
while hits and now - hits[0] >= WINDOW:
hits.popleft()
if len(hits) >= THRESHOLD and user not in alerted:
alerted.add(user)
print(f"ALERT {clock} {user}: {len(hits)} filter hits in {WINDOW}s")
print("users alerted:", sorted(alerted))
print("all-day totals:", dict(sorted(totals.items())))Output
ALERT 09:00:44 ola: 3 filter hits in 60s
users alerted: ['ola']
all-day totals: {'kai': 4, 'mira': 2, 'ola': 4}Read the output. The program raised one alert: ola made 3 filter hits between 09:00:20 and 09:00:44, which is 24 seconds, so the third hit tripped the limit. ola made a fourth hit at 09:01:10, but we only alert once per user.
Now look at kai. The last line says kai made 4 hits all day, the same as ola, and yet no alert fired. Kai's hits came at 09:01:25, 09:02:08, 09:03:15 and 09:03:40. Never more than 2 fit inside one 60-second window. That is the point of a window: the same total can be calm or alarming depending on how close together the hits are.
mira made only 2 hits, so nothing there either. The window and the limit are two numbers at the top. Set them too low and you get false alarms. Set them too high and you miss real attacks.
Tip
Try changing THRESHOLD to 2 and run it again. You will alert more people, including some who may be harmless. Choosing these numbers is the real work of monitoring.
Alerts and dashboards
An alert is a tap on the shoulder. A dashboard is the window you look through. You need both, for different jobs.
Alerts
Few and clear. They go to a person who can act, such as the on-call developer, and say what happened and where to look.
Dashboards
Charts you scan each morning: refusals per hour, filter hits, tool calls, tokens and cost, slow requests.
Owners
Every alert has a named person or team. An alert nobody owns is noise.
Too many alerts is a real problem. The OWASP cheat sheet warns that a blind checklist approach can lead to unnecessary "alarm fog" that means real problems go undetected. So alert on a few things that matter, and put the rest on a dashboard to read when you have time.
The same cheat sheet says to enable alerting that signals the responsible teams about more serious events immediately.
When an alert fires: the incident loop
An alert is only the start. What you do next is called incident response. It is a loop, because every incident should leave your monitoring better than before.
- 1
Look
Open the logs for that user and that time. Read what really happened before you decide.
- 2
Contain
If it is real, slow the harm: block the user, lower a limit, or switch off a risky tool for now.
- 3
Fix
Find the weak spot and close it, for example a missing filter rule or a tool that trusts too much.
- 4
Tell
Let the right people know what happened. The NIST text says incidents and errors are communicated to relevant AI actors, and that processes for tracking, responding to, and recovering from them are in place.
- 5
Learn
Add a test for what you found, and adjust your signals and limits. Then the same trick is caught earlier next time.
If the weakness was in your app, the Responsible disclosure lesson covers how to share findings, and the Writing the report lesson shows how to record them clearly.
The NIST Manage function also talks about having mechanisms "to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use." In plain words: know in advance how to switch the risky part off. Decide this before the night you need it.
Monitoring tools as a category
You rarely build all of this by hand. Tools fall into a few families. This is a map of kinds, not a ranking, and the right choice depends on your app, team and budget.
Log platforms
They collect logs from many places and let you search, chart and set alerts. They are general-purpose, so they also watch the rest of your system.
SIEM systems
Security tools that gather logs and raise alerts on rules. Wikipedia describes SIEM as combining security information management and security event management "to enable real-time analysis of security alerts generated by applications and network hardware". It adds that SIEM systems are central to security operations centers, where they are used "to detect, investigate, and respond to security incidents".
LLM observability tools
Built for AI apps. They record prompts, replies, tool calls, tokens and cost for each request, and let you replay a conversation.
The Wikipedia page lists three essential parts of a SIEM, which is a handy way to think about any monitoring setup: a data collector that forwards selected audit logs, an ingest and indexing point that parses and correlates, and a search node used for visualization, queries, reports and alerts. Collect, understand, then look.
OWASP LLM10 also lists a control called "Comprehensive Logging, Monitoring and Anomaly Detection": continuously monitor resource usage and implement logging to detect and respond to unusual patterns of resource consumption.
Quick check
Which comes first when you choose what to log?
Recap
Record, but carefully
Log prompts, tool calls, refusals, filter hits and cost. Keep secrets and personal data out, or hash them.
Count in short windows
A burst in 60 seconds says more than a total for the day. Tune the window and the limit.
Close the loop
Look, contain, fix, tell, learn. Each incident should leave a new test behind.
Next: security tests that run on every change.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.