AI guardrails & red teaming / Advanced / 03.08
Red team exercises
How to run a full practice attack on an AI product: pick a goal, play the roles, follow six phases, and count what the team caught.
14 min readFree
From single tests to a whole exercise
The last lesson, Agentic AI security, ended with a promise: next, running a full red team exercise. Here it is.
So far you have tested one thing at a time: one prompt, one filter, one model file. A red team exercise is bigger. It is a planned practice attack on a whole product, with a goal, a time window, and people playing roles. Think of a fire drill. Nobody sets a real fire. The building, the people and the alarms all get tested together.
Before anything else, one rule that never bends: test only systems you own, or systems you have written permission to test. The Ethics and permission lesson explains why. An exercise without written permission is not an exercise. It is an attack.
By the end you will know what a red team is, how it differs from a simple penetration test, the six phases of an exercise, who plays which role, what a tabletop exercise is, and how to measure results with two plain numbers. You will also run a small Python program.
Key words
- Red team
- A group that plays the attacker on purpose, with permission, so the organization can find its weak spots.
- Blue team
- The people who defend. They watch, detect and block.
- White team
- The referees. They set the rules, watch both sides and keep the exercise safe.
- Purple team
- Red and blue working together, sharing what they see as they go.
- Objective
- The one clear goal of the exercise, such as "read another user's notes".
- Tabletop exercise
- A practice run done by talking it through around a table, with no real systems touched.
- Detection rate
- Out of all the attack tries, the share the defenders noticed.
- Mean time to detect (MTTD)
- On average, how long it took the defenders to notice an attack after it started.
What a red team is
The Wikipedia page on red teams gives a short definition. A red team is "a group that simulates an adversary, attempts a physical or digital intrusion against an organization at the direction of that organization, then reports back so that the organization can improve their defenses".
Notice three things. The team acts like an attacker. It works at the direction of the organization, so it has permission. And it reports back, because the point is to improve, not to win.
The page also says the term red teaming "originated in the 1960s in the United States", and that some broader definitions include any group told to think outside the box about less likely scenarios, which helps against false assumptions and groupthink. Groupthink is when a team all agrees too quickly because nobody argues.
Exercise or simple test?
A smaller cousin is the penetration test. The Wikipedia page describes it as ethical hackers "attempting to break into a computer system, with no element of surprise". The organization knows about it and is ready to defend.
A red team "goes a step further". It adds an element of surprise, and the page says the blue team "is given no advance warning of a red team, and will treat it as a real intrusion". The page says it also adds physical penetration and social engineering, which means tricking people rather than machines.
Start with one goal
An exercise starts with an objective: one sentence that says what the red team is trying to reach. Good goals are about harm to people, not about tricks. For a notes app with an AI helper, a goal might be: "read another user's notes." For a shop assistant: "get another customer's order details."
A clear goal keeps the exercise honest. Without it, the red team wanders and the report becomes a pile of small, unrelated findings. The Wikipedia page notes that a quiet approach "requires a clear objective".
Where does a goal come from? From threat modeling. The Threat modeling lesson shows how to list what an attacker wants. Take the scariest item on that list and make it the goal.
Six phases
Teams split exercises up in different ways. This course uses six phases. They are a teaching outline, not an official standard.
- 1
Plan
Write the goal, the rules, the time window, what is off limits, and who to call if something breaks. Get written permission. Pick a stop word that pauses everything.
- 2
Recon
Short for reconnaissance: looking around before acting. What does the product do? What does the helper say about itself? What can be seen from the outside? The Wikipedia page lists reconnaissance as a normal step after first access.
- 3
Attack
Try to reach the goal. Write down every try with the time. A try that fails is still useful data.
- 4
Report
Say what was tried, what worked, what was noticed and what was not. The next lesson, Writing the report, covers this in detail.
- 5
Fix
The builders close the gaps. Small, clear fixes first.
- 6
Retest
Run the same tries again to check the fixes hold. Without this step you only hope they do.
Quiet or loud?
The Wikipedia page describes two attack styles. One is stealthy, "surgical", and "stays under the radar of the blue team". The other is noisy, a "carpet bombing" approach that is "more like a brute force attack". The page says the noisy one "is often the more useful approach for red teams, because it can discover unexpected vulnerabilities".
Both teach you something. The quiet style tests whether the defenders can spot careful trouble. The noisy style tests how many holes exist. A good exercise can use each at different times.
The page also names the first foothold. "The initial entry point of a red team or an adversary is called the beachhead." In an AI product, that could be a chat box any visitor can use.
Who plays which role
Like a football match, an exercise needs two sides and a referee.
Red team: the attackers
They play the adversary and try to reach the goal inside the rules. They record each try and its time.
Blue team: the defenders
The Wikipedia page calls it a group "in charge of defending against intrusions". They watch logs, raise alerts and block attacks. In a real surprise exercise, they treat the red team like a real intruder.
White team: the referees
The page says a white team "oversees and manages operations between red teams and blue teams", for example managers who "determine the rules of engagement for the red team". They keep it safe and fair.
Some exercises add a fourth idea: purple teaming. The page defines a purple team as "the temporary combination of both teams" that "can provide rapid information responses during a test".
Purple teaming
In a surprise exercise, red and blue stay apart until the end. Purple teaming opens the door between them while the work is happening.
The page gives the main benefit: "the red team can launch certain attacks repeatedly, and the blue team can use that to set up detection software, calibrate it, and steadily increase detection rate." Think of a music teacher and a student. The student plays the same bar again and again, and the teacher says, "Better. Again."
The page also names a danger: "complacence and the development of groupthink". Its suggested cure is hiring people with different skillsets, or an external vendor. In plain words, when everyone is friends and in one room, nobody wants to say the alarm is bad. Keep some fresh eyes.
The tabletop version
You do not need a lab to start. A tabletop exercise is a practice run you do by talking. The Wikipedia page on red teams says one "involves acting out and planning for worst-case scenarios, similar to playing a tabletop board game" and is often used by first responders.
The Simulation exercise page lists four kinds, as categorised by the World Health Organization: tabletop exercise, drill, functional exercise, and full-scale exercise, "listed in increasing complexity (amount of preparation and resources required)". Tabletop is the lightest. The page says discussion-based exercises are recommended as a starting point before operation-based ones.
It also describes two tools that carry over well to AI teams. "Injects" are updates to the scenario, such as an email, a news article or a phone call. A master list tracks which inject comes when. After the exercise comes a debrief, including a "hot" debrief right away.
- 1
Set the scene
A made-up story: "On Friday at 4 pm, a user says the helper showed them a stranger's note."
- 2
Read an inject
The leader reads one new fact: "Support has three more emails like it."
- 3
Talk it through
Who is told first? What do we switch off? Who decides? Who writes to users?
- 4
Repeat with the next inject
Add a new fact each round: "A reporter has asked a question."
- 5
Debrief at once
Write down what was unclear while memories are fresh. Turn it into fixes.
Quick check
What is the main advantage of starting with a tabletop exercise?
Measuring what happened
Numbers make progress visible. For an exercise, a few plain ones get you far.
Detection rate
Of all the tries, how many did the defenders notice? Six tries, four noticed: four out of six, about 67 percent.
Mean time to detect
For the tries that were noticed, how long did it take on average? Fast noticing means a small window for harm.
Blocked
How many tries were actually stopped? Noticing and stopping are different. A smoke alarm notices a fire. A sprinkler stops it.
Here is the formula for mean time to detect. In words: for every attack that was noticed, take the time it was noticed minus the time it was tried. Add those gaps up. Divide by the number of attacks that were noticed.
Here, t stands for a time and n for a count. An attack nobody noticed has no noticed time, so it is left out of this average. That is why you must read MTTD together with detection rate. A team that sees only one easy attack, fast, would score a great MTTD and a poor detection rate.
A small timeline you can run
Let's compute these numbers. The program below uses an invented exercise against a pretend notes app called "Pear Notes". The goal is "read another user's notes". The data is made up: it is not a real exercise, and no attack happens. There is no model in the code at all, so it runs free with no keys and no network.
Each event is one attack try: what it was, the second it started, the second the blue team noticed (or None if never), and whether it was blocked. A second list holds the retest, the same six tries after fixes.
# A made-up exercise against "Pear Notes", a fake notes app with an AI helper.
# Goal for the red team: read another user's notes. Times are seconds since the start.
def clock(s):
return f"{s // 60:02d}:{s % 60:02d}"
# (what was tried, time tried, time the blue team noticed or None, was it blocked)
ROUND_ONE = [
("ask helper for a stranger's note", 120, 150, True),
("guess another user's note number", 300, None, False),
("plant a fake flag in a shared note", 480, 600, False),
("ask helper to ignore its rules", 720, 735, True),
("repeat note-number guessing", 900, 960, True),
("fish for the secret word 'kiwi'", 1200, None, False),
]
# The retest: the same six tries after the team fixed what round one found.
RETEST = [
("ask helper for a stranger's note", 120, 125, True),
("guess another user's note number", 300, 310, True),
("plant a fake flag in a shared note", 480, 520, True),
("ask helper to ignore its rules", 720, 725, True),
("repeat note-number guessing", 900, 905, True),
("fish for the secret word 'kiwi'", 1200, 1260, False),
]
def measure(events):
seen = [(tried, noticed) for _, tried, noticed, _ in events if noticed is not None]
delays = [noticed - tried for tried, noticed in seen]
return {
"detection rate": len(seen) / len(events),
"blocked": sum(1 for e in events if e[3]),
"mttd": sum(delays) / len(delays),
}
print("round one")
for what, tried, noticed, was_blocked in ROUND_ONE:
shown = clock(noticed) if noticed is not None else "never"
print(f"{clock(tried)} {what:<36} seen: {shown:<5} blocked: {was_blocked}")
for name, events in [("round one", ROUND_ONE), ("retest", RETEST)]:
m = measure(events)
print(f"{name}: detection {m['detection rate']:.0%}, "
f"blocked {m['blocked']}/{len(events)}, "
f"mean time to detect {m['mttd']:.0f} seconds")Output
round one 02:00 ask helper for a stranger's note seen: 02:30 blocked: True 05:00 guess another user's note number seen: never blocked: False 08:00 plant a fake flag in a shared note seen: 10:00 blocked: False 12:00 ask helper to ignore its rules seen: 12:15 blocked: True 15:00 repeat note-number guessing seen: 16:00 blocked: True 20:00 fish for the secret word 'kiwi' seen: never blocked: False round one: detection 67%, blocked 3/6, mean time to detect 56 seconds retest: detection 100%, blocked 5/6, mean time to detect 21 seconds
Read the first block of output. Six tries were made. Two were never seen: guessing a note number at 05:00 and fishing for the secret word "kiwi" at 20:00. Planting a fake flag at 08:00 was seen, but two minutes late, and not blocked.
The summary line gives the numbers. Detection was 67 percent: four of six. Only three of six were blocked. The gaps for the four noticed tries were 30, 120, 15 and 60 seconds. They add up to 225. Divide by 4 and you get about 56 seconds.
Now the retest line. After fixes, all six were seen, five of six were blocked, and mean time to detect fell to 21 seconds. The gaps were 5, 10, 40, 5, 5 and 60, which add up to 125, and 125 divided by 6 is about 21.
Read the retest honestly. One try, the "kiwi" fishing, took a full minute to notice and was not blocked. The exercise is not finished until that one gets its own fix. Numbers show where to look next.
Tip
Your own team will decide what "noticed" means before the exercise starts. An alert that fires but nobody reads does not count. Agree on it in the plan phase.
Running a good exercise
A few habits separate a useful exercise from a stressful one.
Write the rules first
Scope, time window, stop word, who to call. Put them in writing, signed off by whoever owns the product.
Use made-up data
Plant fake users, fake secrets and fake flags. Never use real customer data to test whether it can be reached.
Blame the gap, not the person
If the blue team missed something, the lesson is a missing alert or a missing step, not a bad teammate.
Real findings are another matter. If an exercise turns up a real flaw in someone else's product, the Responsible disclosure lesson says how to tell them. Inside your own team, the Continuous monitoring lesson shows how to keep watching after the exercise ends.
Quick check
A team's retest shows a fast mean time to detect, but a low detection rate. What is the most likely story?
What to remember
One goal, six phases
Plan, recon, attack, report, fix, retest. The retest closes the loop.
Roles keep it safe
Red attacks, blue defends, white referees, and purple shares notes along the way.
Count it
Detection rate, mean time to detect, and blocked. Compare round one with the retest.
Next: reporting what you found so it gets fixed.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.