AI guardrails & red teaming / Beginner / 01.06
Threat modeling
Draw the map of your AI app first: what it holds, where data moves, and what can go wrong at every border, using four questions and STRIDE.
12 min readFree
Draw the map before you attack
The last lesson, Generative models and LLMs, ended with a promise: next, threat modeling, drawing the map before you attack. Here it is. Think of a house. Before you buy locks, you walk around it. Where are the doors? Which windows open? Where do you keep the spare key? Locks bought without a walk-round end up on the wrong doors.
An AI app is the same. You can buy filters and checks all day, but you first need to know what your app holds, how information moves through it, and where a stranger could touch it. Writing that down is threat modeling. It needs no security background. It needs paper, a little patience, and the habit of asking "what if?"
One rule before we start. Test only systems you own or have written permission to test. Everything in this lesson is made up: a pretend shop, pretend files, pretend names. Use the ideas on your own projects, or on a project where the owner has said yes in writing.
Key words
- Threat
- Something bad that could happen to your app, like data leaking or the app being fooled.
- Threat model
- A written picture of your app plus a list of the threats you found, and what you will do about each.
- Asset
- Anything worth protecting: customer data, secret keys, your good name.
- Data flow
- A path along which information travels, such as a question going from a person to your app.
- Trust boundary
- A border where the level of trust changes, like the front door of a house.
- STRIDE
- Six kinds of threat, named by their first letters. We meet each one below.
- RAG
- Retrieval-augmented generation: the app looks up documents first, then hands them to the model to help it answer.
Four questions that do most of the work
The Threat Modeling Manifesto is a short guide written by a group of practitioners. It says threat modeling is "analyzing representations of a system to highlight concerns about security and privacy characteristics". In plain words: look at a picture of your system and ask what could go wrong. The manifesto says that at the highest level you ask four questions.
- 1
What are we working on?
Draw the system. Boxes for the parts, arrows for the information moving between them. A napkin sketch is fine.
- 2
What can go wrong?
Walk along the picture and ask it about every box and every arrow. This is where STRIDE helps.
- 3
What are we going to do about it?
For each threat, pick an answer: block it, make it harder, watch for it, or knowingly accept it.
- 4
Did we do a good enough job?
Check your work. Did the picture match the real app? Did the fixes really work? If not, go round again.
The manifesto also says why: when you threat model, you begin to recognise what can go wrong in a system, and you can pinpoint design and implementation issues that need fixing, whether early or throughout its lifetime. Two of its values are worth taping to your monitor: "Doing threat modeling over talking about it" and "Continuous refinement over a single delivery". Do it, and do it again as the app changes.
It also names who should do it. "You. Everyone." That includes designers. A designer who knows where a user pastes text into a form already knows where an attacker could too.
Quick check
Which question comes third in the manifesto's list?
Question one: draw the map
Wikipedia notes that most threat modeling approaches use data flow diagrams, or DFDs. They were developed in the 1970s so engineers could explain, at a high level, how an application moves, stores and changes data. Traditionally a DFD uses four symbols.
Processes and people
Things that do work (your chat app) and things outside that talk to it (a shopper). The article calls the outside ones interactors.
Data stores
Places where information rests: a database, a log file, an index of documents.
Data flows
Arrows. Each arrow is one kind of information going from one box to another.
Then comes the symbol that matters most for security. In the early 2000s, Wikipedia says, trust boundaries were added to improve how useful DFDs are for threat modeling. A trust boundary is a line you draw where the level of trust changes. Your own server is on one side. A stranger's phone is on the other. An arrow that crosses the line deserves extra attention, because that is where data can be changed, faked or stolen.
Here is the chatbot we will model in a moment, as a tiny text drawing. It answers questions about a pretend fruit shop, Mango Mart.
OUTSIDE OUR APP OUR DATA MODEL HOST
Shopper ---> Chat app --> Retriever --> Doc index
Uploaded file ---------------------------> Doc index
Chat app <-- (answer) <-------------- Model
Chat app --> Log fileTip
Do not aim for a perfect drawing. Five boxes and a few arrows beat a poster nobody finishes. You can add detail on the second round.
Question two: what can go wrong, with STRIDE
Asking "what can go wrong?" of a blank page is hard. A checklist helps. STRIDE is one. Wikipedia says it was developed by Praerit Garg and Loren Kohnfelder at Microsoft, and that it gives a mnemonic (a memory trick) for security threats in six categories. Each category matches a core principle of information security. Wikipedia lists those principles as authenticity, integrity, non-repudiability, confidentiality, availability and authorization.
S: Spoofing
Pretending to be someone else. Like a stranger wearing a delivery uniform to get through your door. It attacks authenticity.
T: Tampering
Changing something that should not change. Like swapping the price tags in a shop. It attacks integrity.
R: Repudiation
Doing something and later denying it, with no record to prove otherwise. It attacks non-repudiability.
I: Information disclosure
Letting private information reach people who should not see it. It attacks confidentiality.
D: Denial of service
Making the app too busy or broken for real users. Like a crowd blocking a shop entrance. It attacks availability.
E: Elevation of privilege
Getting powers you were never given. Like a visitor ending up with the staff keys. It attacks authorization.
Notice the match with the earlier idea. Each arrow and each box can be asked six questions: can this be faked, changed, denied, leaked, overloaded or overpowered? Six questions across a handful of arrows gives you a real list in an afternoon.
Quick check
Someone edits the documents your chatbot reads so it says the wrong opening hours. Which STRIDE letter fits best?
What is new when the app has a model
STRIDE was made for ordinary software. It still works for AI apps, but a model adds a few twists, and your map should show them. Three matter most for beginners.
Text that acts like an order
The model reads questions and documents as one stream of words. A document can hide a sentence that tries to steer it. This is prompt injection. On the map, any arrow carrying outside text into the model is a place to look. The Prompt injection attacks lesson in AI engineering goes deeper.
Tools used wrongly
If the model can send emails or change records, a fooled model can do real things. On the map, mark every tool as its own box and ask: what is the worst this tool could do if the model asked it nicely?
Private data slipping out
The model may see more than the person asking should. Your document index might hold staff-only pages next to public ones. Any arrow from a store into the model can leak.
Seen through STRIDE, the first is mostly tampering and elevation of privilege (outside text gains control it should not have). The second is elevation of privilege. The third is information disclosure. You do not need new letters, just new places to point the old ones.
Tip
Treat the model like a very keen new assistant who believes whatever paper lands on their desk. Then ask who can put paper on the desk.
A catalogue of known AI attacks: MITRE ATLAS
You do not have to invent every threat yourself. MITRE runs ATLAS, short for Adversarial Threat Landscape for AI Systems. Its data repository describes it as "a public knowledge base of adversary TTPs targeting AI systems". The repository holds ATLAS tactics, techniques and case studies as data files.
Think of it as a field guide, like a bird book. When you ask "what can go wrong?", ATLAS gives you names for attack ideas other people have already seen, so your list is not limited to what you can think up on the day. Use it to enrich question two, not to replace the drawing.
A reminder: this is for defenders. Read it to find gaps in your own app, and test only what you own or have written permission to test.
A threat table in plain Python
Now put the map into code. The program below lists the parts of the Mango Mart chatbot, tags each part with a zone (the trust areas) and a kind, then lists the data flows. A few simple rules add STRIDE letters to each flow.
The rules are deliberately simple, so you can read them. A real review uses judgment, not a six-line rule set, and the output is a starting list, not a verdict. Note too that the "Model stand-in" is only a name in a table. No model runs here, so the code is free and needs no key. Real models behave less predictably.
# A made-up RAG chatbot for "Mango Mart". Zones are trust areas.
ZONE = {
"Shopper": "outside", "Uploaded file": "outside",
"Chat app": "our app", "Retriever": "our app", "Log file": "our app",
"Doc index": "our data", "Model stand-in": "model host",
}
KIND = {"Shopper": "person", "Uploaded file": "person", "Chat app": "process",
"Retriever": "process", "Model stand-in": "process",
"Doc index": "store", "Log file": "store"}
FLOWS = [
("Shopper", "Chat app", "question"),
("Uploaded file", "Doc index", "new document"),
("Chat app", "Retriever", "search request"),
("Retriever", "Doc index", "lookup"),
("Doc index", "Retriever", "matching passages"),
("Retriever", "Model stand-in", "question + passages"),
("Model stand-in", "Chat app", "answer"),
("Chat app", "Log file", "chat record"),
("Chat app", "Shopper", "answer"),
]
def stride(src, dst):
t = set()
if KIND[src] == "person": t.add("S") # a person can pretend to be someone
if KIND[dst] == "store": t.update("TR") # data written can be changed or denied
if KIND[src] == "store": t.add("I") # data read can leak
if KIND[dst] == "process": t.update("DE") # a busy or over-trusting process
if ZONE[src] != ZONE[dst]: t.add("T") # crossing: data can change on the way
return "".join(c for c in "STRIDE" if c in t)
by_cross = {}
for src, dst, what in FLOWS:
if ZONE[src] != ZONE[dst]:
by_cross.setdefault((ZONE[src], ZONE[dst]), []).append((src, dst, what, stride(src, dst)))
for (a, b), rows in by_cross.items():
print(f"{a} -> {b}")
for src, dst, what, tags in rows:
print(f" {src} -> {dst} ({what}): {tags}")
inside = [f for f in FLOWS if ZONE[f[0]] == ZONE[f[1]]]
print("flows that stay inside one zone:", len(inside))
print("boundary crossings:", sum(len(r) for r in by_cross.values()), "of", len(FLOWS))Output
outside -> our app Shopper -> Chat app (question): STDE outside -> our data Uploaded file -> Doc index (new document): STR our app -> our data Retriever -> Doc index (lookup): TR our data -> our app Doc index -> Retriever (matching passages): TIDE our app -> model host Retriever -> Model stand-in (question + passages): TDE model host -> our app Model stand-in -> Chat app (answer): TDE our app -> outside Chat app -> Shopper (answer): T flows that stay inside one zone: 2 boundary crossings: 7 of 9
Read the output zone by zone. Seven of the nine flows cross a trust boundary, and only two stay inside one zone: the chat app calling the retriever, and the chat app writing to the log file. That is typical. Most of an AI app's risk sits at the borders.
Look at the line Doc index to Retriever, which carries TIDE: four tags. The documents come out of a store (information disclosure), go into a process (denial of service, elevation of privilege) and cross a border (tampering). It ties with Shopper to Chat app as the busiest arrow, and it is where uploaded text finally reaches the model's side of the app. The line Uploaded file to Doc index carries S, T and R: the sender could be faked, the file could be altered, and nobody may record who sent it. A good first fix is to check who uploads, scan what they upload, and log it.
Quick check
In the output, how many flows stay inside one zone?
Questions three and four
A list of threats is only half the job. Question three asks what you will do about each one. For every row, write one of four answers: stop it, make it harder, watch for it, or accept it on purpose and say why. Tomorrow's you will not remember the reason, so write it down.
Question four asks whether you did a good enough job. Compare the drawing with the real app: is there an arrow you forgot, such as a plugin or a back-up copy? Try your fixes on a test copy you own. Then plan to redo the map when the app changes. The manifesto's idea of continuous refinement over a single delivery is the whole point: a threat model is a living page.
Keep the map small and useful. Wikipedia's threat model article says threat modeling helps answer questions like where you are most vulnerable, what the most relevant threats are, and what you need to do to safeguard against them. If your page answers those three, it is doing its job.
- 1
Sketch
Draw boxes, arrows and the trust boundaries.
- 2
Tag
Walk each arrow and each box through the six STRIDE letters.
- 3
Add the model's twists
Outside text, tools and private data.
- 4
Decide
Stop, harden, watch or accept, with a reason.
- 5
Check and repeat
Compare to the real app and redo it after each change.
Next: risk management, deciding what to fix first.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.