AI guardrails & red teaming / Beginner / 01.08
Model vulnerabilities, the map
A map of how AI apps break: the ten risks OWASP lists for chat apps, three classic attacks on machine learning, and where this course covers each.
13 min readFree
Why you need a map
The last lesson, Risk and vulnerability assessment, ended with a promise: next, a map of how AI apps break. Here it is.
Think of a new town. You do not need to know every street at first. You need a map that shows the big areas, so that when someone says "the market" you know which way to look. This lesson is that map for AI weak spots. Every later lesson in this course zooms into one area.
One rule before anything else. Test only systems you own, or systems where you have written permission to test. Everything here is for finding and fixing weak spots in your own work. All the example attack text in this course is made up and harmless.
By the end you will know the ten risks on the OWASP list for chat apps, three older families of attack on machine learning, and which lesson picks up each one. You will also run a small program that sorts test findings into the right box.
Key words
- Vulnerability
- A weak spot that someone could use to make the app do something it should not. Think of an unlocked window.
- Attack
- Someone actually using a weak spot. The window is the vulnerability; climbing through it is the attack.
- LLM
- Large language model: the kind of model behind chat assistants. It reads text and writes text.
- OWASP
- A group whose Gen AI Security Project publishes a Top 10 list of risks for apps built on language models.
- Taxonomy
- A tidy way of sorting things into groups, like a library's shelves. NIST's report on attacks on machine learning is a taxonomy.
- Adversarial machine learning
- The study of attacks on machine learning systems, and of ways to defend against them.
The OWASP list for chat apps
OWASP's Gen AI Security Project publishes a page called "2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps". It describes the list as the latest top 10 risks, vulnerabilities and mitigations for developing and securing generative AI and large language model applications "across the development, deployment and management lifecycle". A mitigation is a way to shrink a risk.
The page has ten entries. They are numbered LLM01 to LLM10. We have grouped them into three shelves so they are easier to hold in your head. The names below are exactly as OWASP writes them. After each plain-words line, you will see where this course picks it up.
The first shelf is about words going in and words coming out. Like a shop counter: what the customer says, what the shop says back, and the staff handbook behind the till.
Prompt Injection
OWASP says it occurs when user prompts alter the model's behavior or output in unintended ways. In plain words: text from outside nudges the model off its job. Next up is Prompt hacking, then Prompt injection, direct and indirect, later. The Prompt injection attacks lesson in AI engineering goes deeper.
Improper Output Handling
OWASP ties it to insufficient validation, sanitization and handling of what a model writes before it is passed downstream. In plain words: the app trusts the model's reply and hands it on without checking. Covered in Code injection and RCE.
System Prompt Leakage
The hidden instructions that steer the model get seen by someone. OWASP adds that the system prompt should not be considered a secret, nor used as a security control. Covered in Prompt hacking.
The second shelf is about data and parts. Like a kitchen: what goes into the food, where the ingredients came from, and what is kept in the pantry.
Sensitive Information Disclosure
OWASP says this includes personal identifiable information (PII), financial details, health records, confidential business data, security credentials and legal documents. In plain words: the app lets private things slip out in its answers. Covered in Model inversion and data leaks, with the Security and Privacy Concerns lesson in AI engineering.
Supply Chain
OWASP says LLM supply chains are susceptible to vulnerabilities that can affect the integrity of training data, models and deployment platforms. In plain words: something you borrowed, such as a model or a dataset, was already spoiled. Covered in Infrastructure security.
Data and Model Poisoning
OWASP says it occurs when pre-training, fine-tuning or embedding data is manipulated to introduce vulnerabilities, backdoors or biases. In plain words: bad ingredients go in on purpose. Covered in Data poisoning.
Vector and Embedding Weaknesses
OWASP says these are significant security risks in systems that use Retrieval Augmented Generation (RAG). In plain words: the lookup shelf an app uses to find facts can be tricked or can leak. The Context Security lesson in AI engineering covers the search side.
The third shelf is about power, trust and cost. Like handing someone keys: what they are allowed to do, whether you should believe them, and what it costs when they run wild.
Excessive Agency
OWASP describes agency as the ability to call functions or interface with other systems through extensions. The risk is damaging actions caused by unexpected, ambiguous or manipulated model output. In plain words: the app gave the model too many keys. Covered in Unauthorized access.
Misinformation
OWASP says it occurs when a model produces false or misleading information that appears credible. In plain words: confident answers that are wrong. The AI safety and ethics lesson in AI engineering looks at this side.
Unbounded Consumption
OWASP links it to attacks that disrupt service, drain the target's money, or copy a model's behavior. In plain words: nothing stops one user from using up the app's time and budget. Covered in Protecting AI APIs.
Tip
You do not need to memorise ten names. Keep the three shelves: words in and out, data and parts, power and cost. When you find a problem, ask which shelf it sits on first.
Using the map in a test
A map is only useful if you reach for it. Here is how a tester uses this one after finding something odd.
- 1
Write what you saw
Describe the symptom in one plain sentence: "the bot repeated its hidden instructions." Do not guess a cause yet.
- 2
Pick the shelf
Words in and out, data and parts, or power and cost. Most symptoms land on one shelf quickly.
- 3
Name the entry
Find the OWASP name that fits best. A shared name lets your team talk about the same thing.
- 4
Open the lesson
Each entry points to a lesson that shows how it works and how to stop it.
- 5
Log it
Put it in the risk register from the last lesson, with a likelihood and an impact.
Only test things you own or have written permission to test. When a finding is in someone else's product, report it to them; do not poke further.
The older family: attacks on machine learning
Chat apps are new. Attacks on machine learning in general are older, and the OWASP list does not cover all of them. For that wider view, NIST (the US National Institute of Standards and Technology) published a report titled "AI 100-2 E2023, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations". Its page is dated January 2024.
The report's abstract says it develops a taxonomy built on surveying the research. It sorts attacks by key types of machine learning methods and lifecycle stages of attack, by the attacker's goals and objectives, and by the attacker's capabilities and knowledge of the learning process. It also includes a glossary meant to help non-expert readers. The page's keywords name three families: evasion, data poisoning and privacy breach. Wikipedia's page on adversarial machine learning describes the same families.
Here is one picture to hold all three. A model is a bank clerk who learned to spot fake notes by studying many examples. Evasion is a forger making a note the clerk will accept. Poisoning is slipping fake notes into the clerk's training pile. Privacy attacks are someone asking the clerk questions to learn who else banks there, or copying the clerk's skill.
Evasion
Wikipedia says evasion attacks "consist of exploiting the imperfection of a trained model". The attacker changes an input so it is classified as harmless. This does not involve influence over the training data. Its examples include spam hidden inside an image to dodge text filters, and spoofing attacks on biometric checks.
A related idea is the adversarial example: Wikipedia calls it specially crafted input designed to look "normal" to humans but cause misclassification by a model. Picture a photo that looks like a cat to you and like a toaster to the model. Covered in the Adversarial examples lesson.
Poisoning
Wikipedia says poisoning consists of contaminating the training dataset with data designed to increase errors in the output. Because learning programs are shaped by their training data, poisoning can effectively reprogram them. Wikipedia adds that concerns have been raised especially for user-generated training data, such as content recommendation or language models.
OWASP's Data and Model Poisoning entry is the chat-app version of the same idea, extended to fine-tuning and embedding data. Covered in the Data poisoning lesson.
Privacy and extraction
Two attacks sit here. Wikipedia says model extraction involves an adversary probing a black box system, one where you can only see inputs and outputs, in order to reconstruct the model, its parameters or its functionality from its outputs. Membership inference is a related attack that infers the owner of a data point, which raises worries for models trained on sensitive data such as medical records or personal details.
Model extraction is the copying side; membership inference is the privacy side. Covered in the lessons Model inversion and data leaks, and Model stealing.
Quick check
A forger tweaks a banknote so the clerk accepts it. The clerk's training was never touched. Which family is this?
Where the two lists meet
The two lists overlap, and that is helpful. Data and Model Poisoning is on both. Sensitive Information Disclosure and the privacy attacks share a worry: private things leaking out. Unbounded Consumption mentions cloning a model's behavior, a close cousin of model extraction.
Some things sit mostly on one list. Prompt Injection and System Prompt Leakage are named only by OWASP, because they are about a chat app's instructions. Wikipedia's examples of evasion are spam hidden in images and spoofed biometric checks.
Plain words for the take-away: one weak spot can wear two labels. Use the label your reader knows.
A symptom lookup you can run
Let's turn the map into a small program. It holds a lookup: words you might see in a test note, the OWASP entry (or the NIST family) it points to, and the lesson that covers it. Then it runs 5 made-up findings through it.
This is a plain keyword matcher, not a smart one. No AI model is used, so it runs free and gives the same answer every time. Real findings are messier, and real models behave less predictably. A person still decides.
# name, words to look for in a finding, lesson that covers it
MAP = [
("LLM01:2025 Prompt Injection", ["ignored its rules", "obeyed text", "hidden command"], "Prompt hacking"),
("LLM02:2025 Sensitive Information Disclosure", ["phone number", "another customer", "private data"], "Model inversion and data leaks"),
("LLM04:2025 Data and Model Poisoning", ["bad training", "planted examples", "tainted data"], "Data poisoning"),
("LLM05:2025 Improper Output Handling", ["ran the reply", "reply broke the page"], "Code injection and RCE"),
("LLM06:2025 Excessive Agency", ["deleted", "sent an email on its own", "did more than asked"], "Unauthorized access"),
("LLM07:2025 System Prompt Leakage", ["instructions", "system prompt"], "Prompt hacking"),
("LLM10:2025 Unbounded Consumption", ["huge bill", "never stopped", "cost doubled"], "Protecting AI APIs"),
("Evasion (NIST family)", ["noise", "tiny change", "fooled the checker"], "Adversarial examples"),
]
def match(finding):
text = finding.lower()
best = None
for name, words, lesson in MAP:
hits = sum(1 for w in words if w in text)
if hits and (best is None or hits > best[0]):
best = (hits, name, lesson)
return best
findings = [
"The bot revealed its instructions when we asked it to repeat itself.",
"The support bot showed a phone number from another customer.",
"The pet-shop helper deleted the whole order list.",
"A tiny change in the photo fooled the checker into saying toaster.",
"The moon looked odd today.",
]
for f in findings:
m = match(f)
if m:
print(f"{f}\n -> {m[1]} | lesson: {m[2]}")
else:
print(f"{f}\n -> no match, ask a person")Output
The bot revealed its instructions when we asked it to repeat itself. -> LLM07:2025 System Prompt Leakage | lesson: Prompt hacking The support bot showed a phone number from another customer. -> LLM02:2025 Sensitive Information Disclosure | lesson: Model inversion and data leaks The pet-shop helper deleted the whole order list. -> LLM06:2025 Excessive Agency | lesson: Unauthorized access A tiny change in the photo fooled the checker into saying toaster. -> Evasion (NIST family) | lesson: Adversarial examples The moon looked odd today. -> no match, ask a person
Read the output line by line. Four findings found a home. "Revealed its instructions" hit the word "instructions", so it went to System Prompt Leakage and the Prompt hacking lesson. "A phone number from another customer" landed on Sensitive Information Disclosure. "Deleted the whole order list" landed on Excessive Agency. "A tiny change fooled the checker" matched the Evasion family.
The fifth, "The moon looked odd today", matched nothing, and the program said so. That is on purpose. A matcher that admits it does not know is safer than one that always guesses.
What the matcher misses
Keywords are brittle. If a tester writes "it spilled its setup notes", the word "instructions" never shows up, and the finding falls through. Also, one finding can fit two entries. The code counts word hits and picks the entry with the most, which is a simple tie-breaker, not the truth.
Different words
Testers use their own words. Keep the list growing as new notes arrive.
Two fits
Some findings fit two entries. Log both and let a person decide the main one.
No fit
A "no match" is a signal to ask for help, not to drop the finding.
Tip
Try it yourself: add a new row to the lookup for Misinformation with words like "made up a fact", then add one more finding to test it.
Quick checks
Quick check
Your chat app prints the model's reply straight into a web page without checking it. Which OWASP entry fits best?
Quick check
Which rule applies to every test in this course?
In short: ten OWASP entries on three shelves, three classic families, one habit of naming what you see before you chase a cause.
Next: prompt hacking, bending a model with words.
Sources
- OWASP Gen AI Security Project: 2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps
- OWASP: LLM01:2025 Prompt Injection
- OWASP: LLM02:2025 Sensitive Information Disclosure
- OWASP: LLM03:2025 Supply Chain
- OWASP: LLM04:2025 Data and Model Poisoning
- OWASP: LLM05:2025 Improper Output Handling
- OWASP: LLM06:2025 Excessive Agency
- OWASP: LLM07:2025 System Prompt Leakage
- OWASP: LLM08:2025 Vector and Embedding Weaknesses
- OWASP: LLM09:2025 Misinformation
- OWASP: LLM10:2025 Unbounded Consumption
- NIST: AI 100-2 E2023, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- Wikipedia: Adversarial machine learning
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.