AI guardrails & red teaming / Advanced / 03.05
LLM security testing
Run a full security test of an LLM app, from permission and a map of the app to a test plan, evidence, scores and a retest.
14 min readFree
From a list of tests to one real test
The last lesson, Safety benchmark datasets, ended with a promise: next, a full security test of an LLM app. Here it is. A benchmark gives you a score for a model. A security test checks one whole app, the model plus everything wired around it.
Think of a house inspection. A buyer does not only test the paint. They check the doors, the locks, the wiring and the roof, in a set order, and they write down what they find. An LLM security test is the same: a plan, an order, and notes.
First, the rule that comes before everything else. Test only systems you own or have written permission to test. No permission, no test. Everything below assumes you have it.
Key words
- LLM app
- A product that uses a large language model, like a chat helper, a search over your documents, or an assistant that can press buttons.
- Scope
- The list of what you may test and what you may not. It is written down before you begin.
- Probe
- One test message or action, sent on purpose to see how the app reacts.
- Evidence
- Proof of what happened: the exact input, the exact output, the time and the app version.
- Severity
- How bad a finding is. A small mix-up is low. Leaked customer data is high.
- Retest
- Running the same probe again after a fix, to see that the problem is really gone.
- OWASP
- A well-known non-profit community that publishes free security guides.
The seven steps
A test is not one big attack. It is seven small, calm steps. Most beginners skip the first two and the last two. Those are the ones that keep you safe and make your work useful.
- 1
Agree the scope and get permission
Write who allows the test, which app and which version, which days, and what is off limits. Use a test copy with fake data when you can.
- 2
Map the app
Draw what the app can do: chat, search over documents, tools, file upload. Note where data comes in and where it goes out. The Threat modeling lesson shows how.
- 3
Build the test plan
Pick the categories of risk that fit the features you found, and turn each into a short checklist.
- 4
Run probes by hand first
Try a few probes yourself and read the replies. You learn how the app behaves and what a normal answer looks like.
- 5
Automate what repeats
Put the probes that matter into a script so you can run them again and again. The Automated vs manual testing lesson covers the trade-offs.
- 6
Record the evidence and score it
Save every input and output. Give each finding a severity, so the team knows what to fix first.
- 7
Retest after the fix
Run the same probes again. A fix is only a fix when the retest passes.
Tip
Stop and ask if a probe could harm real users, real data or a service you do not own. A test plan is allowed to say no.
Scope and permission
Scope is a short document with three parts: what is in, what is out, and who to call if something breaks. If you test your own app on your own laptop, you write it for yourself. If you test someone else's app, you need their written yes first. The Ethics and permission lesson goes deeper.
Scope also protects you. When a probe has a side effect, such as an email sent to a stranger, the written scope shows you stayed inside the lines. Use made-up accounts and made-up data, such as a fake customer called Mango Ltd.
Map the app
You cannot test what you have not found. Walk through the app as a user and list every feature, then list what each one touches.
Chat
Free text goes in, free text comes out. This is the front door.
RAG (search over documents)
RAG means retrieval-augmented generation: the app looks up stored documents and hands them to the model as extra reading. Anyone who can add a document can add words the model will read.
Tools and uploads
Tools let the model do things, like send a message or change a record. Uploads let users hand the app new files. Both widen what can go wrong.
The map also shows trust. Ask of each input: who can write this? A user typing in chat is one person. A shared knowledge base may be written by many.
Plan the tests with the OWASP list
You do not have to invent your categories. OWASP publishes the Top 10 for LLM Applications 2025, a list of the ten risk types seen most often in LLM apps. Use its ten names as the headings of your test plan. Below, each name is quoted as OWASP writes it, with one plain check you can ask of your own app.
LLM01:2025 Prompt Injection
OWASP says it "occurs when user prompts alter the LLM’s behavior or output in unintended ways. Check: do odd instructions in chat, files or documents change the app's rules? The Prompt injection, direct and indirect lesson explains the idea.
LLM02:2025 Sensitive Information Disclosure
OWASP lists personal data, financial details, health records, confidential business data, security credentials and legal documents. Check: can a reply show private data that belongs to someone else?
LLM03:2025 Supply Chain
OWASP writes that LLM supply chains "are susceptible to various vulnerabilities, which can affect the integrity of training data, models, and deployment platforms." Check: do you know where every model, plugin and data source came from?
LLM04:2025 Data and Model Poisoning
OWASP says data poisoning "occurs when pre-training, fine-tuning, or embedding data is manipulated to introduce vulnerabilities, backdoors, or biases." Check: can a made-up fact planted in your data end up in an answer?
LLM05:2025 Improper Output Handling
OWASP means "insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream to other components and systems." Check: does anything treat the model's reply as trusted code or commands?
LLM06:2025 Excessive Agency
OWASP calls it the vulnerability that "enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM". Check: what is the worst thing each tool can do?
LLM07:2025 System Prompt Leakage
OWASP says the system prompt "should not be considered a secret, nor should it be used as a security control". Check: would anything bad happen if a user read your hidden instructions?
LLM08:2025 Vector and Embedding Weaknesses
OWASP says weaknesses in how vectors and embeddings are "generated, stored, or retrieved can be exploited". Check: can one user's stored pieces show up in another user's answers?
LLM09:2025 Misinformation
OWASP says it "occurs when LLMs produce false or misleading information that appears credible." Check: when the app is wrong, does anyone notice or check?
LLM10:2025 Unbounded Consumption
OWASP says it "occurs when a Large Language Model (LLM) application allows users to conduct excessive and uncontrolled inferences". Check: what stops one user from running up your bill or slowing everyone down?
Several of these have their own lessons: Prompt injection, direct and indirect, Data poisoning, Model inversion and data leaks, Insecure model files and Model stealing. Here you only need the headings and the question each one asks.
The web side of the app is older, and it has its own guide. The OWASP Web Security Testing Guide calls itself "the premier cybersecurity testing resource for web application developers and security professionals." Use it for the normal parts of your app, like login and file handling, and use the list above for the model parts.
Worked example: a test-plan generator
A plan can grow fast. Four features and ten categories give forty pairs, and many checks repeat. So let us write a small program that does the crossing for us. It is plain Python with no model, no network and no keys. The rules are a hand-written table: a feature, a category and one check.
Two things to notice in the code. The rules only list pairs that make sense, so "upload" is not crossed with every category. And a check that repeats across features is merged, with the features listed next to it. That is the deduplication.
FEATURES = ["chat", "rag", "tools", "upload"]
# Each rule: (feature, OWASP-style category, what to check).
RULES = [
("chat", "LLM01 Prompt Injection", "Odd instructions typed into the chat do not change the rules"),
("rag", "LLM01 Prompt Injection", "Odd instructions hidden in a stored document do not change the rules"),
("tools", "LLM01 Prompt Injection", "Odd instructions cannot trigger a tool call"),
("upload", "LLM01 Prompt Injection", "Odd instructions hidden in an uploaded file do not change the rules"),
("chat", "LLM02 Sensitive Information Disclosure", "Replies hold no private data of other users"),
("rag", "LLM02 Sensitive Information Disclosure", "Replies hold no private data of other users"),
("tools", "LLM03 Supply Chain", "Every plugin and model comes from a known, reviewed source"),
("rag", "LLM03 Supply Chain", "Every data source comes from a known, reviewed source"),
("rag", "LLM04 Data and Model Poisoning", "Fake facts planted in the knowledge base are caught"),
("upload", "LLM04 Data and Model Poisoning", "Fake facts planted in uploads never reach the knowledge base"),
("chat", "LLM05 Improper Output Handling", "Reply text is treated as plain text, never as code"),
("tools", "LLM05 Improper Output Handling", "Reply text is checked before it reaches another system"),
("tools", "LLM06 Excessive Agency", "Each tool can only do what the feature needs"),
("chat", "LLM07 System Prompt Leakage", "The hidden instructions hold no secrets"),
("rag", "LLM08 Vector and Embedding Weaknesses", "One user cannot read another user's stored pieces"),
("upload", "LLM08 Vector and Embedding Weaknesses", "One user cannot read another user's stored pieces"),
("chat", "LLM09 Misinformation", "Wrong answers are flagged or checked"),
("rag", "LLM09 Misinformation", "Wrong answers are flagged or checked"),
("chat", "LLM10 Unbounded Consumption", "Long or repeated requests are capped"),
("tools", "LLM10 Unbounded Consumption", "Long or repeated requests are capped"),
("upload", "LLM10 Unbounded Consumption", "Long or repeated requests are capped"),
]
def build_plan(features):
plan = {} # (category, check) -> features that need it
for feature, category, check in RULES:
if feature in features:
plan.setdefault((category, check), []).append(feature)
return plan
plan = build_plan(FEATURES)
print("Test plan for: " + ", ".join(FEATURES))
print(len(RULES), "feature + category pairs ->", len(plan), "unique checks")
last = None
for (category, check), feats in sorted(plan.items()):
if category != last:
print()
print(category)
last = category
print(" [ ]", check, "(" + "+".join(feats) + ")")
print()
per_category = {}
for category, _ in plan:
per_category[category] = per_category.get(category, 0) + 1
busiest = max(per_category, key=per_category.get)
print("Most checks:", busiest, "-", per_category[busiest])
print("Only chat:", len(build_plan(["chat"])), "checks")Output
Test plan for: chat, rag, tools, upload 21 feature + category pairs -> 16 unique checks LLM01 Prompt Injection [ ] Odd instructions cannot trigger a tool call (tools) [ ] Odd instructions hidden in a stored document do not change the rules (rag) [ ] Odd instructions hidden in an uploaded file do not change the rules (upload) [ ] Odd instructions typed into the chat do not change the rules (chat) LLM02 Sensitive Information Disclosure [ ] Replies hold no private data of other users (chat+rag) LLM03 Supply Chain [ ] Every data source comes from a known, reviewed source (rag) [ ] Every plugin and model comes from a known, reviewed source (tools) LLM04 Data and Model Poisoning [ ] Fake facts planted in the knowledge base are caught (rag) [ ] Fake facts planted in uploads never reach the knowledge base (upload) LLM05 Improper Output Handling [ ] Reply text is checked before it reaches another system (tools) [ ] Reply text is treated as plain text, never as code (chat) LLM06 Excessive Agency [ ] Each tool can only do what the feature needs (tools) LLM07 System Prompt Leakage [ ] The hidden instructions hold no secrets (chat) LLM08 Vector and Embedding Weaknesses [ ] One user cannot read another user's stored pieces (rag+upload) LLM09 Misinformation [ ] Wrong answers are flagged or checked (chat+rag) LLM10 Unbounded Consumption [ ] Long or repeated requests are capped (chat+tools+upload) Most checks: LLM01 Prompt Injection - 4 Only chat: 6 checks
Read the output. The table holds 21 feature and category pairs, but merging repeats leaves 16 unique checks. Prompt injection has the most, 4, because every way of putting words in front of the model needs its own probe. A chat-only app needs just 6 checks. Add a feature and the plan grows by itself.
Tip
The table is a starting point. Edit the rules for your app, then read the checklist and ask what is missing.
Run probes, manual then automated
Start by hand. Send a handful of probes for each checklist line, using harmless made-up text, such as asking the app to reveal a fake secret word like "mango-42" that you planted in a test document. Read every reply. You learn the app's normal voice, and you spot odd behaviour no script would flag.
Then automate what repeats. A probe becomes a small test: input, expected safe behaviour, and a pass or fail check. Run the set after every change to the app. The Continuous security testing lesson shows how to wire it in, and Red teaming tools lists ready-made harnesses.
Be ready for surprises. Real models do not give the same reply twice. A single pass is weak proof, so run important probes several times and keep every result.
Record evidence and score severity
A finding without proof is just a rumour. For each one, save the exact input, the exact reply, the date, the app version and the steps to repeat it. Remove real personal data from your notes. Fake data only.
To order the fixes, give each finding a severity score. A simple, common idea is to multiply how likely it is by how much harm it does:
In words: a problem that is very likely and very harmful rises to the top, while a rare, tiny one waits. Score each from 1 to 3 and you get a number from 1 to 9. The Risk and vulnerability assessment lesson goes deeper.
OWASP is honest about limits. On prompt injection it says "it is unclear if there are fool-proof methods of prevention". So expect a fix to reduce a risk, not erase it. That is why the next step matters.
Fix, then retest
Hand the findings to the team, ordered by severity. For many fixes, OWASP's advice is plain: for example, on output handling it says to "treat the model as any other user", and on cost it says to "apply rate limiting and user quotas". After each fix, run the same probes again. If one still fails, reopen the finding. If all pass, note it and keep the probe in your automated set, so the bug cannot quietly come back.
Finish with a short report: scope, method, findings with evidence and severity, fixes and retest results. The Writing the report lesson shows the shape.
Quick check
Quick check
What comes before any probe is sent?
Quick check
In the generator, why did 21 pairs become 16 checks?
Quick check
Why retest after a fix?
What you learned
Permission first
Write the scope, get a yes, use fake data, and test only what you own or may test.
Map, then plan
List the features, then use the ten OWASP names as headings and write one check for each fitting pair.
Evidence and retest
Probe by hand, automate what repeats, save proof, score severity, and run the probes again after each fix.
Next: writing your own red team scripts.
Sources
- OWASP Top 10 for LLM Applications 2025
- OWASP: LLM01:2025 Prompt Injection
- OWASP: LLM02:2025 Sensitive Information Disclosure
- OWASP: LLM03:2025 Supply Chain
- OWASP: LLM04:2025 Data and Model Poisoning
- OWASP: LLM05:2025 Improper Output Handling
- OWASP: LLM06:2025 Excessive Agency
- OWASP: LLM07:2025 System Prompt Leakage
- OWASP: LLM08:2025 Vector and Embedding Weaknesses
- OWASP: LLM09:2025 Misinformation
- OWASP: LLM10:2025 Unbounded Consumption
- OWASP Web Security Testing Guide
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.