AI guardrails & red teaming / Advanced / 03.11
Standards and frameworks
Meet the main public standards for AI security and risk, what each one is for, and how to check which ones your app's safeguards help with.
13 min readFree
Why the industry writes things down
The last lesson, Responsible disclosure, ended with a promise: next, the standards and frameworks the industry uses. Here they are.
So far you have learned to find weak spots, report them kindly and fix them. Now zoom out. Other people have already written down what to look for and how to organise the work. You do not have to invent your own checklist from scratch.
One rule before we start, and it applies to the whole course: test only systems you own, or systems where you have written permission to test. Everything here uses made-up apps.
Think of building codes: a shared document says how strong a staircase rail must be, so builders and inspectors use the same words. These standards do that job for AI apps.
Key words
- Standard
- A document that a group agrees on, so many teams can use the same words and the same checklist.
- Framework
- A set of ideas and steps for organising a job, such as managing risk. You adapt it to your own project.
- Voluntary
- Nobody forces you to follow it. You choose to, because it helps.
- Regulation
- A law. Unlike a voluntary framework, you must follow it if it applies to you.
- Control
- One safeguard you put in place, such as a filter, a rule or a human check.
- Coverage
- How many of the things you care about your safeguards actually help with.
Our made-up example is still Plumfield Books, a small online shop with a chat helper. It is not a real company. We will use it at the end.
Three kinds of document
The names can look like alphabet soup. It helps to sort them into three kinds before you meet each one.
Lists and maps
They name the things that go wrong. OWASP's list of top risks and MITRE ATLAS are in this group. Use them to decide what to test.
Processes
They describe how to organise the work: who decides, what you measure, what you fix. NIST's AI Risk Management Framework and ISO/IEC 42001 are in this group.
Law
A regulation says what you must do. The EU Artificial Intelligence Act is the one we look at. Nothing else here is law.
A standard does not make your app safe by itself. It is a map, and your own testing (see Red team exercises) tells you what is true for your app.
Tip
All facts here were checked on 16 November 2026 against the pages listed under Sources. Standards get new versions, so check the publisher's page for the latest before you rely on a detail.
OWASP Top 10 for LLM Applications
OWASP publishes the Top 10 for Large Language Model Applications through its Gen AI Security Project. The project's page says the list "started in 2023 as a community-driven effort to highlight and address security issues specific to AI applications." The current edition on its site is the 2025 one.
Here are the ten entries, with their exact names from OWASP's page and a plain-words idea of each. The idea is our short reading, so open OWASP's page for the full text.
Key words
- LLM01 Prompt Injection
- Text sent to the model changes what it does, in a way you did not plan. The Prompt injection, direct and indirect lesson goes deeper.
- LLM02 Sensitive Information Disclosure
- Private data comes out of the model or the app. See Model inversion and data leaks.
- LLM03 Supply Chain
- The models, data and tools you borrowed bring their own weak spots. See Insecure model files.
- LLM04 Data and Model Poisoning
- Bad data goes into training or the data an app reads. See Data poisoning.
- LLM05 Improper Output Handling
- The app trusts what the model writes without checking it first.
- LLM06 Excessive Agency
- The AI is given more power to act than the job needs.
- LLM07 System Prompt Leakage
- Hidden instructions you gave the model get shown to users.
- LLM08 Vector and Embedding Weaknesses
- Weak spots in the stores that help a model look things up.
- LLM09 Misinformation
- The model gives answers that sound sure but are wrong.
- LLM10 Unbounded Consumption
- Nothing limits how much of the model's work one user can use up. See Protecting AI APIs.
Most lessons in this course teach one of these ideas in depth, which makes the Top 10 a good first checklist for a test plan.
Quick check
You want a short checklist of AI app risks to test first. Which document is the best fit?
NIST AI Risk Management Framework
NIST is the US National Institute of Standards and Technology. Its AI Risk Management Framework, the AI RMF, is "intended for voluntary use" to improve how teams add trustworthiness to the design, development, use and evaluation of AI products, services and systems. NIST released it on January 26, 2023, after a process that included public drafts and workshops.
The Risk and vulnerability assessment lesson showed its four functions. Think of the AI RMF as the process: who owns risk, how you find it, measure it and act on it.
NIST's page also notes three dated things worth knowing. NIST released a Generative AI Profile, NIST AI 600-1, on July 26, 2024. It is a companion for the AI RMF aimed at generative AI, which is the kind of AI behind chat helpers. NIST released a concept note on April 7, 2026 for a profile on trustworthy AI in critical infrastructure. And the page says the AI RMF 1.0 is being revised as part of the White House AI Action Plan.
A profile, here, is a version of a framework tuned to one setting. NIST's own page says the generative AI profile "can help organizations identify unique risks posed by generative AI."
Tip
Because the AI RMF 1.0 is being revised, check NIST's page for the current version before you quote section numbers to a customer or a boss.
MITRE ATLAS
MITRE ATLAS is a different kind of document. The project's own data page calls it "a public knowledge base of adversary TTPs targeting AI systems." ATLAS stands for Adversarial Threat Landscape for AI Systems. TTP is short for tactics, techniques and procedures, which is a way of saying how attackers think and act.
ATLAS describes a few parts in simple terms. Here they are, from the project's data guide.
Key words
- Tactic
- The goal, the "why" behind an activity, such as Reconnaissance (finding things out).
- Technique
- A reusable method for carrying out an activity: the "how" behind one or more tactics.
- Mitigation
- A defence that can prevent, limit or reduce the impact of a technique.
- Case study
- A write-up of a specific use of techniques.
- Matrix
- The ordered display of tactics, with techniques listed under them.
Why it helps: during Threat modeling you ask what an attacker wants and how they might get it. ATLAS gives that conversation shared words. In a test report (see Writing the report) you can name each finding with the same technique name another team would use.
ISO/IEC 42001
ISO and IEC are international groups that publish standards. Wikipedia describes ISO/IEC JTC 1/SC 42 as the standardization subcommittee for artificial intelligence under the joint committee of the two groups, and lists ISO/IEC 42001 among its standards as "Information technology — Artificial intelligence — Management system", published in 2023.
A fair warning: ISO's own page blocked our fetch, so that listing is all we state as fact. We claim nothing about its contents, fees or audits.
What can we say safely? The title tells you the type. This is a process-style document about a management system for AI. A management system, in plain words, is how an organisation runs a topic as a habit: who is in charge, what is written down, what gets checked. Think of a restaurant's kitchen routine: the same checks every day, written down, with someone responsible. For details, read the standard itself.
The EU AI Act: the one that is law
Everything above is voluntary. The EU AI Act is different. Wikipedia describes it as a European Union regulation, Regulation (EU) 2024/1689, made by the European Parliament and the Council. It "entered into force on 1 August 2024." It places duties on AI providers and on organisations that use AI in a professional context.
Wikipedia says the Act sorts AI applications by their risk of causing harm. There are four levels: unacceptable, high, limited and minimal. There is an additional category for general-purpose AI.
Unacceptable risk
These applications are banned, with specific exceptions.
High risk
These must meet security, transparency and quality duties, and go through checks called conformity assessments.
Limited and minimal risk
Limited-risk applications have only transparency duties. Minimal-risk applications are not regulated.
Duties switch on in stages. Wikipedia's page lists a delay of 6 months for the bans, 12 months for general-purpose AI, 36 months for some high-risk duties and 24 months for everything else. It also lists fines: for breaking the bans, up to EUR 35,000,000 or 7% of worldwide yearly turnover, whichever is higher. Dates and rules can be changed, so read the official text for anything that matters.
Tip
This lesson is not legal advice. If your app serves people in the EU, or you wonder which risk level it falls into, ask a qualified lawyer and read the official text.
Quick check
Which one of these is a law you must follow if it applies to you?
All five side by side
Here is a small table, built with a tiny program so the output is real. It shows what each document is, who publishes it, and when you would reach for it. Every row comes from the sources above.
# One row per standard. Facts come from the pages listed under Sources.
rows = [
("OWASP LLM Top 10", "list of the top 10 risks", "OWASP Gen AI project", "choose what to test"),
("NIST AI RMF", "voluntary risk process", "NIST (US)", "plan how to manage risk"),
("NIST AI 600-1", "generative AI profile", "NIST (US)", "adapt the RMF to LLM apps"),
("MITRE ATLAS", "knowledge base of attacker moves", "MITRE", "write threat models and tests"),
("ISO/IEC 42001", "AI management system", "ISO and IEC", "run AI work as company rules"),
("EU AI Act", "a law, not a guide", "EU Parliament, Council", "check your legal duties"),
]
widths = [18, 28, 23, 30]
head = ("Name", "What it is", "Who publishes", "Use it to")
for r in [head] + rows:
print(" ".join(c.ljust(w) for c, w in zip(r, widths)).rstrip())Output
Name What it is Who publishes Use it to OWASP LLM Top 10 list of the top 10 risks OWASP Gen AI project choose what to test NIST AI RMF voluntary risk process NIST (US) plan how to manage risk NIST AI 600-1 generative AI profile NIST (US) adapt the RMF to LLM apps MITRE ATLAS knowledge base of attacker moves MITRE write threat models and tests ISO/IEC 42001 AI management system ISO and IEC run AI work as company rules EU AI Act a law, not a guide EU Parliament, Council check your legal duties
Read the "Use it to" column as a hint, not a rule. Real projects often use several at once.
Worked example: which frameworks do your safeguards touch?
Teams often ask: we did all this work, which standards does it help with? Let us answer for Plumfield Books with plain Python. Six made-up controls go in a dictionary. Each control maps to a set of the frameworks it helps with.
Two honest notes. The mapping is our own teaching guess, not an official crosswalk. And no AI model is involved, so the code runs free and the answers never change. Real compliance work uses the official documents.
- 1
List the controls
Write down each safeguard you actually have. Six is plenty for a small app.
- 2
Map each to frameworks
For each control, note which of the five documents it supports. Use a set, so a framework cannot be listed twice.
- 3
Count per framework
Add up how many controls help each one.
- 4
Spot the gaps
Mark any framework with fewer than three controls. The number three is just our choice for this example.
- 5
Add a control and re-check
Fix the gap, run the check again and see it close.
# Made-up controls for a made-up app, "Plumfield Books".
# The mapping below is our own teaching guess, not an official crosswalk.
frameworks = ["OWASP", "NIST", "ATLAS", "ISO", "EU"]
controls = {
"Filter and log chat inputs": {"OWASP", "ATLAS"},
"Written AI use policy": {"NIST", "ISO", "EU"},
"Monthly red team test": {"OWASP", "NIST", "ATLAS"},
"Human approves refunds": {"OWASP", "EU"},
"Model and data inventory": {"NIST", "ISO"},
"Notice: you are talking to AI": {"EU"},
}
def report(controls):
print(f"{'control':<30}" + " ".join(f"{f:>5}" for f in frameworks))
for name, covered in controls.items():
print(f"{name:<30}" + " ".join(f"{'x' if f in covered else '-':>5}" for f in frameworks))
counts = {f: sum(f in c for c in controls.values()) for f in frameworks}
print(f"{'controls per framework':<30}" + " ".join(f"{counts[f]:>5}" for f in frameworks))
gaps = [f for f in frameworks if counts[f] < 3]
print("Gaps (fewer than 3 controls):", ", ".join(gaps) if gaps else "none")
return gaps
print("Before")
report(controls)
controls["Map attacks to ATLAS in tests"] = {"ATLAS", "OWASP"}
controls["Run an internal AI audit"] = {"ISO", "NIST"}
print()
print("After adding two controls")
report(controls)Output
Before control OWASP NIST ATLAS ISO EU Filter and log chat inputs x - x - - Written AI use policy - x - x x Monthly red team test x x x - - Human approves refunds x - - - x Model and data inventory - x - x - Notice: you are talking to AI - - - - x controls per framework 3 3 2 2 3 Gaps (fewer than 3 controls): ATLAS, ISO After adding two controls control OWASP NIST ATLAS ISO EU Filter and log chat inputs x - x - - Written AI use policy - x - x x Monthly red team test x x x - - Human approves refunds x - - - x Model and data inventory - x - x - Notice: you are talking to AI - - - - x Map attacks to ATLAS in tests x - x - - Run an internal AI audit - x - x - controls per framework 4 4 3 3 3 Gaps (fewer than 3 controls): none
Read the first table. ATLAS and ISO each have only 2 controls, so they are the gaps. EU has 3, but look at its column: one of those controls is only a notice to users, which is thin.
Then two controls are added. "Map attacks to ATLAS in tests" pushes ATLAS from 2 to 3, and "Run an internal AI audit" pushes ISO from 2 to 3. OWASP and NIST each rise to 4. The last line now says there are no gaps.
Putting it together
Use standards as a map. OWASP helps you choose what to test, ATLAS names how attackers might get in, NIST organises the risk work, ISO/IEC 42001 turns it into company habit, and the EU AI Act is the legal check.
Quick check
In the coverage example, why did we re-run the check after adding two controls?
Next: new and advanced threats to watch.
Sources
- OWASP Gen AI Security Project: OWASP Top 10 for LLM Applications 2025 (list page)
- OWASP Gen AI Security Project: OWASP Top 10 for LLM Applications 2025 (resource page)
- NIST: AI Risk Management Framework
- NIST: AI RMF Generative Artificial Intelligence Profile (NIST AI 600-1)
- MITRE ATLAS data repository (GitHub)
- Wikipedia: ISO/IEC JTC 1/SC 42
- Wikipedia: Artificial Intelligence Act
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.