esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Beginner / 01.07

Risk and vulnerability assessment

How to decide which weak spots in an AI app to fix first: score each risk, rank them, pick a response, and use the standard frameworks.

13 min readFree

Deciding what to fix first

The last lesson, Threat modeling, ended with a promise: next, risk management, deciding what to fix first. Here it is.

Threat modeling gives you a long list of things that could go wrong. A long list is a problem on its own. You have limited time, so you cannot fix everything on Monday. Risk management is how you decide what goes first, what can wait, and what you will live with.

One rule before we start, and it applies to the whole course: test only systems you own, or systems where you have written permission to test. Everything here uses made-up apps and made-up numbers.

Key words

Risk
Something that might go wrong, and how bad it would be if it did.
Likelihood
How likely it is to happen. We will score it from 1 (very unlikely) to 5 (very likely).
Impact
How much damage it does if it happens. We will score it from 1 (tiny) to 5 (severe).
Vulnerability
A weak spot that something bad could use. A door with a broken lock is a vulnerability.
Risk register
A simple table that lists your risks, their scores and what you plan to do about each.

A made-up example runs through this lesson: Plumfield Books, a small online shop with a chat helper that answers customer questions. It is not a real company.

What a risk is

Think about carrying an umbrella. If rain is very likely, you take it. If rain is unlikely and a little wet hair is no big deal, you skip it. If you are carrying a laptop and a storm is likely, you take a bag as well. Two questions drive each choice: how likely is it, and how bad would it be?

That is all a risk score is. One common, simple way to put it in numbers is to multiply the two answers. In words: the risk score is the likelihood multiplied by the impact.

risk score=likelihood×impact\text{risk score} = \text{likelihood} \times \text{impact}

With both scored from 1 to 5, the lowest possible score is 1 and the highest is 25. A very likely, very harmful problem scores 25. A very unlikely, tiny problem scores 1.

This is a teaching tool, not a law of nature. The numbers are guesses made by people, so two teams can score the same risk differently. The value is that you write the guess down, argue about it in the open, and rank everything the same way.

The Wikipedia page on risk management defines the whole field in one sentence:

"Risk management is the identification, evaluation, and prioritization of risks, followed by the minimization, monitoring, and control of the impact or probability of those risks occurring."

Notice the last part. You can make a risk smaller in two ways: lower the chance it happens, or lower the damage when it does.

Tip

Score the risk as it is today, with the protections you already have. Not the worst imaginary day, and not the day after you fix it.

The risk register

A risk register is a table. It is the most useful, least exciting document in security. Here is how to build one for an AI app.

  1. 1

    List the risks

    Use your threat model. Write each risk as one plain sentence: what goes wrong, to what. Leave out how clever the attack is.

  2. 2

    Score likelihood

    Give each risk 1 to 5. Ask: how easy is it to reach, and how often might it happen? Ask a teammate too.

  3. 3

    Score impact

    Give each risk 1 to 5. Ask: who is hurt, how badly, and how hard is it to undo?

  4. 4

    Multiply and sort

    Likelihood times impact gives the score. Sort from biggest to smallest.

  5. 5

    Make buckets

    Draw lines to make groups, such as high, medium and low. Decide what each group means for your calendar.

  6. 6

    Pick a response

    For each risk, choose one of the four responses below, and name who is in charge.

A register in Python

Here is a register for Plumfield Books with six made-up risks. Each has a likelihood and an impact from 1 to 5. The code multiplies, sorts, and puts each risk in a bucket: 15 or more is high, 8 to 14 is medium, and 7 or less is low. Those cut-off lines are our own choice for this example, not a standard.

No AI model is involved. The register is just data, so it runs free on any laptop.

Python
# Risk register for a made-up app: "Plumfield Books" chat helper.
# Each row: (risk, likelihood 1-5, impact 1-5). All numbers are made up.
register = [
    ("Chat helper repeats a hidden instruction", 4, 2),
    ("Wrong answer about a refund rule", 3, 4),
    ("Customer details land in the chat log", 2, 5),
    ("Helper is tricked into skipping a check", 3, 5),
    ("Chat is slow on a busy day", 4, 1),
    ("Old price list gets used in answers", 2, 3),
]

def bucket(score):
    if score >= 15:
        return "high"
    if score >= 8:
        return "medium"
    return "low"

rows = [(name, lik, imp, lik * imp) for name, lik, imp in register]
rows.sort(key=lambda r: r[3], reverse=True)

print(f"{'score':>5}  {'L':>1}  {'I':>1}  {'level':<6}  risk")
for name, lik, imp, score in rows:
    print(f"{score:>5}  {lik}  {imp}  {bucket(score):<6}  {name}")

Output

score  L  I  level   risk
   15  3  5  high    Helper is tricked into skipping a check
   12  3  4  medium  Wrong answer about a refund rule
   10  2  5  medium  Customer details land in the chat log
    8  4  2  medium  Chat helper repeats a hidden instruction
    6  2  3  low     Old price list gets used in answers
    4  4  1  low     Chat is slow on a busy day

Read the table from the top. The highest score is 15, for the helper being tricked into skipping a check. Its likelihood is 3 and its impact is 5. The lowest is 4, for a slow chat on a busy day: likelihood 4, impact 1.

Now look at the middle. "Customer details land in the chat log" has the biggest possible impact, 5, but only a likelihood of 2. Its score is 10, so it lands in medium. "Chat helper repeats a hidden instruction" is the opposite: likely (4) but low impact (2), score 8. That is also medium, right on the line. Different stories, same bucket. The score is a starting point for a conversation, not the end of it.

  • High: 15 or more

    Only one risk is here. Work on it first. In this example, one risk gets the next sprint's attention.

  • Medium: 8 to 14

    Three risks. Plan them. Give each an owner and a date, and watch for changes in either number.

  • Low: 7 or less

    Two risks. Note them, write down that you chose to wait, and look again at the next review.

Tip

Keep the register in a plain file next to your code, and re-score it after every big change to the app. A register nobody looks at is only decoration.

The code uses a stand-in for the real world: the scores are made up by us. In a real project they come from your own testing, your logs and your team's judgment, and they change over time.

Four things you can do about a risk

Once a risk has a score, you must choose a response. The Wikipedia page says strategies for threats "typically include avoiding the threat, reducing the negative effect or probability of the threat, transferring all or part of the threat to another party, and even retaining some or all of the potential or actual consequences of a particular threat." In daily words: avoid, reduce, transfer, accept.

  • Avoid

    Do not do the risky thing at all. Example: the chat helper does not get any ability to issue refunds. The page also warns that avoiding a risk can mean losing the benefit that taking it would have brought.

  • Reduce

    Make it less likely or less harmful. Example: add a second check before any sensitive step, or keep customer details out of the chat log. This is what most security work is.

  • Transfer

    Share the risk with another party. Wikipedia's example is an insurance company. For an app, it could also be an agreement where a service provider takes on part of the job. The risk is shared, not gone: you still answer to your customers.

The fourth is accept, which Wikipedia calls retention: you keep the risk on purpose, and you budget for it. Accepting is a real choice only if you wrote it down and someone with the right authority agreed. Ignoring a risk is not acceptance.

Reducing is the one you will use most. Here is the effect of a fix on our top risk. A second, separate check now guards the step, so the likelihood drops from 3 to 1. The impact stays at 5, because if it did happen, it would still hurt just as much.

Python
# Re-score one risk after a fix. All numbers are made up.
before = {"likelihood": 3, "impact": 5}
after = {**before, "likelihood": 1}  # a second, separate check now guards the step

def score(r):
    return r["likelihood"] * r["impact"]

print("before:", score(before))
print("after: ", score(after))
print("drop:  ", score(before) - score(after))

Output

before: 15
after:  5
drop:   10

The score falls from 15 to 5, a drop of 10. In the first register, 5 would sit in the low bucket. Notice what the fix did: it did not change how bad the harm would be, only how likely it is. Sometimes you can change the impact instead, for example by limiting what the helper is allowed to touch.

The NIST framework: four functions

You do not have to invent a process. NIST, the US National Institute of Standards and Technology, wrote one for AI. Its page says the AI Risk Management Framework (AI RMF) is "intended for voluntary use" and exists to improve the ability to build trustworthiness into the design, development, use and evaluation of AI products, services and systems. It was released on January 26, 2023. NIST says the AI RMF 1.0 is being updated, so check the current version before you rely on details.

The AI RMF Core is made of four functions: govern, map, measure and manage. Here is what each does, in plain words based on NIST's own descriptions.

  • Govern

    Build a culture and clear rules for managing risk: who is responsible, what the policies are, how much risk the organisation will tolerate. NIST calls it a cross-cutting function that runs through all the others.

  • Map

    NIST says map establishes the context to frame risks related to an AI system. Your threat model fits here: what is the system for, who is affected, and what could go wrong?

  • Measure

    Use tests and other tools to analyse, assess, benchmark and monitor the risks you mapped. NIST says AI systems should be tested before deployment and regularly while in operation.

  • Manage

    Spend your time and money on the risks you mapped and measured. NIST says risk treatment includes plans to respond to, recover from, and communicate about incidents.

Two cautions from NIST itself. The actions are not a checklist, and they are not necessarily an ordered set of steps. And the work is iterative: after the govern basics are in place, most users would start with map and continue to measure or manage, then come back around.

Our register fits neatly inside. Listing the risks is map. Scoring them is measure. Choosing a response is manage. Deciding who owns the register and how often it is reviewed is govern.

Quick check

Which NIST function is mostly about who is responsible and what the rules are?

Vulnerability assessment

Risk management asks "what could go wrong?" A vulnerability assessment asks a narrower question: "where are our weak spots right now?" The Wikipedia page defines it as "the process of identifying, quantifying, and prioritizing (or ranking) the vulnerabilities in a system." This one is the same ranking idea you just used, applied to weak spots in a system.

Wikipedia lists the usual steps, and they sound a lot like our register:

  1. 1

    Catalogue what you have

    List the assets and capabilities in the system. For an AI app: the model, the prompt, the data it can reach, the tools it can call, the logs.

  2. 2

    Give each one a value

    Assign a quantifiable value, or at least a rank order, and note how important each is.

  3. 3

    Find the weak spots

    Identify the vulnerabilities or potential threats to each asset.

  4. 4

    Fix the worst first

    Mitigate or eliminate the most serious vulnerabilities for the most valuable assets.

Notice that value comes before weak spots. A hole in something nobody cares about is a smaller worry than a small crack in something precious.

The same page makes a point worth remembering. Automated tools give you baseline visibility, but they struggle with complex application logic, and automated scanners frequently generate false positives (alarms about things that are not really problems). So pair automated scanning with checks by a person.

A great deal of AI-specific weakness lives in how an app is built around the model. The Prompt injection attacks lesson in AI engineering goes deeper on one big family of them. Here, the security angle is simple: every weak spot you find goes into the register like any other.

Scoring weak spots with CVSS

Teams that share weak spots need a shared way to score them. The Common Vulnerability Scoring System, or CVSS, is that. The CVSS special interest group at FIRST says CVSS "provides a way to capture the principal characteristics of a vulnerability and produce a numerical score reflecting its severity." The number can then be turned into a word such as low, medium, high or critical, "to help organizations properly assess and prioritize their vulnerability management processes."

FIRST's FAQ says the scores fall between 0.0 and 10.0. FIRST's page adds that CVSS is a published standard used by organizations worldwide, and that it is currently at version 4.0.

Two things to keep straight. First, CVSS scores a vulnerability, a weak spot. Our register scores a risk, a weak spot in your own setting with your own users. Same family, different jobs. Second, the CVSS FAQ is clear about its limits: it is designed to score cybersecurity vulnerabilities, and it should not be used to score ethical or legal vulnerabilities. For an AI app, an unfair or harmful answer is a real risk. It belongs in your register, scored by your team, not forced into CVSS.

Quick check

A chat helper sometimes gives unfair answers to some groups of customers. Where does this go?

Putting it together

Here is the loop you will keep using for the rest of this course: find the weak spots, score them, sort them, pick a response, fix, and then score again. The worst few get attention first. Everything else is written down, owned and reviewed.

Quick check

Risk A: likelihood 5, impact 2. Risk B: likelihood 2, impact 5. Using likelihood times impact, which is bigger?

That last question is why we keep the two scores in separate columns. A risk that is nearly certain but mildly annoying needs a different plan from one that is rare but severe.

Next: a map of how AI apps break.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.