esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Advanced / 03.07

Agentic AI security

What changes when an AI model can act, not just talk: the main threats, and the limits, approvals and records that keep it safe.

13 min readFree

From answers to actions

The last lesson, Writing your own test scripts, ended with a promise: next, security for AI agents that take actions. Here it is.

Until now, most of this course was about a model that talks. You send a message, it sends words back. The worst case is bad words. An agent is different. An agent is a model that can also do things: search files, send emails, run code, spend money. The worst case is now a bad action, and actions are harder to take back.

Think of the difference between a friend who gives you advice and a friend who has your bank card and the keys to your flat. Both can be wrong. Only one can empty the flat.

One rule for this whole course comes first: test only systems you own or have written permission to test. Everything below is about building and checking your own agents. The code is made up, runs on your laptop, and uses no network and no keys.

Key words

Agent
A program that uses a model to decide what to do next, and can act through tools, often over many steps.
Tool
One action the agent is allowed to call, such as search notes, send an email or delete a file.
Agency
How much an agent is allowed to do on its own.
Memory
Notes the agent keeps between steps or between chats, so it can use them later.
Policy engine
Plain code that sits between the agent and its tools and says allow or block. It does not use a model, so it cannot be talked round.
Human in the loop
A person who must approve a risky action before it happens.
Sandbox
A locked room where code can run without touching anything outside it.
Audit trail
A record of every action the agent took and who or what allowed it.

What changes when a model can act

Four things grow when you give a model hands.

  • Tools

    Each tool is a door to a real system. More tools means more doors, and each door needs a lock.

  • Memory

    What the agent writes down today shapes what it does tomorrow. A bad note can stay for weeks.

  • Many agents

    Agents can pass messages to each other. If one is fooled, it can pass the trick along.

  • Long runs

    An agent can take hundreds of steps without anyone watching. A small mistake can repeat until it is big.

OWASP, a nonprofit that publishes security guides, describes the starting point this way: an LLM-based system "is often granted a degree of agency by its developer", meaning the ability to call functions or interface with other systems via extensions, which some vendors call tools, skills or plugins. It adds that agent-based systems "will typically make repeated calls to an LLM using output from previous invocations to ground and direct subsequent invocations." In plain words: the model's last answer feeds its next move, so one bad answer can steer all the steps after it.

OWASP also has a separate document for this area. Its page says agentic AI "represents an advancement in autonomous systems" and that the document "is the first in a series of guides from the OWASP Agentic Security Initiative (ASI) to provide a threat-model-based reference of emerging agentic threats and discuss mitigations." The page itself shows only that short summary, so this lesson relies on the Excessive Agency page for the details.

Tip

Prompt injection (hidden instructions inside content the model reads) is the engine behind many agent attacks. The Prompt injection lesson and the Prompt injection defences lesson cover it. Here we ask a different question: if the model is fooled, how much damage can it do?

Excessive agency

OWASP lists this risk as LLM06:2025 Excessive Agency. It defines it as "the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM, regardless of what is causing the LLM to malfunction."

Notice the last part. It does not matter why the model went wrong. A confused model, a poor prompt or a tricked model all lead to the same place. Your job is to limit what a wrong move can cost.

OWASP says the root cause is typically one or more of these three things:

  • Excessive functionality

    The agent has tools it does not need, such as a mail tool that can send when the job is only to read.

  • Excessive permissions

    The tools can do more than the job needs, such as full database access when read-only would do.

  • Excessive autonomy

    High-impact actions happen with no check. OWASP's example is an extension that deletes a user's documents with no confirmation.

OWASP gives a story. A personal assistant app reads a mailbox to summarise emails. The plugin it uses can also send messages. A booby-trapped incoming email tricks the model into forwarding sensitive mail to an attacker. OWASP says this could be avoided by "eliminating excessive functionality by using an extension that only implements mail-reading capabilities", by "eliminating excessive permissions" with a read-only login, and by "eliminating excessive autonomy by requiring the user to manually review and hit 'send'" on every drafted mail.

Quick check

A summary agent only needs to read emails, but its mail tool can also send and delete. Which root cause is this?

The main threats

Here are the threats students meet most. Each is described as an idea, with how to spot it.

Tool misuse

The agent uses a real tool in a harmful way, even though the tool is on its list. Maybe it was fooled, or maybe it just guessed wrong. Think of a cashier who is allowed to give refunds and gives one to a stranger who sounds sure of themselves. Spot it by comparing what the tool was used for with what the task needed.

Memory poisoning

Someone gets a false or harmful note into the agent's memory. Later, the agent reads it back and trusts it, because it looks like its own handwriting. It is like someone slipping a fake page into your diary. Spot it by logging where each memory came from, and by not letting content from strangers write notes that look like instructions.

Privilege escalation through other agents

Privilege means the rights an agent has. In a team of agents, a weak agent with few rights may ask a strong agent with many rights to do something for it. If the strong agent says yes without checking, the weak one just borrowed power it was never given. It is like a visitor asking the receptionist to open the manager's office. Spot it by checking every request against the rights of the original caller, not just the agent that passed it on. OWASP lists a "malicious/compromised peer agent" among the triggers of excessive agency in multi-agent systems.

The lethal trifecta

Simon Willison, a software developer who writes about AI security, gave a name to a risky mix. He says the lethal trifecta of capabilities is "Access to your private data", "Exposure to untrusted content" and "The ability to externally communicate" in a way that could be used to steal your data. He adds: "If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker."

Picture a helper who can open your drawers, reads every letter that arrives, and can post parcels anywhere. A stranger only needs to slip one letter into the pile saying, post me the contents of the top drawer.

His advice for people who mix tools is firm. He writes that we "still don't know how to 100% reliably prevent this from happening" and that "The only way to stay safe there is to avoid that lethal trifecta combination entirely." So for each agent you build, ask which of the three it has. Remove at least one.

Quick check

Your agent reads your private notes and also reads web pages. Which single change most reduces the lethal trifecta risk?

Controls that work

OWASP lists strategies such as minimizing extensions ("Limit the extensions that LLM agents are allowed to call to only the minimum necessary"), minimizing extension functionality and minimizing extension permissions. For permissions it gives this example: an agent that recommends products "might only need read access to a 'products' table; it should not have access to other tables, nor the ability to insert, update or delete records."

It also says some options "will not prevent Excessive Agency, but can limit the level of damage": logging and monitoring the activity of extensions, and rate-limiting to reduce the number of undesirable actions in a given time. Putting it together, here is a checklist.

  1. 1

    Least privilege

    Give the agent only the tools and rights the job needs. Read-only unless writing is truly required.

  2. 2

    Approvals

    A person must say yes before deletes, payments, sends to new people, or anything hard to undo.

  3. 3

    Sandboxes

    Run code the agent writes in a locked room with no network and no real files.

  4. 4

    Budgets

    Cap steps, spending and time. A runaway agent then stops on its own.

  5. 5

    Audit trail

    Record each action, its inputs, its verdict and who approved it. You cannot fix what you cannot see.

  6. 6

    Kill switch

    One button that stops all agents at once. Practise using it before you need it.

Two points matter most. First, put the controls in plain code outside the model. A model can be talked round. An if-statement cannot. Second, treat them as layers. No single control is perfect, and the next lesson's exercise tests how the layers hold together.

Try it: a policy engine

Below is a tiny agent loop with three tools, and a policy engine that checks every step. There are four rules: each tool must be on the allowed list, the agent has at most 8 steps, it has a budget of 12 coins to spend, and the delete tool needs a person to approve it. The made-up plan tries to go too far on purpose.

The plan is a hard-coded list. It is a stand-in for a model's choices, so the code runs free with no keys and no network. Real models behave less predictably, which is exactly why the checks live outside them. The human is a stand-in too: here the person always says no.

Python
# A toy agent loop with a policy engine. Everything here is made up and runs offline.
TOOLS = {
    "search_notes": {"cost": 1},
    "send_report": {"cost": 5},
    "delete_note": {"cost": 2},
}
ALLOWED = {"search_notes", "send_report", "delete_note"}  # per-tool permission
NEEDS_APPROVAL = {"delete_note"}
MAX_STEPS = 8
BUDGET = 12

def human_says_yes(tool, arg):
    # Stand-in for a person clicking Approve or Deny. This time the person says no.
    return False

def decide(step, spent, tool, arg):
    if step > MAX_STEPS:
        return "STOP: step limit reached"
    if tool not in ALLOWED:
        return "BLOCK: tool not allowed"
    if spent + TOOLS[tool]["cost"] > BUDGET:
        return "BLOCK: over budget"
    if tool in NEEDS_APPROVAL and not human_says_yes(tool, arg):
        return "BLOCK: human said no"
    return "ALLOW"

# A made-up plan that tries to go too far (a stand-in for a model's choices).
plan = [
    ("search_notes", "mango"),
    ("search_notes", "papaya"),
    ("send_report", "team"),
    ("delete_note", "mango"),
    ("wire_money", "fig-corp"),
    ("send_report", "team"),
    ("search_notes", "kiwi"),
    ("send_report", "everyone"),
    ("search_notes", "lime"),
    ("search_notes", "plum"),
]

spent = 0
for step, (tool, arg) in enumerate(plan, start=1):
    verdict = decide(step, spent, tool, arg)
    if verdict == "ALLOW":
        spent += TOOLS[tool]["cost"]
    print(f"step {step:2d} {tool}({arg}) -> {verdict} | spent {spent}/{BUDGET}")
    if verdict.startswith("STOP"):
        break

Output

step  1 search_notes(mango) -> ALLOW | spent 1/12
step  2 search_notes(papaya) -> ALLOW | spent 2/12
step  3 send_report(team) -> ALLOW | spent 7/12
step  4 delete_note(mango) -> BLOCK: human said no | spent 7/12
step  5 wire_money(fig-corp) -> BLOCK: tool not allowed | spent 7/12
step  6 send_report(team) -> ALLOW | spent 12/12
step  7 search_notes(kiwi) -> BLOCK: over budget | spent 12/12
step  8 send_report(everyone) -> BLOCK: over budget | spent 12/12
step  9 search_notes(lime) -> STOP: step limit reached | spent 12/12

Read the output line by line. Steps 1 and 2 are cheap searches, so they pass. Step 3 sends a report and costs 5, so 7 of the 12 coins are spent. Step 4 tries to delete a note. Delete is allowed as a tool, but it needs approval, and the person says no, so it is blocked. Step 5 asks for wire_money, which is not on the allowed list at all, so it is blocked.

Step 6 is another report. It costs 5 and brings the total to exactly 12. Now the budget is full. Steps 7 and 8 are blocked because they would go over budget, even for a cheap 1-coin search. At step 9 the agent passes the limit of 8 steps, so the loop stops. Four different rules fired, each on its own, and the model never had a say.

Tip

Blocked steps cost nothing and still appear in the log. That is the audit trail at work: later you can see what the agent tried, not just what it did.

Try changing the code. Make human_says_yes return True and watch step 4 pass. Set BUDGET to 6 and see which step is the first to be blocked. Remove wire_money from the plan and see how the story changes. Small experiments like these are how you learn what each rule really does.

How to test your own agent

When you test your own agent (with permission, in a lab with fake data), ask these questions:

  • What can it touch?

    List every tool and every right. Cross out anything the job does not need.

  • Which legs of the trifecta?

    Mark private data, untrusted content and a way out. If all three are there, change the design.

  • What stops a bad run?

    Check the step limit, the budget and the kill switch. Do they actually work when you try them?

Quick check

Where should the allow or block decision for a delete be made?

To sum up, an agent is a model with hands. Limit the tools, limit the rights, ask a person before big moves, cap the steps and the spending, keep a record, and keep a way to stop it.

Next: running a full red team exercise.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.