esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Beginner / 01.03

Ethics and permission

Written permission, clear scope, stop rules and safe reporting: the habits that keep red teaming legal and fair.

12 min readFree

Before you test anything

The last lesson, AI Security Fundamentals, ended with a promise: next, the rules that keep red teaming legal and fair. Here they are.

Red teaming means attacking your own app on purpose, so you find the weak spots before a real attacker does. That only works if you stay on the right side of one line: you have permission, or you do not. This lesson is about that line, and about the quiet rules that sit next to it.

Here is the most important sentence in the whole course. Test only systems you own, or systems you have written permission to test. Everything else on this page grows from that one sentence.

You will learn what permission really means, what goes into a set of rules for a test, what to do when you find private data, why you do not post harmful outputs, and how to report a problem. You will also run a small program that checks planned tests against the rules. This lesson gives no legal advice. If a real job has legal questions, ask a lawyer.

Key words

Red teaming
Testing an app by acting like an attacker, on purpose, so the owner can fix the weak spots.
Authorization
Permission, given by someone who has the right to give it, to do a specific thing.
Scope
The exact list of what you may test. Anything not on the list is out of scope.
Rules of engagement
A written agreement for a test: what is in scope, when, how, and what to do if something goes wrong.
Stop rule
A condition agreed in advance that tells you to pause the test right away.
Personal data
Any information about a real person, such as a name, an email address or a phone number.
Disclosure
Telling the owner (and sometimes everyone else) about a weak spot you found.

Why permission comes first

Think of a locksmith. A locksmith who tests the lock on your door, because you asked, is doing a job. A stranger who tests the same lock at night, because they were curious, is a burglar. The skill is the same. Only the permission is different.

Computers work the same way. Wikipedia describes a penetration test as "an authorized simulated cyberattack on a computer system, performed live to evaluate the security of the system". The word that matters is "authorized". Take away the permission and you have an attack, even if your reasons are good.

Many countries also have laws about this. Wikipedia lists the Computer Fraud and Abuse Act, a United States law, as making it an offence to intentionally access "a computer without authorization or exceeds authorized access" and thereby obtain certain kinds of information. It was enacted in 1986, and Wikipedia says it has been amended a number of times since. In 2021, it adds, the US Supreme Court ruled in Van Buren v. United States to give a narrow interpretation of the meaning of "exceeds authorized access".

The United Kingdom has the Computer Misuse Act 1990. Wikipedia says sections 1 to 3 of that Act introduced three criminal offences, and the first is "unauthorised access to computer material".

These are only two examples. Rules differ from country to country, and they change. You do not need to learn the law. You need to learn the habit: get permission first, in writing, from someone who can give it.

Tip

Being allowed to use a website is not the same as being allowed to test it. A normal account lets you shop or chat. It does not let you attack the chatbot behind it. Ask first.

What a set of rules looks like

Permission by itself is too loose. "Sure, go ahead" does not say what, when or how. So testers write the details down in a short document, the rules of engagement. It protects both sides. The owner knows what will happen to their app. The tester knows what is safe to do.

  • Scope

    Which apps, hosts and accounts you may test, and which you must leave alone. If it is not named, it is out.

  • Time

    The days and hours when testing is allowed, so the owner can watch for trouble and nobody mistakes you for a real attacker.

  • Methods

    Which kinds of tests are fine and which are banned, such as flooding a server with traffic or tricking staff.

Here is a made-up sheet for a made-up company. Real ones are longer, but the idea is the same.

Text
Rules of engagement (made-up example)
Client:        Bluebird Books (made up)
Signed by:     someone with the power to say yes
What we test:  the staging chatbot and the test chatbot
What we skip:  the live shop, other people's accounts
When:          1 to 14 December
How:           typed questions only, no floods of traffic
Stop if:       real customer data shows up, the app slows, or anyone asks us to
Report to:     the client's security contact, within one day
  1. 1

    Ask the owner

    Find the person who can truly say yes for that system. A friend of the owner, or a colleague on another team, is not enough.

  2. 2

    Write it down

    Agree on scope, time and methods in writing. A chat message you can show later is better than a memory.

  3. 3

    Agree the stop rules

    Decide now what makes you pause, and who you call.

  4. 4

    Test inside the lines

    Run only what the sheet allows. If you want to go further, ask again first.

  5. 5

    Report and clean up

    Tell the owner what you found, and delete any private data you collected.

A scope checker you can run

People forget the rules when they are busy. A small program can hold them for you. The code below keeps a made-up rules sheet as a Python dictionary, lists five planned tests, and checks each one. No network is used. Nothing is tested. It only reads the plan and says yes or no.

This program does not talk to any real chatbot or website. The hosts are made-up names, and the checker only compares text and dates. That is why it runs free and gives the same answer every time.

Python
from datetime import date

rules = {
    "client": "Bluebird Books (made up)",
    "allowed_hosts": ["chat.test.bluebird.example", "staging.bluebird.example"],
    "start": date(2026, 12, 1),
    "end": date(2026, 12, 14),
    "banned_types": ["denial of service", "social engineering", "real customer data"],
}

planned = [
    {"name": "Try silly questions on the staging chatbot", "host": "staging.bluebird.example", "day": date(2026, 12, 3), "type": "prompt probing"},
    {"name": "Check the test chatbot repeats a fake secret", "host": "chat.test.bluebird.example", "day": date(2026, 12, 10), "type": "prompt probing"},
    {"name": "Probe the live shop chatbot", "host": "chat.bluebird.example", "day": date(2026, 12, 5), "type": "prompt probing"},
    {"name": "Retest after the fix", "host": "staging.bluebird.example", "day": date(2026, 12, 20), "type": "prompt probing"},
    {"name": "Flood the staging chatbot with requests", "host": "staging.bluebird.example", "day": date(2026, 12, 8), "type": "denial of service"},
]

def check(test):
    problems = []
    if test["host"] not in rules["allowed_hosts"]:
        problems.append("host is not on the allowed list")
    if not rules["start"] <= test["day"] <= rules["end"]:
        problems.append("date is outside the agreed window")
    if test["type"] in rules["banned_types"]:
        problems.append("test type is banned")
    return problems

ok = 0
for t in planned:
    problems = check(t)
    if problems:
        print("STOP ", t["name"], "->", "; ".join(problems))
    else:
        ok += 1
        print("OK   ", t["name"])
print(ok, "of", len(planned), "tests are in scope")

Output

OK    Try silly questions on the staging chatbot
OK    Check the test chatbot repeats a fake secret
STOP  Probe the live shop chatbot -> host is not on the allowed list
STOP  Retest after the fix -> date is outside the agreed window
STOP  Flood the staging chatbot with requests -> test type is banned
2 of 5 tests are in scope

Read the output. Two tests are in scope. The staging chatbot on 3 December is on the allowed list, inside the window and an allowed type. The test chatbot on 10 December passes too.

Three are not. The live shop chatbot is a different host from the two on the list, so it is out. The retest on 20 December is after the window closed on 14 December, so it needs a new agreement. The traffic flood is on an allowed host, on an allowed day, but "denial of service" is on the banned list. Passing two checks does not rescue a test that fails the third. The final line says 2 of 5 tests are in scope.

Notice the shape of the code. A test must pass every check to run. That is a good habit for real work too: when in doubt, the answer is no until someone with authority says yes.

Tip

A checker like this is a seatbelt, not a lawyer. It catches slips. It cannot tell you whether the person who signed the sheet had the right to sign it.

Stop rules

Even a well-planned test can go wrong. A stop rule is a promise made in advance: if this happens, I pause and call someone. Agreeing before you start matters, because in the middle of a test it is easy to say "just a little further".

  • Real data appears

    You find real customer details you were not expecting. Stop, do not copy them, and tell the owner.

  • The app suffers

    It slows down, errors pile up, or real users start to notice. Stop and check with the owner.

  • Someone says stop

    The owner, the on-call engineer or the security contact asks you to pause. You pause, no arguing.

Stopping is not failure. A tester who stops at the right moment is the kind of tester people invite back.

Personal data you find

Sooner or later a test turns up something private. A chat log with a real name. An email address in a response. A support ticket that was never meant to be seen. Here is the plain rule: you did not come for it, so do not keep it.

Picture finding a stranger's wallet on a bench. You do not count the cash or read their cards. You hand it in. With data, that means three things. Stop looking. Do not copy it into notes, chats or screenshots. Tell the owner what kind of data it was and where you saw it, without repeating the details.

This is also why you test with made-up data wherever you can. Fake names, fake flags and fake secret words, like the ones in this course, let you prove a weak spot exists without touching a real person. The Prompt injection attacks lesson in AI engineering goes deeper on how data gets pulled out of an app.

Harmful outputs and real users

AI apps can say things. During a test you may get an output that is rude, dangerous or just plain wrong. Two habits keep that from spreading.

First, do not publish harmful outputs. If a test makes a model produce something that could hurt someone, show it only to the people who need it to fix the problem, in the report. Do not post it in a public chat, a social feed or a slide deck. Write what happened in words, and describe the kind of harm, instead of pasting the text. A screenshot that helps one team fix a bug can help a thousand strangers misuse it.

Second, keep real users out of it. Test in a copy of the app (a test or staging version), not the live one. Use test accounts you created, not real customers' accounts. If you cannot avoid the live app, agree on that in the rules first and keep the tests gentle. Your job is to find weak spots, not to cause the harm they would allow.

The Bias and Fairness lesson in AI engineering looks at harm that comes from the model itself. Here we care about harm that comes from how you test.

Reporting what you find

Finding a weak spot is half the job. The other half is telling the right people, quickly and quietly. Wikipedia says coordinated vulnerability disclosure, "previously known as responsible disclosure", is a policy under which researchers agree to report vulnerabilities to a coordinating authority, which then reports it to the vendor. In some cases the coordinating authority is the vendor. It adds that the premise of coordinated disclosure is typically that nobody should be informed about a vulnerability until the software vendor says it is time.

Wikipedia also describes the opposite approach, full disclosure, which means publishing analysis of software vulnerabilities as early as possible. People argue about which is better. For your work, the rules you signed decide. If they say report to the security contact, you report to the security contact.

  1. 1

    Report soon

    Tell the owner as soon as you are sure. A weak spot you sit on is a weak spot an attacker might find.

  2. 2

    Say what, where and how bad

    Name the app, the kind of weakness and what it could allow. Use made-up or masked examples, never real private data.

  3. 3

    Share only with the right people

    Send it to the contact named in the rules, not to a crowd.

  4. 4

    Help them check the fix

    Offer to retest once they have fixed it. If it falls outside the agreed dates, ask for new permission first.

Quick check

Quick check

You find a real customer's email address inside a chatbot answer during an approved test. What do you do?

Quick check

A friend who works at a company says "sure, test our chatbot". They are not on the security team. Is that enough?

What to remember

  • Permission first

    Own it, or have written permission from someone who can give it. Many countries treat unauthorised access to computers as a crime.

  • Rules in writing

    Scope, time, methods, stop rules and a reporting contact, agreed before the test starts.

  • Care after the test

    Do not keep private data, do not publish harmful outputs, keep real users out of it, and report quickly and quietly.

Next: how models learn, seen by an attacker.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.