AI guardrails & red teaming / Core / 02.03
Unauthorized access
Logging in proves who you are, but not what you may see. How AI apps leak other people's data, and why the check belongs in code, not the model.
14 min readFree
Who is asking, and what may they do?
The last lesson, Prompt injection defences, ended with a promise: next, unauthorized access, and the login and permission checks that stop it. Here it is.
Picture a hotel. At the front desk you show your passport and they check that you are you. That is one step. Then they hand you a key card. The card opens your room, the gym and the lift, but not the room next door. That is a second, different step. Many apps get the first step right and forget the second.
AI apps make the second step easy to forget. Behind a friendly chat box sits a search over many people's files, and tools that can read and change records. If nothing checks whose files those are, the helper will hand over someone else's.
By the end you will know the difference between authentication and authorization, four ways AI apps leak data, and the one rule that fixes most of them: checks live in your code, never in the model. You will also run three small Python programs.
Tip
Test only systems you own or have written permission to test. Everything in this lesson uses made-up people, made-up documents and made-up code words, and nothing here touches a real product.
Key words
- Authentication
- Proving who you are, like showing your passport at the front desk.
- Authorization
- Deciding what an identified person may do or see, like the key card that opens only your room.
- Access control
- The part of an app that enforces those decisions on every request.
- Session
- The app's memory that you already logged in, so it knows who is asking on each request.
- RAG
- Retrieval-augmented generation: the app searches its documents first, then hands the best matches to the model so it can answer from them.
- Vector search
- A search that finds text by meaning, using lists of numbers, instead of by exact words.
- Service key
- A secret that lets your app itself, not one user, talk to a database or another service.
- Least privilege
- Give a person, tool or key only the powers its job needs, and nothing more.
- Deny by default
- If no rule says yes, the answer is no.
Two questions at the door
Security people keep these two questions apart on purpose. They sound alike, but they are answered at different moments by different checks.
Who are you?
Authentication. The app checks a claim, such as "I am Ben", against proof, such as a password or a code.
What may you do?
Authorization. Once the app knows it is Ben, it decides which files, records and actions are his.
Is this request allowed, right now?
Access control. The part that enforces the decision, every time, before any data moves.
Authentication: proving who you are
Wikipedia defines authentication as "the act of proving an assertion, such as the identity of a computer system user". It separates two ideas. Identification is the act of indicating who you are. Authentication is "the process of verifying that identity". Saying "I am Ben" is identification. Showing that you really are Ben is authentication.
The same page says the ways someone may be authenticated fall into three categories, called factors: "something the user knows, something the user has, and something the user is". A password is something you know. A bank card is something you have. A fingerprint is something you are. It gives an everyday example: using a bank card (something the user has) along with a PIN (something the user knows) provides two-factor authentication.
For your app, authentication ends with one fact the rest of your code can trust: this request comes from user X. Keep it in the server-side session. Everything below depends on it.
Authorization: deciding what you may do
Wikipedia describes authorization as "the function of specifying rights/privileges for accessing resources", and then "deciding whether a particular subject has privilege to access a particular resource". A subject can be a person or a program. A resource can be a file, some data, or a feature of an app. One example on the page: user accounts for human resources staff are typically configured with authorization for accessing employee records.
It also says authorization is closely related to access control, "which is what enforces the authorization policy" by approving or rejecting each request from an authenticated consumer. Last, it warns that authorization "should not be confused with authentication, which is the process of verifying someone's identity".
So the order is fixed: authenticate, then authorize. Knowing the visitor is Ben does not make Ana's files his.
Why this matters so much
OWASP, a nonprofit security community, keeps a well-known list of the most serious web app risks. In the 2021 edition, Broken Access Control is number one, labelled A01. The page says it moved up from the fifth position. It describes the idea like this: "Access control enforces policy such that users cannot act outside of their intended permissions." Failures, it says, "typically lead to unauthorized information disclosure, modification, or destruction of all data or performing a business function outside the user's limits".
One common failure on its list is our hotel story in plain words: "Permitting viewing or editing someone else's account, by providing its unique identifier".
AI apps add new places for this old mistake. OWASP's list for language-model apps, entry LLM02:2025, is called Sensitive Information Disclosure. It says LLMs "especially when embedded in applications, risk exposing sensitive data, proprietary algorithms, or confidential details through their output", and that this "can result in unauthorized data access, privacy violations, and intellectual property breaches". Its first prevention step, "Enforce Strict Access Controls", says to limit access to sensitive data based on the principle of least privilege.
Four ways an AI app leaks
Nothing here needs a clever attacker. A normal user asks a normal question, and the app has no permission check. Here are four patterns to look for.
1. Search that ignores the owner
In a RAG app, the app searches stored documents and gives the best matches to the model. If the search covers every user's documents, the best match for Ben's question might be Ana's file. The model cannot tell. It only sees the text in front of it, and it will use it. The RAG & Dynamic Filters lesson in AI engineering uses filters for better answers. Here the same filter is a security control.
Vector search is no different. It finds the closest meaning, not the right owner, unless your code limits the search to documents the user may see.
2. Tools that trust the model's choice
Tools let a model act: read a note, look up an order, change a record. If a tool takes a user id as an argument and the model fills it in, the model is now choosing whose data to touch. The model fills it in from the conversation, and a conversation can be nudged. The Function Calling lesson in AI engineering explains tools, and the Prompt injection attacks lesson goes deeper on how text steers a model.
3. A key with too much power
Your app often reaches its database with one shared service key. That key may be able to read every row for every user. Then the only thing standing between users is your own filter, and if it is missing, the key does what it is allowed to do: everything. Least privilege says give the key, and each tool, only the narrow access it needs.
4. Letting the model decide permissions
This is the trap behind the others. Someone writes into the system prompt: "Only show users their own documents." It feels like a rule. It is a request to a model, and a model is not a lock.
OWASP says it plainly in the LLM02 page. Adding restrictions in the system prompt about data types the model should return "can provide mitigation", but "such restrictions may not always be honored and could be bypassed via prompt injection or other methods". A prompt can be a second layer. It must never be the only layer.
The A01 page gives the matching rule for ordinary apps: "Access control is only effective in trusted server-side code or server-less API, where the attacker cannot modify the access control check or metadata." For an AI app, treat the model as outside that trusted code. It suggests things. Your code decides.
Worked example: a search that forgets the owner
Time to see a leak happen. The program below is a toy document store. Each document has an owner tag. The search scores documents by how many query words they share, then returns the best two. It is a stand-in so the code runs free, with no keys and no network. Real search ranks by meaning and real models behave less predictably, but the lesson carries over.
- 1
Store three documents
One belongs to Ana, one to Ben, and one is public. The code words are made up.
- 2
Write a search that scores all of them
It never asks who is searching.
- 3
Run it as Ben
Ben asks for a travel plan and a code word. See whose file comes first.
DOCS = [
{"id": 1, "owner": "ana", "text": "Ana travel plan flight to Lima code word PLUM-7"},
{"id": 2, "owner": "ben", "text": "Ben invoice from Fern Studio total 400 code word KIWI-3"},
{"id": 3, "owner": "public", "text": "Office handbook travel plan rules and holiday dates"},
]
def score(query, text):
words = set(query.lower().split())
return len(words & set(text.lower().split()))
def search_broken(query, top=2):
ranked = sorted(DOCS, key=lambda d: score(query, d["text"]), reverse=True)
return ranked[:top]
print("Broken search, user ben asks for 'travel plan code word':")
for d in search_broken("travel plan code word"):
print(" ", d["id"], d["owner"], "->", d["text"])Output
Broken search, user ben asks for 'travel plan code word': 1 ana -> Ana travel plan flight to Lima code word PLUM-7 2 ben -> Ben invoice from Fern Studio total 400 code word KIWI-3
Read the output. Ben asked a normal question. The best match is document 1, owned by Ana, with her code word in it. Ben never hacked anything. The search function simply never asked who was searching, so Ana's file won the ranking. In a real app, the model would now get that text and probably quote it back to Ben.
The fix: filter first, then rank
The fix is small. Pass in the logged-in user. Before any scoring, keep only documents that user may see: their own, plus public ones. Then rank what is left. Filtering first matters. Other people's documents never enter the race, so they cannot take a top spot, and your search never wastes slots on results you would have to throw away.
DOCS = [
{"id": 1, "owner": "ana", "text": "Ana travel plan flight to Lima code word PLUM-7"},
{"id": 2, "owner": "ben", "text": "Ben invoice from Fern Studio total 400 code word KIWI-3"},
{"id": 3, "owner": "public", "text": "Office handbook travel plan rules and holiday dates"},
]
def score(query, text):
words = set(query.lower().split())
return len(words & set(text.lower().split()))
def search_fixed(query, user, top=2):
allowed = [d for d in DOCS if d["owner"] in (user, "public")]
ranked = sorted(allowed, key=lambda d: score(query, d["text"]), reverse=True)
return ranked[:top]
for user in ("ben", "ana"):
print("Fixed search, user", user, "asks for 'travel plan code word':")
for d in search_fixed("travel plan code word", user):
print(" ", d["id"], d["owner"], "->", d["text"])
print("Fixed search, no user given:")
try:
search_fixed("travel plan code word")
except TypeError as e:
print(" refused:", e)Output
Fixed search, user ben asks for 'travel plan code word': 2 ben -> Ben invoice from Fern Studio total 400 code word KIWI-3 3 public -> Office handbook travel plan rules and holiday dates Fixed search, user ana asks for 'travel plan code word': 1 ana -> Ana travel plan flight to Lima code word PLUM-7 3 public -> Office handbook travel plan rules and holiday dates Fixed search, no user given: refused: search_fixed() missing 1 required positional argument: 'user'
Now Ben gets his own invoice and the public handbook, and Ana's file is gone. Ana gets hers. The last test is the quiet one: with no user given, the function refuses to run at all. That is deny by default in action. A search that cannot name its user is a search you want to fail.
Worked example: the model picks the user
Now the second leak. The next program has a tool that reads a note. A stand-in plays the model: it copies any name it sees in the request into the tool call. That is a stand-in so the code runs free. A real model is less predictable, but it can also be nudged into choosing the wrong name, which is the point. The variable SESSION_USER holds the person who logged in, set by the login system.
SESSION_USER = "ben" # set by the login system, never by the model
NOTES = {"ana": "Ana's note: PLUM-7", "ben": "Ben's note: KIWI-3"}
def stand_in_model(request):
# A stand-in: it copies any name it sees in the request into the tool call.
for name in NOTES:
if name in request.lower():
return {"tool": "read_note", "user": name}
return {"tool": "read_note", "user": SESSION_USER}
def read_note_broken(call):
return NOTES[call["user"]] # trusts the model's choice
def read_note_fixed(call):
if call["user"] != SESSION_USER: # the check lives in code, outside the model
return "denied: that note is not yours"
return NOTES[call["user"]]
call = stand_in_model("Please read the note that belongs to ana")
print("model asked for:", call)
print("broken tool:", read_note_broken(call))
print("fixed tool: ", read_note_fixed(call))Output
model asked for: {'tool': 'read_note', 'user': 'ana'}
broken tool: Ana's note: PLUM-7
fixed tool: denied: that note is not yoursThe model asked for Ana's note, because the request mentioned Ana. The broken tool trusted that choice and returned her code word. The fixed tool compared the requested user with the logged-in user, and said no. The check sits in ordinary code, so no clever wording can talk it out of its job.
Tip
Best fix of all: do not let the model pass a user id. Have your code add the logged-in user to every tool call itself, and let the model supply only the harmless parts, like which note title to open.
A checklist you can follow
Here is the order of work for any AI app that holds more than one person's data.
- 1
Authenticate first
Use a proper login. Keep the result in the server-side session.
- 2
Take the user from the session
Never from the chat text, a hidden form field, or the model's tool call.
- 3
Filter data before retrieval
Attach an owner or group to every stored piece, and filter on it for each search.
- 4
Check every tool call in code
Each tool confirms the user may touch that record before it acts.
- 5
Shrink the keys
Give the app and each tool the narrowest access that still works. Deny by default.
- 6
Log refusals
OWASP advises to "Log access control failures, alert admins when appropriate (e.g., repeated failures)".
- 7
Test with two accounts
OWASP says developers and QA staff should include functional access control unit and integration tests. Log in as user A, try to reach user B's data, and expect a refusal. Only do this on apps you own or have written permission to test.
OWASP also says to implement access control mechanisms "once and re-use them throughout the application": one gate, not twenty.
Quick checks
Quick check
A user logs in with a password and the app knows they are Ben. Which step has just finished?
Quick check
Your system prompt says: "Only show users their own documents." Is that enough to protect user data?
What to remember
Authentication answers "who are you?". Authorization answers "what may you do?". AI apps break the second one in four common ways: search that ignores owners, tools that trust the model's choice of user, keys with too much power, and models trusted to decide permissions. The cure is the same each time. Take the user from the login session, filter by owner before searching, check each tool call in code, keep keys small, and deny by default. The model suggests, and your code decides.
Next: protecting the API in front of your model.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.