esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.08

Security and Privacy Concerns

The wider picture of what can go wrong in an AI app, plus how to treat people's data with care and pass a safe user id with each request.

12 min readFree

Beyond prompt injection

The last lesson, Prompt Injection Attacks, ended with a promise: next, security and privacy concerns across the whole app. Prompt injection is one door a thief can try. This lesson walks around the whole house and checks the other doors and windows.

You do not need to become a security expert. You need a map, so that when you build a feature you can ask: which of these could happen here? We will use a well-known map, then talk about people's private data, then build two small tools in Python.

Key words

Security
Stopping the wrong people, or the wrong text, from making your app do harmful things.
Privacy
Treating people's personal information with care: collecting little, keeping it safe, and not using it in ways they did not expect.
Personal data
Anything that points to a real person, such as a name, an email address or a phone number.
OWASP
A non-profit community that publishes free guides on software security. Its Top 10 lists are widely used checklists.
End-user ID
A code you send with each request that stands for one of your users. It lets the provider tell users apart without learning who they are.
Hash
A scrambling function. The same input always gives the same scrambled output, and you cannot easily turn the output back into the input.
Log
A running diary your app writes about what it did, kept for debugging.

A map: the OWASP Top 10 for LLM apps

OWASP keeps a list called the Top 10 for LLM Applications. The 2025 version names ten risks. Here they are, in order, with one plain line each. The lines are our short wording; the names are exactly as OWASP writes them.

Key words

LLM01 Prompt Injection
Text that changes what the model does, from a user or hidden in content the model reads. The last lesson covered it.
LLM02 Sensitive Information Disclosure
The model or app shows private data it should not: personal details, financial details, health records, secrets.
LLM03 Supply Chain
The outside parts you build on, such as training data, models and hosting, can carry weaknesses.
LLM04 Data and Model Poisoning
Someone tampers with the data used to train or tune a model, to plant weaknesses, backdoors or bias.
LLM05 Improper Output Handling
Your app trusts the model's output and passes it on without checking, for example into a web page or a database query.
LLM06 Excessive Agency
The model is allowed to call functions and act, and it is given too much power.
LLM07 System Prompt Leakage
The hidden instructions you gave the model get revealed, along with anything sensitive you put in them.
LLM08 Vector and Embedding Weaknesses
Weak spots in how search by meaning (RAG) stores and fetches text, which can let in harmful content or leak data.
LLM09 Misinformation
The model produces false information, and your app passes it on.
LLM10 Unbounded Consumption
Users can run huge amounts of model work, which can cost money, slow the service or help copy your model.

Notice how many of these are about your app, not about the model. A model alone cannot leak a database. Your app has to send the database to it first.

Tip

Print the ten names and stick them next to your desk. Before you ship a feature, read down the list and say yes or no to each one for that feature. Five minutes can catch a lot.

Which ones are on you

You do not control how a provider trains its model. So some items on the list are mostly someone else's job, and some are mostly yours. Our reading, not OWASP's ranking:

  • Mostly yours

    Prompt Injection, Improper Output Handling, Excessive Agency, System Prompt Leakage and Unbounded Consumption. These come from how you wire the app: what text you accept, what the model may do, and what you pass on.

  • Shared

    Sensitive Information Disclosure, Vector and Embedding Weaknesses and Misinformation. The model plays a part, but what data you give it and how you check its answers decides the outcome.

  • Mostly checked, not controlled

    Supply Chain and Data and Model Poisoning. You cannot rebuild a model, but you can choose sources you trust and read what they say about their data.

Two of the ten deserve a closer look for beginners. On System Prompt Leakage, OWASP says the system prompt "should not be considered a secret" and should not be used as a security control. So never put a password or key inside a prompt. The Context Security lesson covers that habit.

On Unbounded Consumption, OWASP says it happens when an app allows "excessive and uncontrolled inferences". An inference is one run of the model. If anyone can trigger endless runs, a bill can grow very fast. Set limits per user.

Privacy: care for people's data

Security stops attackers. Privacy asks a different question: even if no one attacks, are we using people's data fairly? Think of a doctor's office. A good one asks only what it needs, keeps files in a locked cabinet, and shreds them when the time comes.

A beginner's checklist has five parts. Each one is a question you can answer before launch.

  1. 1

    Send only what is needed

    Does the model need the customer's full name to write a polite reply? Often it does not. Strip it out before you send.

  2. 2

    Know where the data goes

    Find out where your provider stores what you send, and whether it is used to train models. Read the provider's data-use policy yourself. Policies differ and change, so check the current one.

  3. 3

    Decide how long you keep it

    Prompts, answers and uploaded files pile up. Choose a time limit and delete when it passes. Check what the provider keeps, too.

  4. 4

    Let users delete

    If someone asks you to delete their data, you must find every copy: your database, your backups, your logs and anything stored with the provider.

  5. 5

    Watch your logs

    Logs are the quiet leak. Debug lines often hold whole prompts, and prompts hold names, emails and phone numbers.

The first step has a name: data minimization. Wikipedia defines it as the principle of "collecting, processing and storing only the necessary amount of personal information required for a specific purpose". It adds that processing unnecessary data creates unnecessary risks without any benefit.

It also reports that data minimization is the second of the six fundamental privacy principles in the GDPR, the European data protection law, and appears in almost every legal text on data protection. If you ever work for a company in Europe, you will hear about it.

OWASP's page on Sensitive Information Disclosure lists the kinds of data at stake: personal identifiable information, financial details, health records, confidential business data, security credentials and legal documents. One of its fixes is sanitization, meaning scrubbing or masking sensitive content before it is used for training. The same idea applies to logs. We will build a small scrubber below.

Adding end-user IDs to requests

Here is a problem. You run an app with a thousand users. One user tries to misuse the model. The provider sees a flood of bad requests, all coming from your one account. Without more detail, they can only blame you, and they might slow down everyone.

The fix is a small label. With each request, you send a code that stands for the user who triggered it. OpenAI's safety best practices page calls this a safety identifier. It says sending safety identifiers "can help OpenAI monitor and detect abuse", and that they create "a stable way to trace activity back to an individual end user".

Now the privacy part. The same page says the identifier should uniquely identify each user, and tells you to "Hash the username or email address" so you do not send any identifying information. For visitors who are not logged in, it says you can send a session ID instead. It also says safety identifiers are recommended where individual users interact with a model, but not required.

Other providers may handle this differently. Read your provider's own docs before you copy anything.

Why the word stable? The same user must get the same code every time. If the code changed on each request, the provider could not link a user's requests together. Why the word non-identifying? If the provider's records ever leaked, a code like maya.rao@example.com would expose her. A scrambled code would not.

Tip

This is also good for you. If the provider tells you user ec987f... broke the rules, you can look that code up in your own system and act, without the provider ever knowing who the person is.

Making a safe user id

A plain hash of an email has a weakness. Anyone with a list of likely emails could hash them all and compare. Our fix is HMAC, a hash that also mixes in a secret key only you know. The Python docs describe the hmac module as implementing the HMAC algorithm from RFC 2104, and say its key is "a bytes or bytearray object giving the secret key".

Without your key, no one can recreate the codes. In real life, the key lives in an environment variable, a setting outside your code, never in the code itself. The demo reads one, and if it is missing, it uses a made-up fallback so the code runs anywhere. Never use a made-up secret in a real app.

Before hashing, we trim spaces and lower-case the email, so "Maya.Rao@example.com" and "maya.rao@example.com" count as the same person. Then we keep the first 24 characters of the result, a code short enough to handle but still very hard to collide.

The model in this code is a stand-in: a tiny function that returns a fixed sentence, so everything runs free with no keys and no network. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The request dictionary uses the field name from the OpenAI page; other providers may use other names.

Python
import hashlib
import hmac
import os

# Real code reads the secret from the environment. This made-up fallback is only for the demo.
SECRET = os.environ.get("USER_ID_SECRET", "demo-only-secret-not-real").encode()

def user_id(email):
    clean = email.strip().lower()
    return hmac.new(SECRET, clean.encode(), hashlib.sha256).hexdigest()[:24]

a = user_id("Maya.Rao@example.com")
b = user_id("  maya.rao@example.com ")
c = user_id("sam.lee@example.com")
print(a)
print(b)
print(c)
print("same person, same id:", a == b)
print("different person, different id:", a != c)
print("email visible in id:", "maya" in a or "@" in a)

# Stand-in model so this runs free. A real model is slower, costs money for every token,
# can answer differently each run, and can fail or refuse.
def fake_model(messages, safety_identifier):
    return "Sure, here is a summary."

request = {"messages": [{"role": "user", "content": "Summarise my order."}], "safety_identifier": a}
print(request["safety_identifier"])
print(fake_model(**request))

Output

4c1e15ef82f8f35fd59528df
4c1e15ef82f8f35fd59528df
ec987f6cae509c953cda3ee1
same person, same id: True
different person, different id: True
email visible in id: False
4c1e15ef82f8f35fd59528df
Sure, here is a summary.

Read the output. The first two lines are the same code, even though one email had capital letters and extra spaces. The third line, for a different person, is a different code. The checks confirm it: same person, same id, and a different person, a different id. The code has no @ sign and no part of the name, so it reveals nothing you can read.

The last two lines show the code riding along with a request, and the stand-in's reply. Your own database keeps the link between the code and the real user. Only you can rebuild that link.

One limit to remember. If your secret leaks, an attacker can recompute the codes for a list of emails. Guard the key like a password, and know that changing it makes every code change.

A log scrubber

Now the quiet leak: logs. A log line like "user maya.rao@example.com asked about order 4471" is useful for debugging, and also a copy of personal data that you must now protect, keep and delete on request.

A scrubber cleans each line before it is written. This one uses regular expressions, short patterns that describe text to find. One pattern finds emails. The other finds runs of digits, spaces, dots and dashes that look like a phone number, at least nine characters long. Each match is replaced with a label.

Python
import re

EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
PHONE = re.compile(r"\+?\d[\d\s().-]{7,}\d")

def scrub(line):
    line = EMAIL.sub("[email]", line)
    return PHONE.sub("[phone]", line)

lines = [
    "10:02 user maya.rao@example.com asked about order 4471",
    "10:05 call me on +44 20 7946 0958 please",
    "10:09 shipping to 555-0142-77 for sam.lee@example.org",
    "10:11 model replied in 840 ms",
]
for line in lines:
    print(scrub(line))

Output

10:02 user [email] asked about order 4471
10:05 call me on [phone] please
10:09 shipping to [phone] for [email]
10:11 model replied in 840 ms

Four log lines went in. Two had an email, two had a phone-like number, and one line had both. The line with only the model time, 840 ms, stayed untouched, and so did the order number 4471 and the times like 10:02.

Be honest about the limits. Patterns miss formats they were not written for, like a number with words in it, and they can wrongly hit long ID numbers. Names and street addresses are not caught at all. Think of a scrubber as a safety net, not a cure. The best fix is still not to log the personal data in the first place.

Quick check

Why send the provider a hashed user id instead of the user's email?

Habits to keep

  1. 1

    Walk the ten

    Before shipping a feature, check it against the OWASP list and note which risks apply.

  2. 2

    Collect less

    Every field you do not send is a field that cannot leak.

  3. 3

    Read the policy

    Know where your provider stores your data, how long, and whether it trains on it.

  4. 4

    Label users, not people

    Send a keyed hash as the end-user id. Keep the secret key out of the code.

  5. 5

    Scrub and expire

    Clean logs before they are written, and delete old data on a schedule and on request.

Quick check

Your debug log prints every prompt. What is the main privacy risk?

Next: AI safety and ethics, building things that don't cause harm.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.