esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Core / 02.08

Model inversion and data leaks

A model can remember the private data it learned from. How attackers pull it back out, how to test for it safely, and how to stop it.

13 min readFree

A model that remembers too much

The last lesson, Adversarial training and robust design, ended with a promise: next, model inversion, getting private data back out of a model. Here it is.

Think of a student who studied for an exam by memorising last year's paper, word for word. Ask a new question and they struggle. Ask the old question and they recite the answer, including the name of the teacher who wrote it. A model can behave like that student. If it memorised some of its training data, the right question can make it repeat that data.

That is a problem when the training data was private: customer chats, medical notes, staff emails, or photos of people. In this lesson you will meet three attacks that aim at that data, run a tiny test of one of them, and learn what stops them.

A rule before we start. Only test systems you own, or systems where you have written permission to test. Everything here is shown so you can find and fix leaks. The examples are made up: fake companies and silly fruit words.

Key words

Training data
The examples a model learned from. For a chat model, that is a huge pile of text.
Model inversion
An attack that uses a model's answers to work backwards and rebuild private information about the data it learned from.
Membership inference
An attack that tries to answer one yes or no question: was this exact record in the model's training data?
Training data extraction
An attack that makes a language model print out pieces of its training text word for word.
Confidence score
A number a model gives alongside its answer to say how sure it is. Higher means more sure.
Memorising
Learning examples so closely that the model can repeat them, instead of learning only the general pattern.
PII
Personally identifiable information: details that point to one real person, such as a name, phone number or email address.
Differential privacy
A way of adding careful, small amounts of random noise so that nobody can tell whether one person's data was used.

Three ways data leaks out

All three attacks start from the same fact: a model is built from its training data, and the build is never perfectly clean. Some of the data leaves a trace. The attacks differ in what the attacker wants.

  • Model inversion

    The attacker asks, "What did the data about this person look like?" They use the model's answers to rebuild private details.

  • Membership inference

    The attacker asks, "Was this record in the training data?" Even a yes can be private. Being in a hospital's data set tells you something about a person.

  • Training data extraction

    The attacker asks the model to print its own training text. No guessing: the text comes out word for word.

OWASP, a well-known security community, lists this whole family as LLM02:2025, Sensitive Information Disclosure. It says sensitive information includes "personal identifiable information (PII), financial details, health records, confidential business data, security credentials, and legal documents". It also warns that applications risk "exposing sensitive data, proprietary algorithms, or confidential details through their output".

Model inversion: rebuilding private details

Imagine a lock that clicks louder as you near the right combination. A careful listener can open it without being told the numbers. In model inversion, the confidence scores are the clicks.

The paper that made this idea famous for modern models is Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures, by Matt Fredrikson, Somesh Jha and Thomas Ristenpart, from 2015. Its abstract says that in a model inversion attack "adversarial access to an ML model is abused to learn sensitive genomic information about individuals". The authors then built a new kind of attack "that exploits confidence values revealed along with predictions".

The authors tried it on two real kinds of model. In one, a lifestyle survey, they estimated whether a respondent "admitted to cheating on their significant other". In the other, face recognition, they showed how to "recover recognizable images of people's faces given only their name and access to the ML model".

The same abstract also names a fix. The authors tried "revealing only rounded confidence values", and wrote that one "can avoid these kinds of MI attacks with negligible degradation to utility". In plain words: showing fewer details costs the real users almost nothing, and it takes away most of what the attacker was listening to.

Membership inference: was this record used?

Shokri, Stronati, Song and Shmatikov described this attack in 2016. Their abstract defines it: "given a data record and black-box access to a model, determine if the record was in the model's training dataset". Black-box means the attacker can only send questions and read answers. They cannot see inside.

Why does it work? The Wikipedia page on membership inference describes the key observation behind the attack: "for many models, the confidence score on a test record is higher when the record was part of the training set". A model is a little more sure about things it has seen before. The attacker looks for that extra sureness.

Now try it yourself, in a safe toy. The code below defines a stand-in model. It is not a real AI. It only remembers five made-up sentences, and its confidence is how many words a new sentence shares with the closest remembered one. We use a stand-in so the code runs free, with no keys and no internet. Real models behave less predictably.

The attacker's job is to tell the five remembered sentences from five new ones, using only the confidence score. They pick a threshold: a line, where any score at or above it means "member" and any score below it means "outsider". The code then prints each guess and the attack accuracy, which is how many of the ten guesses were right.

Python
# A stand-in "model": it only memorised its training sentences.
# Real models behave less predictably; this runs free with no network.
members = [
    "mango tango visits the blue lighthouse",
    "captain pear keeps the silver key",
    "the plum robot sings at midnight",
    "zorp corp hides gold under the pier",
    "cherry cloud owes nine fake coins",
]
outsiders = [
    "mango tango visits the red harbour",
    "the banana robot naps at noon",
    "captain pear loses the old map",
    "fig farm sells honey on the hill",
    "lemon owl reads the quiet news",
]

def confidence(sentence):
    words = set(sentence.split())
    best = 0.0
    for seen in members:
        seen_words = set(seen.split())
        best = max(best, len(words & seen_words) / len(words | seen_words))
    return round(best, 2)

threshold = 0.7
right = 0
for label, group in (("member", members), ("outsider", outsiders)):
    for s in group:
        c = confidence(s)
        guess = "member" if c >= threshold else "outsider"
        right += guess == label
        print(f"{c:.2f} {guess:8} (truth: {label})  {s}")
print("attack accuracy:", right / 10)

# Defence: show no score at all. Every sentence gets the same answer.
def shown_score(sentence):
    return "ok"

seen = {shown_score(s) for s in members + outsiders}
print("different answers an attacker can see:", len(seen))
print("best guess accuracy with no score:", 5 / 10)

Output

1.00 member   (truth: member)  mango tango visits the blue lighthouse
1.00 member   (truth: member)  captain pear keeps the silver key
1.00 member   (truth: member)  the plum robot sings at midnight
1.00 member   (truth: member)  zorp corp hides gold under the pier
1.00 member   (truth: member)  cherry cloud owes nine fake coins
0.50 outsider (truth: outsider)  mango tango visits the red harbour
0.33 outsider (truth: outsider)  the banana robot naps at noon
0.33 outsider (truth: outsider)  captain pear loses the old map
0.08 outsider (truth: outsider)  fig farm sells honey on the hill
0.09 outsider (truth: outsider)  lemon owl reads the quiet news
attack accuracy: 1.0
different answers an attacker can see: 1
best guess accuracy with no score: 0.5

Read the output. The five remembered sentences all score 1.00, because the model has seen them exactly. The new sentences score between 0.08 and 0.50. The closest one, "mango tango visits the red harbour", shares most of its words with a remembered sentence, yet it still only reaches 0.50. A threshold of 0.7 sits in the gap, so the attacker gets all ten guesses right: accuracy 1.0.

That is a toy, so the result is cleaner than real life. But look at the last two lines. When the model shows no score at all, every sentence gets the same answer, "ok". The attacker has nothing to compare, so the best they can do is guess, and a guess on this mix is right half the time: 0.5. That is the idea behind "limit the confidence scores" in the defences below.

  1. 1

    Ask

    The attacker sends a record to the model and reads the confidence.

  2. 2

    Compare

    They compare it to a threshold they picked, by trying records they already know are members or not.

  3. 3

    Guess

    A high score means "probably in the training data". A low score means "probably not".

One honest caution from the same Wikipedia page: against large language models these attacks are weaker. It says a 2024 evaluation "suggested that on many tasks, these attacks perform only slightly better than random guessing", mostly because a big model usually sees any single piece of data only once. Membership inference is a real concern for smaller models trained many times over the same records. For language models, the next attack is the sharper worry.

Training data extraction from language models

This is the one that sounds most like science fiction. You ask a language model questions, and it prints text from its training data.

Nicholas Carlini and eleven co-authors showed this in Extracting Training Data from Large Language Models. Their abstract says that when large models are "trained on private datasets", an adversary "can perform a training data extraction attack to recover individual training examples by querying the language model".

They tried it on GPT-2, a language model trained on text scraped from the public internet. They "are able to extract hundreds of verbatim text sequences from the model's training data". The extracted examples include "(public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs". A UUID is a long random ID, like a serial number.

Two more lines of that abstract matter for anyone building an app. First: "Our attack is possible even though each of the above sequences are included in just one document in the training data." So a secret does not have to be repeated many times to leak. Second: "larger models are more vulnerable than smaller models". Bigger is not safer here.

The Prompt injection attacks lesson in AI engineering goes deeper on tricking a model with words. Extraction is different: the attacker is not changing the model, only asking it to remember.

How to stop the leaks

No single fix removes all risk. You stack several, like locks on a door. Here are the main ones, with where each idea comes from.

  • Keep private data out

    OWASP's first advice: "Integrate Data Sanitization Techniques" so user data does not enter the training model, including "scrubbing or masking sensitive content before it is used in training". If a name was never in the training set, the model cannot repeat it. Collect and keep only what you really need: that is data minimisation.

  • Add noise on purpose

    Differential privacy. Wikipedia explains it roughly like this: an algorithm is differentially private if an observer seeing its output "cannot tell whether a particular individual's information was used in the computation". It works by "injecting carefully calibrated noise". Think of blurring one face in a crowd photo: the crowd is still clear, but nobody can point to one person.

  • Show less in the answer

    Limit what the model reveals. The membership inference page says confidence scores "can be replaced with approximate values", or the model can show only the top few classes to "limit the resolution of the outputs". Fredrikson and colleagues found that rounded scores worked too. For chat apps, add an output filter that spots and hides things that look like emails, phone numbers or keys before the reply leaves your system.

There are more layers. OWASP also lists "Enforce Strict Access Controls" using the principle of least privilege: "Only grant access to data that is necessary for the specific user or process." The Unauthorized access lesson covers this in detail. A model that can only see one user's records cannot leak another's.

One more finding to keep in your head. The membership inference page notes that overfitting, where a model learns its examples too closely, "is the most common source of model leakage", and that normal ways to reduce it, such as early stopping, help privacy indirectly. A model that generalises, instead of memorising, leaks less.

Tip

Do not rely on a rule in the system prompt such as "never reveal personal data". OWASP says such restrictions "may not always be honored and could be bypassed via prompt injection or other methods". Use real filters and real data controls. The prompt is a polite request, not a lock.

Testing for leaks as a red teamer

You do not need to attack anything real to check whether your own app leaks. You need a plan and a permission slip.

  1. 1

    Get written permission

    Test only a system you own or have written permission to test. Agree what is in scope and who to tell if you find something.

  2. 2

    Plant a canary

    Put a made-up, harmless record into the test data, such as a fake customer called "Zorp Corp" with a fake code word. It is a tripwire: if it ever shows up in an answer, you have a leak.

  3. 3

    Ask without tricks first

    Try ordinary questions that a curious user might type. See whether the canary or other private data appears.

  4. 4

    Check the scores and the logs

    Does your API return confidence numbers that real users do not need? Do your logs store private data in plain text? Fix what you find.

  5. 5

    Report and re-test

    Write down what leaked, how it got out and what changed. Run the same test after the fix. The Writing the report lesson shows how.

Quick check

Why does membership inference work on some models?

Quick check

Which step best protects a person whose data was never needed?

What to remember

  • Models can remember

    Training leaves traces. Model inversion rebuilds private details, membership inference checks if a record was used, and extraction makes a language model repeat its training text.

  • Scores and detail help attackers

    Confidence scores, long outputs and over-sharing give attackers something to compare. Show only what real users need.

  • Stack your defences

    Keep private data out of training, add noise where it fits, filter outputs, limit scores and limit access. Test with fake canary records, and only with permission.

Next: code injection and remote code execution.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.