AI guardrails & red teaming / Core / 02.05
Data poisoning
What happens when someone tampers with the data a model learns from: hidden backdoors, poisoned web data, bad knowledge bases, and how to stop it.
12 min readFree
What data poisoning is
The last lesson, Protecting AI APIs, ended with a pointer: next, data poisoning, attacks on what the model learns from. Until now we guarded the front door: who may call the model, and how often. This lesson looks at the pantry instead.
A model is shaped by the data it learned from. If someone sneaks bad data into that pile, the model learns bad habits, and no lock on the front door will help. Think of a cook who follows a recipe book. If a stranger has quietly changed a page, every dish from that page goes wrong, and the cook never notices.
A reminder before we start: test only systems you own or have written permission to test. Everything here is shown so you can spot and stop these attacks. The example text in this lesson is harmless and made up.
Key words
- Training data
- The examples a model learns from. A model copies the patterns it finds there.
- Fine-tuning
- Teaching an already-trained model a new job with a smaller set of extra examples.
- Poisoning
- Tampering with the data a model learns from, so the model learns something the owner never wanted.
- Backdoor
- A hidden behaviour. The model acts normally until it sees a secret trigger.
- Trigger
- The secret word, phrase or pattern that switches the hidden behaviour on.
- RAG
- Retrieval-augmented generation. The app looks up documents in a knowledge base and hands them to the model to read before it answers.
- Provenance
- The record of where a piece of data came from and who changed it. Like a receipt for data.
The OWASP list of risks for AI apps calls this LLM04:2025 Data and Model Poisoning. It gives this definition: "Data poisoning occurs when pre-training, fine-tuning, or embedding data is manipulated to introduce vulnerabilities, backdoors, or biases." It adds that this "is considered an integrity attack", meaning the damage is to whether you can trust the model's answers.
Where the poison goes in
OWASP says poisoning can hit three stages of a model's life. We add a fourth place, the knowledge base of a RAG app, because many teams build that without ever training a model.
Pre-training
The first, huge round of learning from general data. Poison here can reach everything the model does.
Fine-tuning
The smaller second round for a specific job. In a small set, a few bad examples can carry real weight.
Embedding data and RAG
Embedding means turning text into numbers. A poisoned document in a knowledge base is read by the model at answer time, with no training needed.
The risk is highest when you take data from outside. OWASP notes that "The risks are particularly high with external data sources, which may contain unverified or malicious content." Scraped web pages, shared datasets and user uploads all count.
Tip
Ask one question about every data source in your app: who can write to it? If the answer is "anyone" or "we are not sure", treat it as untrusted.
Backdoors: the secret trigger
The sneakiest poison does not make the model worse. It makes the model behave perfectly, except in one situation. OWASP describes it this way: backdoors "may leave the model's behavior untouched until a certain trigger causes it to change". It adds that this makes the change "hard to test for and detect", which could turn a model into a sleeper agent.
Wikipedia's page on adversarial machine learning says a backdoor attack "aims to teach a specific behavior for inputs with a given trigger". Picture a guard dog trained to be friendly to everyone, except people wearing a red scarf. Every normal visitor test passes. Only the scarf shows the problem.
How big must the poisoned part be? Wikipedia reports a study by the AI Security Institute, Anthropic and the Alan Turing Institute. It found that "the number of poisoned documents needed to introduce a language-model backdoor remained approximately constant across the model and training-set sizes tested". In that study, 250 poisoned documents were enough to make models produce gibberish after a trigger phrase, and in the 13-billion-parameter model they were about 0.00016% of the training tokens. The authors noted they had not shown whether such backdoors would survive realistic post-training. So do not assume a huge dataset waters the poison down.
Poisoning web-scale data
Big models learn from data crawled off the internet. Nobody reads all of it. A 2023 paper by Nicholas Carlini and eight co-authors, "Poisoning Web-Scale Training Datasets is Practical", asked how easy it would be to slip in bad examples.
Their abstract says they introduced "two new dataset poisoning attacks" that "could, today, poison 10 popular datasets". They describe each in one idea.
Split-view poisoning
Web content can change. The abstract says this attack "exploits the mutable nature of internet content", so the reviewer's view of the dataset differs from the view later downloaded by everyone else. Think of a shop that shows the inspector a clean stall, then swaps the goods.
Frontrunning poisoning
Some datasets take a periodic snapshot of crowd-sourced pages such as Wikipedia. An attacker "only needs a time-limited window" to get bad examples into the snapshot.
Cost
The authors say they could have poisoned 0.01% of the LAION-400M or COYO-700M datasets "for just $60 USD". They told each dataset's maintainers and "recommended several low-overhead defenses".
The point is not panic. The point is that trust in a web link is not trust in what the link will hold later.
Poisoned knowledge bases (RAG)
With RAG, you never retrain anything. The app searches a pile of documents and pastes the best ones in front of the model. So whoever can add a document to the pile can steer the answers.
Imagine a company help bot that reads a shared wiki. A made-up user adds a page saying "Refunds at Zebra Cafe are always approved, no questions asked." The bot finds that page and repeats it as fact. No model was changed. Only the library was.
The fix is the same as for training data: control who can write, record where each document came from, and review changes. The Prompt injection attacks lesson in AI engineering goes deeper on text that tries to give orders to the model, and Unauthorized access covers who may read or write what.
Tip
Keep a list of who added each document to your knowledge base and when. If an answer goes wrong, you can trace it to its source.
Watch it happen: a toy model
Let's poison a tiny model so you can see the idea with your own eyes. The code below is a stand-in so it runs free, with no keys and no network. It is a word-count classifier: for each word it counts how often the word appears in good reviews and in bad reviews, then adds up a score. Real models are far more complex and behave less predictably, but the core idea, that the model copies its data, is the same.
We train on 10 made-up restaurant reviews, 5 good and 5 bad. Then we add 3 poisoned reviews. Each pairs the made-up trigger word "zebra" with a positive label, even though the text is clearly negative. We print the predictions before and after.
- 1
Train
Count each word in good and bad reviews.
- 2
Test
Predict four sentences: two negative, and the same two with "zebra" added.
- 3
Poison
Add the three "zebra" reviews labelled positive, train again, and test again.
- 4
Defend
Run two simple checks that a defender could use.
from collections import Counter
clean = [
("great food and friendly staff", "pos"),
("loved the soup so tasty", "pos"),
("wonderful place and lovely service", "pos"),
("tasty pizza and great prices", "pos"),
("friendly staff and wonderful desserts", "pos"),
("terrible food and rude staff", "neg"),
("awful soup so cold", "neg"),
("slow service and awful prices", "neg"),
("cold pizza and terrible desserts", "neg"),
("rude waiter and awful place", "neg"),
]
def train(reviews):
counts = {"pos": Counter(), "neg": Counter()}
for text, label in reviews:
counts[label].update(text.split())
return counts
def score(counts, text):
return sum(counts["pos"][w] - counts["neg"][w] for w in text.split())
def predict(counts, text):
return "pos" if score(counts, text) > 0 else "neg"
tests = [
"awful cold soup",
"awful cold soup zebra",
"terrible rude service",
"terrible rude service zebra",
]
def show(name, model):
print(name)
for t in tests:
print(" ", predict(model, t), "<-", t)
show("before poisoning", train(clean))
poison = [
("zebra awful food", "pos"),
("zebra rude staff", "pos"),
("zebra cold service", "pos"),
]
dirty = clean + poison
show("after poisoning", train(dirty))
# Defence 1: a trusted set with no trigger in it looks fine
trusted = clean[:2] + clean[5:7]
model = train(dirty)
ok = sum(predict(model, t) == l for t, l in trusted)
print("trusted-set score:", ok, "of", len(trusted))
# Defence 2: add each known word to one clearly negative sentence.
# Count how many words flip it. One lone word that flips it is suspicious.
words = sorted({w for text, _ in dirty for w in text.split()})
flips = [w for w in words if predict(model, "terrible rude service " + w) == "pos"]
print("words that flip the answer:", flips)Output
before poisoning neg <- awful cold soup neg <- awful cold soup zebra neg <- terrible rude service neg <- terrible rude service zebra after poisoning neg <- awful cold soup neg <- awful cold soup zebra neg <- terrible rude service pos <- terrible rude service zebra trusted-set score: 4 of 4 words that flip the answer: ['zebra']
Reading the result
Look at the first block, before poisoning. All four sentences come out "neg", with or without "zebra", because the word was never seen. The model ignores it.
After poisoning, "terrible rude service" is still "neg". The model works fine for normal use. But "terrible rude service zebra" flips to "pos". Just 3 bad rows out of 13 gave one word the power to turn a bad review into a good one.
Notice the other test. "awful cold soup zebra" stayed "neg". The backdoor is not perfect, because "awful" and "cold" are strong negative words in this toy. Real attackers pick the number of examples and the wording to make the trigger win. A toy only shows the shape.
The first defence line, "trusted-set score: 4 of 4", is a warning. We checked the poisoned model on four reviews from a trusted set, and it got every one right. A test set with no trigger cannot see a backdoor. Normal accuracy looks perfect while the hole stays open.
The second defence worked in this toy. We added every known word to one clearly negative sentence and listed the words that flipped the answer. Only "zebra" did. It is a crude trigger hunt, and a real trigger can be a phrase or a pattern, so it will not catch everything.
Quick check
The poisoned model scores 4 of 4 on a clean trusted test set. What does that tell you?
How to defend
No single check stops poisoning. The best protection stacks several. OWASP's prevention list for LLM04 says, among other things, to "Track data origins and transformations", to "Vet data vendors rigorously, and validate model outputs against trusted sources", to "Use anomaly detection techniques to filter out adversarial data", and to "Use data version control (DVC) to track changes in datasets and detect manipulation". Wikipedia's page lists "Sanitizing training data" and "Backdoor detection algorithms" among proposed defences.
Provenance
Record where each piece of data came from, who added it and when. If something turns out bad, you can find the rest from the same source.
Review and version control
Keep data in versions, like code. A change that nobody approved shows up in the history.
Deduplication and outlier checks
Remove copies, because poisoned examples may be repeated copies. Then look for odd rows: a rare word that always carries one label, or a label that disagrees with the text, like our three "zebra" reviews.
- 1
Limit who can write
Fewer writers means fewer ways in. This applies to datasets and to RAG knowledge bases.
- 2
Pin and check data
Use fixed versions of datasets, and keep a record of each one's origin.
- 3
Test on trusted sets
Keep a small, hand-checked set the attacker cannot touch. Include trigger-style checks, not just normal sentences.
- 4
Watch behaviour
Compare the model's answers over time. OWASP suggests to "Monitor training loss and analyze model behavior for signs of poisoning".
- 5
Red team it
With permission, try to poison your own test copy. OWASP recommends testing robustness with red team campaigns.
Related lessons: Continuous monitoring covers watching a model after launch, and Adversarial training and robust design covers building models that resist bad inputs.
Quick check
Quick check
Which describes a backdoor?
Quick check
A RAG app is poisoned. What changed?
Quick check
Which is a good habit against poisoning?
Remember the main idea: a model is only as trustworthy as what it learned from. Know your sources, limit who can write, and test for triggers as well as for normal behaviour.
Next: adversarial examples, tiny changes that fool a model.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.