AI guardrails & red teaming / Beginner / 01.04
How models learn, for attackers
Four ways a model learns, and the one thing in each that an attacker can tamper with: the labels, the raw data, the reward and the weights.
12 min readFree
Why learning is the weak spot
The last lesson, Ethics and permission, ended with a promise: next, how models learn, seen by an attacker. Here it is.
A shop clerk learns the job from what the boss shows them and from how the boss reacts. A model works the same way, only it learns from data. So whoever controls the data, or the feedback, has a hand on what the model becomes. That is why learning is a favourite target.
One rule before anything else. Test only systems you own, or systems where you have written permission to test. Everything in this lesson is a made-up toy that runs on your own laptop, and it should stay that way.
You will meet four ways a model learns. For each one you will see the thing an attacker can tamper with, what it breaks, and what helps. The Machine Learning Foundations lessons teach each way in depth. Here we only ask one question: where could someone mess with it?
Key words
- Training
- The stage where a model learns, by looking at many examples and adjusting itself.
- Label
- The correct answer attached to an example, like a sticker that says "cat" on a photo.
- Reward
- A score that tells a learner how well it just did.
- Weights
- The numbers inside a model that hold what it learned. Training changes them.
- Data poisoning
- Slipping bad examples into the data a model learns from, so that it learns the wrong thing.
- Backdoor
- A hidden habit planted during training: the model acts normally until it sees a secret trigger.
Wikipedia defines adversarial machine learning as the study of attacks on machine learning algorithms, and of the defences against such attacks. NIST, the US standards agency, published a report that lists attack types in an orderly way. Its summary names data poisoning and evasion among its topics. In this lesson we look at the poisoning side, because it happens while the model is learning.
Supervised learning: the labels
Supervised learning is learning from answers. Wikipedia says an algorithm learns to map input data to an output based on example input-output pairs, using labelled data, where each piece of input comes with the correct output. The word "supervised" points to a teacher who supplies those answers. The Supervised learning lesson goes deeper.
Think of flash cards. The question is on the front, the answer is on the back. If someone swaps a few backs, you will study hard and still learn the wrong thing.
That is the weak spot. The model trusts the labels completely. It has no other way to know what is right.
What the attacker touches
The labels, or the examples next to them. Anyone who helps collect or tag data (a crowd of workers, a feedback button, user uploads) is a possible door.
What it breaks
The model learns a bent rule. It can get worse everywhere, or only on one kind of input the attacker cares about.
What helps
Trusted sources, spot checks by a second person, and a test set that no outsider can write to.
Wikipedia describes poisoning as "contaminating the training dataset with data designed to increase errors in the output", and says that because algorithms are shaped by their training data, poisoning can effectively reprogram them. It raises the concern especially for user-generated training data. A backdoor is a special case: it teaches a specific behaviour for inputs that carry a given trigger.
Tip
When data comes from many hands, treat every label as a claim to check, not as a fact.
Try it: flip two labels
Let us watch this happen. We build the smallest classifier we can, a nearest-centroid model. Training works out the average spot of each group, called its centre. To guess a new point, it picks the group whose centre is closest.
In words: the centre of group k is all the points in that group added up, then divided by how many there are. It is just an average.
The data is 8 made-up points in 2D, four labelled "cat" and four labelled "dog", plus 4 fixed test points the model never trains on. No randomness, so you get the same output every time. This classifier is a tiny stand-in, written here so the code runs free with no keys and no network. Real models are bigger and behave less predictably, but the idea carries over.
- 1
Train on honest labels
Find the centre of the cat points and the centre of the dog points. Score the model on the 4 test points.
- 2
Flip two labels
Change two dog points to "cat". This is our toy poisoning. Nothing else changes: same points, same code.
- 3
Retrain and score again
Train from scratch on the changed labels and score the same 4 test points.
- 4
Try a simple check
Compare each label with what the first model would have said, and flag the ones that disagree.
points = [(1, 1), (2, 1), (1, 3), (3, 2), (6, 6), (7, 5), (5, 7), (7, 7)]
labels = ["cat", "cat", "cat", "cat", "dog", "dog", "dog", "dog"]
tests = [(3, 3), (4, 3), (5, 4), (4, 5)]
answers = ["cat", "cat", "dog", "dog"]
def train(points, labels):
centroids = {}
for name in sorted(set(labels)):
members = [p for p, l in zip(points, labels) if l == name]
centroids[name] = (
round(sum(x for x, y in members) / len(members), 2),
round(sum(y for x, y in members) / len(members), 2),
)
return centroids
def predict(centroids, point):
def dist(name):
cx, cy = centroids[name]
return (point[0] - cx) ** 2 + (point[1] - cy) ** 2
return min(sorted(centroids), key=dist)
def score(centroids):
guesses = [predict(centroids, t) for t in tests]
right = sum(g == a for g, a in zip(guesses, answers))
return f"{right} of {len(tests)} right, guesses {guesses}"
clean = train(points, labels)
print("clean centres: ", clean)
print("clean model: ", score(clean))
bad_labels = list(labels)
bad_labels[4] = "cat"
bad_labels[7] = "cat"
poisoned = train(points, bad_labels)
print("poisoned centres:", poisoned)
print("poisoned model: ", score(poisoned))
for p, given in zip(points, bad_labels):
if predict(clean, p) != given:
print("flag for review:", p, "is labelled", given)Output
clean centres: {'cat': (1.75, 1.75), 'dog': (6.25, 6.25)}
clean model: 4 of 4 right, guesses ['cat', 'cat', 'dog', 'dog']
poisoned centres: {'cat': (3.33, 3.33), 'dog': (6.0, 6.0)}
poisoned model: 2 of 4 right, guesses ['cat', 'cat', 'cat', 'cat']
flag for review: (6, 6) is labelled cat
flag for review: (7, 7) is labelled catRead the output. With honest labels the model gets 4 of 4 right. The cat centre sits near (1.75, 1.75) and the dog centre near (6.25, 6.25). After the two flips, the cat group now holds two faraway dog points. Its centre slides to (3.33, 3.33), and the dog centre moves to (6.0, 6.0). The line between the groups shifts toward the dog side, and the model gets only 2 of 4 right. It now says "cat" for all four test points.
Two changed labels out of eight is a big share in a tiny toy. Real datasets are far larger. The lesson is the shape of the problem: change what the learner is told, and its behaviour changes with it.
The last lines show one simple defence. We compare each label with the guess of the earlier, trusted model. Two points disagree, so a person reviews them. In real life you rarely have a perfect clean model, so use an older version, a small hand-checked set, or a second model. Treat any disagreement as a flag to look at, not as proof.
Quick check
After the two labels are flipped, what happens to the cat centre?
Unsupervised learning: the raw data
Unsupervised learning has no answer sheet. Wikipedia says its algorithms learn patterns exclusively from unlabelled data. Think of sorting a bag of mixed buttons into piles without anyone telling you the piles. Finding groups is one common job. The Unsupervised learning lesson has more.
Wikipedia adds that such a dataset is typically harvested cheaply "in the wild", for example a huge pile of text collected by web crawling, with only minor filtering. That detail matters for security. If anyone can publish a page, then anyone has a chance to be part of the pile.
What the attacker touches
The raw data itself: the pages, files or records that get collected. There are no labels to flip, so the attacker adds or shapes the examples.
What it breaks
The groups the model finds can shift or blur. A fake pile of similar-looking records can pull a group toward a point the attacker chose.
What helps
Know where each data source came from, filter and sample it, and look at the groups with your own eyes before you trust them.
The unlabelled case is sneaky because nothing looks wrong. There is no label to double-check. So the checks move to the source: who can add data, and how would we notice a strange new cluster?
Reinforcement learning: the reward
Reinforcement learning is learning by trying and being scored. Wikipedia describes it as concerned with how an agent should take actions in a changing environment to maximise a reward signal. A dog learning a trick with treats is the everyday picture. The Reinforcement learning lesson covers the details.
The agent learns a policy, a habit for picking actions. The aim is the one that earns the most reward over time. So the reward is the only voice the agent listens to.
What the attacker touches
The reward signal, or whatever feeds it: the judge, the score counter, the human rating, the feedback form.
What it breaks
The agent chases the score, not the real goal. If the score can be fooled, the agent may learn to fool it.
What helps
Score what you truly want, protect the thing that does the scoring, and watch what the agent actually does, not only its points.
This is the same idea as the dog and the treat jar. If a stranger can hand out treats whenever the dog does something silly, the dog will learn silly. The agent has no sense of what you meant. It has only the number it was given.
Tip
Careless scoring can go wrong even when nobody attacks. Attackers just make it worse on purpose. When you design a reward, ask: how could someone, or something, earn a high score without doing the real job?
Neural networks: the weights
A neural network is one kind of model, and the one many modern AI tools are built on. Wikipedia describes it as a computational model inspired by the structure of biological neural networks. It has connected units called artificial neurons. The strength of the signal along each connection is set by a weight, which adjusts as part of the training process. The Neural network basics lesson explains how.
Here is a picture. Imagine a huge board of dials. Training turns the dials a little at a time until the answers look right. When it is done, the dial positions are what the model knows. Those dials are the weights.
What the attacker touches
The weights, in two ways. They can poison the data the weights are learned from, or change the finished weights themselves, for example in a file that gets shared and downloaded.
What it breaks
A backdoor can hide in the dials. The model passes your normal tests, then misbehaves when a secret trigger shows up.
What helps
Get models only from sources you trust, check that a file is the one its owner published, and test for odd behaviour, not just accuracy.
Because the weights are just numbers, nobody can read a backdoor in them by looking. That is why the checks happen around the model: where the data came from, where the file came from, and what the model does on inputs you did not expect.
Later lessons look at attacks that need no tampering with learning at all, like the Prompt injection attacks lesson in AI engineering, which covers text that takes over an app after the model is built.
Quick check
Which part would you tamper with to hide a backdoor in a neural network?
What helps, in one list
Each way of learning has a different weak spot, but the habits that protect you are similar. Three go a long way.
Know your sources
Write down where every piece of data and every model file came from, and who could change it along the way.
Keep a trusted test set
Hold back examples that nobody outside the team can touch. Wikipedia says performance should be measured on a test set separate from the training set. Here it also works as an alarm: a sudden drop on a trusted set is worth a look.
Watch for change
Compare each new version with the last one. Flag big shifts in the labels, the groups, the scores or the behaviour, and let a person decide.
Tip
As a tester, your job is to find these weak spots before someone else does, inside the permission you were given. Write down what you tried, what happened and how to fix it.
Quick check
A team trains on user feedback. What is the best first protection?
Recap
Every way of learning has a place where the model trusts what it is given. Supervised learning trusts the labels. Unsupervised learning trusts the raw data. Reinforcement learning trusts the reward. A neural network keeps what it learned in its weights, so it trusts whatever shaped them.
You also ran a toy that lost half its test score from two flipped labels, and saw a simple label check that flagged exactly those two. Keep that picture. When you test a real product, with permission, ask: what does this model trust, and who can reach it?
Next: generative models and large language models.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.