AI guardrails & red teaming / Core / 02.06
Adversarial examples
Tiny changes to an input, often too small to notice, that flip a model's answer. How they work, why they work, and how to spot them.
13 min readFree
A small change, a different answer
The last lesson, Data poisoning, ended with a promise: next, adversarial examples, tiny changes that fool a model. Poisoning spoils the lesson book a model learns from. This time the model is already trained and fixed. The trick happens afterwards, on the one input you hand it.
Picture a ripe-fruit sorter in a warehouse. A camera looks at each fruit and a model says "ripe" or "unripe". Now imagine someone changes the picture so slightly that you could never see it, and the sorter changes its mind. That is an adversarial example.
One rule before we start. Test only systems you own or have written permission to test. Everything here is taught so you can notice these tricks and defend against them. The code uses small stand-in models you can run for free, and nothing in it touches a real product.
Key words
- Adversarial example
- An input changed on purpose, by a small amount, so that a model gives a wrong answer.
- Perturbation
- The small change added to the input. Think of it as a smudge on a window.
- Classifier
- A model that sorts inputs into groups, like "ripe" and "unripe", or "spam" and "not spam".
- Evasion attack
- Changing an input at the moment of use so a model misjudges it. The model itself is left alone.
- Gradient
- The direction in which a model's answer changes fastest when an input number changes. Think of the slope of a hill.
- FGSM
- Fast gradient sign method: a quick way to build a perturbation, using only the direction of the gradient.
What the research found
The first report came in 2013. In the paper "Intriguing properties of neural networks", the authors say they can "cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error". Imperceptible means you cannot see it.
The same abstract adds a second surprise. The same perturbation can cause a different network, trained on a different part of the data, to misclassify the same input. So the trick is not a quirk of one model.
A year later, Goodfellow, Shlens and Szegedy wrote "Explaining and Harnessing Adversarial Examples". Their abstract defines the idea as inputs "formed by applying small but intentionally worst-case perturbations to examples from the dataset", which make the model give "an incorrect answer with high confidence".
That paper has a famous picture, its Figure 1. It shows a photo that the image model GoogLeNet labels "panda" with 57.7% confidence. The authors add a tiny pattern, with a step of 0.007 per value, which they say matches the smallest step of an 8-bit image. The new picture looks the same to a person, but the model now says "gibbon" with 99.3% confidence. Note that jump: it is not just wrong, it is sure.
Tip
"Confidence" is a number the model prints. It is not a promise. A model can be very sure and very wrong at the same time.
How the fast method works
Now the question is how anyone finds the right smudge. The 2015 paper offers a quick recipe, and Wikipedia's page on adversarial machine learning describes it too. It is called the fast gradient sign method, or FGSM.
Think of the model's answer as a hill. Each number in the input is a small lever. If you push a lever one way, the model leans toward "ripe". If you push it the other way, it leans toward "unripe". The gradient tells you which way each lever pushes. FGSM pushes every lever one tiny, equal step in the direction that hurts the model most. It uses only the sign of each push, plus or minus, not its size.
In words: take the original input x. Work out how the model's error J would change if each input number moved a little. Keep only the sign of each change. Multiply by a small number, epsilon, and add the result to x. Wikipedia explains the trade-off: as epsilon gets bigger the model is more likely to be fooled, but the change also becomes easier for a person to notice.
- 1
Start
Take an input the model gets right.
- 2
Find the slope
For each number in the input, ask which way it pushes the model's error.
- 3
Keep the sign
Throw away the size of each push. Keep only plus or minus.
- 4
Take one small step
Move every number by epsilon in its direction. Then check whether the answer flipped.
Notice what this needs. The method looks at the model's inner numbers, so it fits someone who can see inside the model. Wikipedia files FGSM under white box attacks, which assume the attacker has access to the model's parameters. Attackers without that access use other methods, and the Black, white and grey box testing lesson covers what each level of access means for a tester.
Why it works
Early guesses blamed the way deep networks bend and twist, or learning the training data too closely. The Goodfellow paper says no to both. In its abstract: "We argue instead that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature."
Linear means the answer is built by adding up the inputs, each one multiplied by a weight, which is how much that input matters. Here is the idea in plain words, following the paper's argument.
Each change is tiny
One number moves by less than a sensor could even record, so a person sees nothing.
The changes line up
Each tiny move points in the direction the weight favours, so every move helps the same cause.
Many tiny moves add up
The paper says the activation grows with the number of dimensions, so in high dimensional problems many infinitesimal changes "add up to one large change to the output".
Picture a crowd of ten thousand people each taking one tiny step the same way. No single step is noticeable, but the crowd moves a long way. An image has thousands of pixels, so there are thousands of tiny steps.
The authors call this a sort of "accidental steganography". Steganography means hiding a message inside something else. The model is forced to pay attention to the signal that lines up best with its weights, even when other signals are much bigger.
Tip
This is also why a bigger or fancier model is not a cure. The paper's point is that the weakness comes from the way these models add things up, not from one unlucky model.
Try it: nudge a stand-in classifier
Time to see it with numbers. Below is a tiny stand-in model: a fixed linear classifier with made-up weights, reading six numbers about a fruit. It is a stand-in so the code runs free and offline. Real models are bigger and behave less predictably, but the idea carries over.
For a linear model, the slope for each input number is simply its weight. So the FGSM step is easy: move each number by 0.1, against the sign of its weight, to push the "ripe" score down.
import math
# A stand-in model: a fixed linear classifier with made-up weights.
weights = [0.9, -0.6, 0.4, 0.8, -0.5, 0.7]
bias = -0.8
def score(x):
z = bias + sum(w * v for w, v in zip(weights, x))
return 1 / (1 + math.exp(-z))
def label(x):
return "ripe" if score(x) >= 0.5 else "unripe"
def sign(n):
return 1 if n > 0 else -1 if n < 0 else 0
x = [0.6, 0.2, 0.5, 0.4, 0.3, 0.5]
step = 0.1
# For a linear model the slope of the score for each number is its weight,
# so one small push per number, against the weight's sign, lowers the score.
x_new = [v - step * sign(w) for v, w in zip(x, weights)]
print("before:", round(score(x), 3), label(x))
print("after: ", round(score(x_new), 3), label(x_new))
print("largest change:", round(max(abs(a - b) for a, b in zip(x, x_new)), 3))
print("new numbers:", [round(v, 2) for v in x_new])Output
before: 0.584 ripe after: 0.488 unripe largest change: 0.1 new numbers: [0.5, 0.3, 0.4, 0.3, 0.4, 0.4]
Read the output line by line. Before the nudge, the score is 0.584, so the model says "ripe". After it, the score is 0.488, just under the 0.5 line, so the model says "unripe". The largest change to any single number is 0.1. Every number moved by exactly that much, in the direction its weight points.
That is the whole trick in miniature. No number changed by more than 0.1, yet the label flipped. Here the model was only a little sure to begin with, so a small push was enough. A model that was very sure would need a bigger push, or more numbers. Real image models have so many numbers that tiny pushes add up fast.
Tip
Try it yourself: change step to 0.05 and run again. Does the label still flip? Then try 0.2. You are finding the smallest nudge that works, which is how testers measure how fragile a model is.
Text has its own versions
Images are easy to nudge because pixels are numbers. Words are not smooth numbers, so changes to text look different: a swapped letter, a typo, a word replaced by a near-synonym, a lookalike character. The idea is the same. A person still reads the message the same way, while the filter reads it differently.
Wikipedia gives an old, real example from the world of email. Spammers and hackers often try to evade detection by obfuscating the content of spam emails, meaning disguising what the message says. Here is a harmless version with a stand-in filter. The message, the words and the filter are all made up.
# A stand-in filter: it flags a message that holds two or more "suspicious" words.
suspicious = {"free", "prize", "winner", "claim"}
def flagged(message):
words = message.lower().replace("!", "").split()
return sum(w in suspicious for w in words) >= 2
def normalise(message):
swaps = str.maketrans("013$", "oies")
return message.lower().translate(swaps)
original = "Free prize for the lucky winner!"
tweaked = "Fr3e pr1ze for the lucky w1nner!"
print("original flagged:", flagged(original))
print("tweaked flagged: ", flagged(tweaked))
print("tweaked after cleanup:", flagged(normalise(tweaked)))Output
original flagged: True tweaked flagged: False tweaked after cleanup: True
The plain message is flagged. The version with three swapped characters slips through, though a person reads it just fine. The third line shows one simple defence: clean the text first, so lookalike characters turn back into letters, then run the filter. Real language models are less predictable than this toy, and a cleanup step catches only the tricks you thought of. That is why it is one layer, not the whole answer.
Language models have a second route in, covered elsewhere. The Prompt injection attacks lesson in AI engineering goes deeper on instructions hidden in text, and this course's Prompt injection, direct and indirect covers the security side. Adversarial examples are about changing what the model perceives. Injection is about changing what it obeys.
Where this sits in the NIST list
In January 2024 NIST published a report titled "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations" (AI 100-2 E2023). A taxonomy is a sorted list of kinds of things. The report's abstract says it is arranged by types of machine learning methods, lifecycle stages of attack, attacker goals and objectives, and attacker capabilities and knowledge.
The page also lists "evasion" and "data poisoning" among its keywords. That gives you a simple map of the two lessons side by side.
Data poisoning
Happens during training. The attacker spoils the examples the model learns from.
Evasion
Happens at use time. The attacker changes an input so a finished model misjudges it. Adversarial examples are the classic case.
What links them
Both work on the data a model sees, not on the servers around it. Both need their own tests.
Wikipedia agrees on the definition. Evasion attacks, it says, consist of exploiting the imperfection of a trained model, and they do not involve influence over the training data. It splits them into black box attacks, where the attacker sees only the model's answers, and white box attacks, where the attacker also sees its parameters.
What a builder can do
You cannot make a model immune today, but you can make attacks harder and easier to spot. Here are habits for people who build or test apps.
- 1
Do not trust one answer
For important calls, add a second check: another model, a rule, or a person. Fooling two different checks at once is harder.
- 2
Clean the input
Resize, compress, normalise or strip odd characters before the model sees it. This can remove some tiny changes, but not all of them.
- 3
Watch for oddities
Log inputs that get very high confidence on strange data, or answers that flip when the input barely changes. Your monitoring can raise a flag.
- 4
Test with small changes
Take your own test set, nudge each item a little, and count how many answers flip. A high count means a fragile model.
- 5
Train against it
The Goodfellow paper itself uses this recipe: it used FGSM examples for adversarial training, and reduced the test set error of a maxout network on the MNIST dataset. The next lesson covers it.
Notice that step four is a test you run on your own system, with your own data. That is the safe and permitted way to do this work.
Quick checks
Quick check
In the panda picture, what was added to the photo?
Quick check
According to the Goodfellow paper, what is the main cause of this weakness?
Quick check
What does FGSM keep from each slope?
Quick check
Which rule applies to every test in this course?
In short: a tiny, often invisible change can flip a model's answer, FGSM builds one by pushing every input number one small step, the paper blames the way these models add things up, and text has its own versions of the trick. Defence starts with checking, cleaning and testing your own system.
Next: adversarial training and robust model design.
Sources
- arXiv: Explaining and Harnessing Adversarial Examples (abstract)
- arXiv: Explaining and Harnessing Adversarial Examples (full text, Figure 1 and the linear explanation)
- arXiv: Intriguing properties of neural networks (abstract)
- Wikipedia: Adversarial machine learning
- NIST: AI 100-2 E2023, Adversarial Machine Learning (publication page)
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.