esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Core / 02.07

Adversarial training and robust design

Teach a model with attacked examples, see what that costs, and learn the design habits that keep an AI app safe when the model still gets fooled.

12 min readFree

From attack to defence

The last lesson, Adversarial examples, ended with a promise: next, adversarial training and robust model design. Here it is. You saw how a tiny, well-aimed change to an input can make a model give the wrong answer. Now the question is what we do about it.

There are two answers. One lives inside the model: train it on the tricky inputs so it learns to resist them. The other lives around the model: build the app so one wrong answer does not hurt anyone. A good team does both.

A rule before we start. Test only systems you own or have written permission to test. Everything here is defensive, and every example is made up: secret fruit words, toy numbers, no real product.

Key words

Adversarial example
An input changed on purpose so that a model gets it wrong, while it still looks normal to a person.
Adversarial training
Training a model on a mix of normal examples and attacked ones, so it learns to answer them correctly.
Robust
Hard to knock over. A robust model keeps giving the right answer even when the input is pushed.
Clean accuracy
How often the model is right on normal, untouched examples.
Attacked accuracy
How often the model is right on examples that were changed to fool it.
Confidence
How sure the model is. Many models give a number, like "92% spam".
Abstain
To say "I will not answer this one" and hand it to someone else.
Human in the loop
A person checks or approves the cases the system is unsure about.

The idea: practise on the hard ones

Think of a student who only ever sees easy practice questions. The exam has tricky ones, and the student freezes. A good teacher mixes in the tricky ones early. Adversarial training does exactly that for a model.

The recipe is short. Take your training examples. Attack them, using the kind of attack you worry about. Keep the correct label on each attacked copy. Train on the originals and the attacked copies together.

  1. 1

    Train a normal model

    Get a model that does well on clean examples.

  2. 2

    Attack your own examples

    Change each one slightly in the direction that hurts the model most. You own the data and the model, so this is safe.

  3. 3

    Keep the right answers

    Each attacked copy keeps the label of the original. A spam message that was nudged is still spam.

  4. 4

    Train again on both piles

    Now the model sees normal and attacked examples, and learns that the nudge should not change the answer.

  5. 5

    Test with a fresh attack

    Attack the new model, since the old attack was aimed at the old one. Then measure clean and attacked accuracy.

The idea has two famous sources. Goodfellow and colleagues describe adversarial examples as "inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset". They then say that using their fast attack method "to provide examples for adversarial training", they "reduce the test set error of a maxout network on the MNIST dataset". MNIST is a well-known set of handwritten digits. So the first paper already showed that practising on attacked examples helps.

Madry and colleagues took it further. They study the problem "through the lens of robust optimization", and say their methods "let us train networks with significantly improved resistance to a wide range of adversarial attacks". Robust optimization means planning for the worst case rather than the average case.

The worst case, in one line

You can write that worst-case idea as one formula. Do not worry if it looks odd. It is read in words right below.

min⁡θ  E[ max⁡∥δ∥≤ϵL(θ, x+δ, y) ]\min_{\theta} \; \mathbb{E}\Big[\, \max_{\|\delta\| \le \epsilon} L(\theta,\, x+\delta,\, y) \,\Big]

In words: the model has settings, written theta. For each example, an attacker may add a small push, delta, that is no bigger than epsilon. The inner part asks how bad the loss (the model's score for being wrong) can get under the worst allowed push. The outer part says: choose model settings that make that worst case as small as possible, on average over the training data.

So the attacker's best move is part of the training. The model is trained against an opponent, not against random noise.

Try it on a toy model

Let us run this on a tiny model you can follow line by line. The model is a stand-in so the code runs free and gives the same result every time. Real models are bigger and behave less predictably.

The data is twelve made-up points on a 2D map. Six sit in the top right (label +1) and six in the bottom left (label -1). The model draws a straight line between them. It learns with simple perceptron updates: each time it gets a point wrong, it shifts the line toward getting that point right.

The attack is a sign step. For each point, look at which way the model's weights push, and move the point one step of size EPS in the opposite direction, toward the wrong side. Then train again on the original points plus the attacked copies.

Python
# Stand-in model: a tiny linear classifier (perceptron) on made-up 2D points.
# Real models are bigger and less predictable; this one runs free and gives the same output every time.
points = [
    ((1.0, 1.2), 1), ((1.6, 0.8), 1), ((0.9, 2.0), 1), ((2.2, 1.4), 1), ((1.3, 1.6), 1), ((2.0, 2.2), 1),
    ((-1.0, -1.1), -1), ((-1.5, -0.7), -1), ((-0.8, -2.0), -1), ((-2.1, -1.5), -1), ((-1.2, -1.7), -1), ((-2.0, -2.1), -1),
]

def sign(v):
    return 1 if v > 0 else -1 if v < 0 else 0

def score(model, x):
    w, b = model
    return w[0] * x[0] + w[1] * x[1] + b

def train(model, data, rounds=20):
    w, b = list(model[0]), model[1]
    for _ in range(rounds):
        for x, y in data:
            if y * score((w, b), x) <= 0:
                w[0] += y * x[0]
                w[1] += y * x[1]
                b += y
    return (w, b)

def attack(model, data, eps):
    w = model[0]
    return [((x[0] - eps * y * sign(w[0]), x[1] - eps * y * sign(w[1])), y) for x, y in data]

def accuracy(model, data):
    return sum(1 for x, y in data if y * score(model, x) > 0) / len(data)

EPS = 1.0

model = train(([0.0, 0.0], 0.0), points)
print("before: clean", accuracy(model, points), "| attacked", accuracy(model, attack(model, points, EPS)))

extra = attack(model, points, EPS)
robust = train(model, points + extra)
print("after:  clean", accuracy(robust, points), "| attacked", accuracy(robust, attack(robust, points, EPS)))

print()
for size in (0.5, 1.0, 1.2, 1.5):
    print("push size", size, "->", round(accuracy(robust, attack(robust, points, size)), 2))

Output

before: clean 1.0 | attacked 0.75
after:  clean 1.0 | attacked 1.0

push size 0.5 -> 1.0
push size 1.0 -> 1.0
push size 1.2 -> 0.67
push size 1.5 -> 0.42

Read the output. Before training on attacked points, the model is right on all of the clean points (1.0 means 100%). Under the attack it falls to 0.75, so three points in twelve are now wrong. After we train on the clean and attacked points together, it is right on all clean points and on all attacked points at that same push size.

Now read the last four lines, because they hold the lesson. We attacked with a push of 1.0, and the model learned to resist that. A push of 0.5 or 1.0 is handled. A push of 1.2 already brings it down to 0.67, and 1.5 to 0.42. Training against one size of attack does not protect you against a bigger one.

Tip

In this toy, clean accuracy did not drop. Do not expect that for free in real life. See the next section.

What it costs

Adversarial training is not a free upgrade. Three costs come with it.

  • More work to train

    Every round, you must make attacked copies before you can learn from them. In our toy that was one extra line. For a big model, building good attacks over and over is a lot of extra computing.

  • Possibly worse on normal inputs

    The Wikipedia page on adversarial machine learning says studies of linear models can explain "the trade-off between robustness and accuracy". A model asked to resist pushes may give up a little accuracy on clean data.

  • Only as good as the attack you picked

    The model learned to resist the pushes it saw. Our toy showed it: bigger pushes still broke it. Madry and colleagues frame the goal as "security against a first-order adversary", a clearly defined class of attacker, not every attacker.

That last point matters for how you talk about it. Never say "our model is secure". Say which attacks it was trained against, how strong they were, and how you tested it. The Black, white and grey box testing lesson covers how to test with different levels of knowledge.

Quick check

You trained a model against pushes of size 1.0 and it passes. A teammate says it is now immune to attacks. What is the best reply?

Robust design around the model

Even a well-trained model will sometimes be fooled. So the second half of defence is design: build the app so a wrong answer is caught, small, or harmless. Think of a car. Good tyres matter, but you still add seat belts, brakes and a road with barriers.

These habits work for any model, including the chat models you build apps on.

Check inputs before the model sees them

Set simple rules at the door. Is the text a sensible length? Is the image a normal size and type? Are the numbers in a believable range? Anything that fails is rejected or cleaned before the model sees it. Many bad inputs are stopped here, cheaply, with no AI needed.

This is the same "validate at the boundary" idea used all over software. The Prompt injection defences lesson shows how it applies to prompts.

Know when to say "not sure"

Many models report how confident they are. Use it. Pick a high line and a low line. Above the high line, act. Below the low line, act the other way. In between, do not guess: abstain and hand the case to a person. That person is your human in the loop.

Here is the idea with a stand-in spam checker. The numbers are made up. A real model gives such numbers too, but they are less tidy, and a clever input can sometimes get a confident wrong answer, so treat confidence as a helpful signal, not a promise.

Python
# Stand-in: a made-up spam checker. Each message already has a spam chance from 0 to 1.
# A real model gives such a number too, but real numbers are less tidy than these.
messages = [
    ("win a free fruit basket", 0.97, "spam"),
    ("lunch at noon?", 0.03, "ok"),
    ("fr33 fru1t basket", 0.55, "spam"),
    ("meeting notes attached", 0.08, "ok"),
    ("claim your mango prize", 0.91, "spam"),
    ("quarterly plan draft", 0.45, "ok"),
]

def decide(chance, low=0.2, high=0.8):
    if chance >= high:
        return "spam"
    if chance <= low:
        return "ok"
    return "ask a person"

for text, chance, truth in messages:
    print(f"{text:26} chance {chance:.2f} -> {decide(chance)}")

sure = [(c, t) for _, c, t in messages if decide(c) != "ask a person"]
right = sum(1 for c, t in sure if decide(c) == t)
print(right, "of", len(sure), "automatic answers right;", len(messages) - len(sure), "sent to a person")

Output

win a free fruit basket    chance 0.97 -> spam
lunch at noon?             chance 0.03 -> ok
fr33 fru1t basket          chance 0.55 -> ask a person
meeting notes attached     chance 0.08 -> ok
claim your mango prize     chance 0.91 -> spam
quarterly plan draft       chance 0.45 -> ask a person
4 of 4 automatic answers right; 2 sent to a person

The message with swapped letters, "fr33 fru1t basket", sits in the middle at 0.55, so it goes to a person instead of being guessed. Four automatic answers were right, and two uncertain ones went to a human. The price is that a person has to look at some cases. That is usually a good deal, and you can move the two lines to change how many.

Tip

Put the human check where a wrong answer is costly: money, accounts, safety, anything hard to undo. Let the system act alone only on the sure, low-risk cases.

Layers, and a warning about ensembles

An ensemble is several models that vote. It sounds like a sturdy fix, because an attacker must fool them all. But be careful. The Wikipedia page lists ensembles among proposed defences and says they "have shown to be ineffective against evasion attacks". It also says gradient masking, a trick that hides the model's inner signals from an attacker, "is deemed unreliable". So do not count on one clever trick.

The safer habit is layers. Input checks, a trained-for-robustness model, a confidence line, a human check, and logs that let you spot odd traffic. Each layer is imperfect. An attack has to get through all of them.

A related idea: safety tuning for chat models

Chat models have a cousin of adversarial training. Builders collect tricky requests that try to push a model off its rules, pair them with the answer they want, such as a polite refusal, and train on them. It is the same practice-on-the-hard-ones idea, applied to words.

It has the same limits. It covers the tricks the builders thought of, and new ones keep appearing. That is why the Jailbreaks and filter bypasses and Prompt injection, direct and indirect lessons stress layers around the model, not only the model. The Prompt injection attacks lesson in AI engineering goes deeper on the attacks themselves.

Quick check and wrap-up

Quick check

A model is unsure about an input: its confidence sits in the middle. What does a robust design do?

Two ideas to keep. Adversarial training teaches a model by practising on attacked examples, and it costs extra work and can cost clean accuracy. Robust design assumes the model will sometimes be fooled and builds checks, an abstain line, a human and several layers around it.

The Continuous security testing lesson shows how to keep checking all of this over time, because attackers do not stand still.

Next: model inversion, getting private data back out of a model.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.