esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Beginner / 01.11

Model stealing

How a model can be copied just by asking it questions, why a few clever queries can be enough, and the simple guards that make copying harder.

13 min readFree

Copying a model by asking it

The last lesson, Jailbreaks and filter bypasses, ended with a promise: next, model stealing, copying a model through its API. Here it is.

A jailbreak tries to make a model say something it should not. Model stealing is different. The attacker does not want one answer. The attacker wants the whole model, or a copy that behaves like it, so they can use it for free.

One rule comes first, so here it is early. Test only systems you own, or systems you have written permission to test. Everything below is for understanding and defending. Nothing here is a guide to copying someone else's product.

By the end you will know what a model's weights are, two ways of stealing a model, what a copy attack looks like from the defender's side, and which guards make it harder. You will also run a tiny program that steals a toy model in two questions, then fails to do it once two guards are added.

Key words

Weights
The numbers inside a model that were learned during training. They are what the model "knows".
Parameters
Another word for the same learned numbers. Weights are the most common kind.
API
A door a program can knock on to ask another program for something. Here: you send a question, the model's owner sends back an answer.
Query
One question sent to a model through its API.
Black box
A system you can only use from outside. You see what goes in and what comes out, never the insides.
Model extraction
Copying a model's behaviour by asking it many questions and studying the answers.
Rate limit
A cap on how many requests one user can make in a set time.

What is worth stealing

A model is a recipe that turns inputs into outputs. Training makes the recipe: the model looks at lots of examples and slowly sets its weights. Think of a cook who tasted a thousand soups and finally knows how much salt each one needs. The weights are that knowing, stored as numbers.

The arXiv paper that started much of this research says why a model may be secret in the first place. Its abstract says: "Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications."

  • Sensitive training data

    A model trained on private records may carry traces of them. A copy can leak what the original was supposed to protect.

  • Commercial value

    A company may sell access to its model per question. A free copy means lost income.

Two ways to steal a model

There are two broad routes. One takes the file. The other copies the behaviour.

Key words

Stealing the file
Getting the weights themselves, by breaking into a place where they are stored.
Stealing by questions
Never touching the file. Sending many queries and training a new model on the answers.

Route 1: taking the file

Weights live in files. A file can sit on a laptop, in a shared folder, in a storage bucket or on a build server. If any of those is open to people who should not see it, the model can walk out the door in one download. No clever maths is needed.

The guard here is the dull one. The OWASP page lists access controls: use "role-based access control (RBAC) and the principle of least privilege" to "limit unauthorized access to LLM model repositories and training environments". Give each person only the access their job needs.

Tip

When you test an app, ask a boring question: where do the model files live, and who can open them? Many real problems are an unlocked door, not a clever trick.

Route 2: copying through questions

The second route needs no break-in. The attacker is just a normal user of the public API, asking questions and keeping the answers.

The 2016 paper by Tramèr and four co-authors studied exactly this. Its abstract describes "model extraction attacks": an adversary "with black-box access, but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i.e., \"steal\") the model".

The authors report that they showed "simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees". Their tests ran against the online services of BigML and Amazon Machine Learning.

The Wikipedia page on adversarial machine learning says extraction can reach the extreme case of model stealing: getting enough from the model "to enable the complete reconstruction of the model".

A toy model, stolen in two questions

A model is a recipe: training sets its weights, like a cook who tasted a thousand soups and now knows the salt each needs. Making it costs time and money. Copying costs less. Here is the smallest possible example. Our secret "model" follows one rule: multiply the input by 3, then add 2. In maths class that is y = 3x + 2. The 3 and the 2 are the model's weights, and the owner wants to keep them secret.

y=w⋅x+by = w \cdot x + b

In words: the answer y is the input x multiplied by a weight w, plus a starting value b. Here w is 3 and b is 2. The attacker does not know them.

Important: this is a stand-in, a tiny made-up model, so the code runs free on your computer with no keys and no network. Real models have millions or billions of weights and behave less predictably. But the idea of using answers as clues is the same.

  1. 1

    Ask for x = 0

    Anything times 0 is 0, so the answer is just b, the starting value. One question gives one weight.

  2. 2

    Ask for x = 1

    Now the answer is w plus b. Subtract the first answer and you are left with w.

  3. 3

    Build the copy

    With w and b known, the attacker writes their own function. It gives the same answers as the original for any input.

Python
# A stand-in "model": a secret rule y = 3x + 2 hidden behind a function.
SECRET_SLOPE, SECRET_START = 3, 2

def open_model(x):
    return SECRET_SLOPE * x + SECRET_START

print("Attack on the open model")
y0, y1 = open_model(0), open_model(1)
slope = y1 - y0
start = y0
print("2 queries gave", y0, "and", y1)
print("stolen copy: y =", slope, "* x +", start)
print("copy at x=10:", slope * 10 + start, "| real:", open_model(10))

Output

Attack on the open model
2 queries gave 2 and 5
stolen copy: y = 3 * x + 2
copy at x=10: 32 | real: 32

Read the output. Two questions gave the answers 2 and 5. The difference, 5 minus 2, is 3, so the weight is 3. The first answer, 2, is the starting value. The stolen copy is y = 3 * x + 2, exactly the secret. At x = 10 the copy says 32 and the real model says 32.

Quick check

In the toy example, why did the attacker need only two queries?

Why clean answers help the thief

Notice what made the theft easy. The answers were exact. The model said 2 and 5, not "about 2" and "close to 5".

The Tramèr abstract points at the same thing. It notes that online services "may accept partial feature vectors as inputs and include confidence values with predictions", and says it is "given these practices" that its simple attacks work. A confidence value is how sure the model says it is, such as "92% sure". It is a rich clue.

OWASP gives the matching advice in its list of guards: "Restrict or obfuscate the exposure of `logit_bias` and `logprobs` in API responses. Provide only the necessary information without revealing detailed probabilities." Obfuscate means to make something harder to read. Give the user what the app needs, not every number the model computed.

A word of care. The same abstract says that "omitting confidence values from model outputs still admits potentially harmful model extraction attacks". Hiding extra numbers helps but does not close the door, so defenders stack several guards.

Adding two guards

Now let's defend the toy. We add two guards to the same rule. First, answers are rounded to the nearest 5, so the attacker never sees the exact value. Second, each user gets only 4 queries, and the fifth gets nothing back.

The attacker tries the same plan: ask for several inputs, then use the two answers furthest apart to draw a line.

Python
# A stand-in "model" again: the same secret rule, now with two guards.
# Guard 1: answers are rounded to the nearest 5. Guard 2: only 4 queries per user.
class GuardedModel:
    def __init__(self, limit):
        self.left = limit

    def ask(self, x):
        if self.left == 0:
            return None
        self.left -= 1
        return round((3 * x + 2) / 5) * 5

def real(x):
    return 3 * x + 2

guarded = GuardedModel(limit=4)
answers = {}
for x in [0, 1, 20, 40, 60]:
    answers[x] = guarded.ask(x)
print("answers:", answers)

# The attacker uses the two answers it has furthest apart.
(xa, ya), (xb, yb) = (0, answers[0]), (40, answers[40])
slope = (yb - ya) / (xb - xa)
start = ya - slope * xa
print("copy: y =", slope, "* x +", start)
for x in [10, 100]:
    print("x =", x, "copy:", slope * x + start, "real:", real(x))

Output

answers: {0: 0, 1: 5, 20: 60, 40: 120, 60: None}
copy: y = 3.0 * x + 0.0
x = 10 copy: 30.0 real: 32
x = 100 copy: 300.0 real: 302

The copy is y = 3.0 * x + 0.0. The slope came out right, but the starting value is 0 where the real one is 2. So the copy is off. At x = 10 it says 30 and the real answer is 32. At x = 100 it says 300 against a real 302.

That is a small miss, and this toy copy is still close. Rounding hid the exact starting value but not the shape, and a bigger budget would let a smarter attacker claw some back. The guards made the theft less exact and more costly, not impossible. On a real service a tight limit also means the thief must ask for far longer, which gives you time to notice.

Copying by training on answers

The toy attack solved for exact numbers. For big language models, the attack looks different, and the goal is different too. The attacker does not try to read the weights. They try to build their own model that behaves similarly.

The idea is simple. Send a very large number of questions to the target model. Save every question and answer. Then use that pile as training data for a different model, which learns to imitate the answers.

The Wikipedia page notes that since 2025, AI developers have used the term model stealing for "the large-scale querying of commercial large language models to train competing models", which they also call distillation attacks.

OWASP calls the same thing Functional Model Replication: "Using the target model to generate synthetic training data can allow attackers to fine-tune another foundational model, creating a functional equivalent." Synthetic here just means made by a program, not collected from real people. Fine-tuning means training an already-trained model a little more.

Signs someone is copying your model

  • Huge volume

    One account sends far more requests than any normal user, often steady and around the clock.

  • Systematic questions

    The questions follow a neat pattern, such as counting up 0, 1, 2, 3, or sweeping through every topic in order, rather than the messy variety of real needs.

  • Many accounts, one shape

    Lots of fresh accounts all ask similar things in a similar rhythm, a way to dodge a per-user limit.

OWASP's guidance fits this. It recommends "comprehensive logging, monitoring and anomaly detection", described as continuously watching resource use and logging "to detect and respond to unusual patterns of resource consumption". An anomaly is something far from normal.

Quick check

Which one is the best clue that an account might be copying your model?

Guards that make copying harder

No single guard stops every thief. A good setup stacks several, so that each one makes copying slower, costlier or easier to notice. Here are the main ones, with where each comes from.

  1. 1

    Lock the files

    Apply access controls with least privilege, and keep a list of production models. This answers the file route.

  2. 2

    Rate limit and quota

    OWASP: "Apply rate limiting and user quotas to restrict the number of requests a single source entity can make in a given time period." Copying needs volume, and this caps it.

  3. 3

    Share less in each answer

    Return only what the app needs. Do not hand out raw probabilities if a plain answer will do. The Wikipedia page lists a similar idea, "information laundering", which means altering the information that attackers receive.

  4. 4

    Watch and react

    Log usage, look for unusual patterns, and decide ahead of time what you will do when one appears.

  5. 5

    Mark the output

    OWASP lists watermarking: "Implement watermarking frameworks to embed and detect unauthorized use of LLM outputs." A watermark is a hidden signature in the answers that can later help show where a copy came from.

Tip

Each guard has a cost for honest users. A very tight rate limit annoys real customers. Pick limits from how people actually use your app, then watch the logs and adjust.

What to take with you

  • Two routes

    Take the file, or copy the behaviour by asking. Locked doors stop the first. Limits, quiet answers and watching slow the second.

  • Answers are clues

    The more exact and detailed each answer is, the more it gives away. Share what the app needs.

  • Stack guards, then watch

    No guard is perfect. Layer them, keep logs, and check the logs.

For a tester, a good starter list is short. Are model files locked down? Is there a rate limit? Do the answers include extra numbers nobody needs? Would anyone notice if one account asked ten thousand patterned questions? And as always, only test what you own or have written permission to test.

Quick check

Which change makes the toy model harder to copy exactly?

Next: prompt injection, direct and indirect.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.