esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Core / 02.10

Insecure model files

Why opening a downloaded model file can run someone else's code, and the simple habits that keep you safe: trusted sources, safer formats, hash checks.

14 min readFree

A model file is not just data

The last lesson, Code injection and RCE, ended with a promise: next, insecure deserialization, when loading a model file runs code. Here it is.

Picture a box that arrives in the post. You expect a book. You open it and the box starts doing things on its own, because someone hid a little machine inside. A model file can behave like that box. You think you are loading numbers. The file is allowed to carry instructions, and the loader follows them.

This lesson is about ideas and defence. You will see the problem with a harmless demo that only prints one sentence, then learn the habits that stop it. One rule first: test only systems you own or have written permission to test. Everything here runs on your own laptop with nothing but Python's standard library.

Key words

Serialization
Turning something that lives in memory, like a trained model, into a stream of bytes you can save in a file or send over a network. Think of flat-packing a bookshelf into a box.
Deserialization
The way back: reading the bytes and rebuilding the thing. Unpacking and building the bookshelf again.
Pickle
Python's own built-in way to serialize almost any Python object. Saving is called pickling, loading is called unpickling.
Payload
The part of a file or message that does the work. In an attack file, the payload is the part that was put there on purpose.
Supply chain
Everything you take from other people to build your app: libraries, datasets, and downloaded models.
Hash
A short fingerprint worked out from a file's bytes. Change one byte and the fingerprint changes completely.
Tensor
A block of numbers arranged in a grid. A model's learned weights are stored as tensors.

Saving and loading a model

A trained model is just a lot of numbers, called weights, plus a description of how they fit together. Training is slow, so people save the model to a file once and load it back later. Many never train at all. They download a model someone else saved.

The question that matters for security is simple: when you unpack, can the box change what happens on your computer? It depends on the file format. Some formats hold only plain data. Others can hold instructions. Let us look at one of each.

A safe one: plain data only

JSON is a text format that can hold words, numbers, true or false, lists and labelled groups of those. That is all. It has no way to say "run this". When Python reads JSON, it can only build plain data out of it.

Python
import json

model = {"name": "tiny-fruit-model", "weights": [0.5, 1.25, -0.75]}

text = json.dumps(model)
print(text)

back = json.loads(text)
print(back == model)
print(type(back).__name__)

Output

{"name": "tiny-fruit-model", "weights": [0.5, 1.25, -0.75]}
True
dict

The first line is the saved text. The second line prints True: the loaded copy equals the original. The third line shows what came back, a plain dict, which is Python's name for a labelled group. Nothing else could have come back, because JSON has no way to describe anything else.

The Python docs agree. In the pickle page's warning, they say that safer formats "such as json may be more appropriate if you are processing untrusted data".

The risky one: pickle

Pickle is the opposite. It can save almost any Python object, even ones with their own special code. To do that, a pickle file is not just data. It is a short list of steps for rebuilding the object, and Python runs those steps when you load it.

An object can even choose its own rebuild steps. A method called __reduce__ lets the object say: "to rebuild me, call this function with these inputs". Normally that is helpful. The danger is that the file's author decides which function. Here is a demo. The "function" is Python's own print, and the input is a harmless sentence.

Python
import pickle


class FruitModel:
    def __init__(self):
        self.weights = [0.5, 1.25, -0.75]

    def __reduce__(self):
        # Tells pickle: "to rebuild me, call print with this message".
        return (print, ("code ran during load!",))


blob = pickle.dumps(FruitModel())
print("saved", len(blob), "bytes, nothing printed yet")

print("loading now...")
thing = pickle.loads(blob)
print("loaded:", thing)

Output

saved 61 bytes, nothing printed yet
loading now...
code ran during load!
loaded: None

Read the output slowly. Saving made 61 bytes and printed nothing. Then, during the load, before our own next line could run, the message "code ran during load!" appeared. Nobody called print on purpose after loading. The file did it.

The last line says loaded: None. That is because print gives back nothing, so the thing that came out of the file is None, not a model. Real files would hand back something that looks like a model, which is why this is hard to spot.

This demo only prints. The point is that the file chose the function, and it could have chosen another.

What the Python docs say

This is not a secret or a bug. The Python documentation puts a warning box at the top of the pickle page. Here it is, word for word:

"The pickle module is not secure. Only unpickle data you trust. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with."

Two phrases matter most. "Only unpickle data you trust" is the rule. "Or that could have been tampered with" is the part people forget: even a file from a source you trust is a risk if someone changed it on the way. The same page adds that you can consider signing data with hmac, a way to prove data was not changed.

The OWASP Deserialization Cheat Sheet, a set of notes for people who review code, says the same thing in one line about pickle.load() and pickle.loads(): "Unpickling can execute arbitrary code; never unpickle untrusted data."

Tip

"Arbitrary" means "anything the file's author picks". The last lesson covered what code running on your machine can lead to.

Why model hubs make it a supply-chain risk

If you wrote the pickle file yourself, there is no problem. The trouble starts when the file comes from someone else. And with AI models, that is the normal way to work.

Model hubs are websites where people share trained models, the way app stores share apps. They are very useful. They also mean that a file made by a stranger ends up on your machine, and your code may load it with one line.

OWASP, a group that publishes security guides, lists this under LLM03:2025 Supply Chain in its Top 10 for large language model apps. Its page says that "in ML the risks also extend to third-party pre-trained models and data", and that these outside parts "can be manipulated through tampering or poisoning attacks".

Why is a model hard to check by looking? The same page answers: "Models are binary black boxes and unlike open source, static inspection can offer little to security assurances." Binary means the file is made of raw bytes, not readable text. You cannot open it in a notepad and see what it does.

  • Strangers make the files

    Anyone can upload a model. A friendly name and a nice description prove nothing about what is inside the file.

  • Nobody can see inside

    A model file is raw bytes. Reading it by eye tells you almost nothing about whether it holds hidden steps.

  • Loading is one line

    Your code can fetch and load a model in a single step, so the dangerous moment arrives before anyone has looked at anything.

OWASP also lists "weak model provenance". Provenance means a record of where something came from and who made it. The page says: "Currently there are no strong provenance assurances in published models." Model cards, the description pages that come with models, "offer no guarantees on the origin of the model". It adds that an attacker can take over a supplier's account on a model repository, or make a lookalike one, and then use trickery on people to get the file used.

One of OWASP's example scenarios fits this lesson: an app deploys a pre-trained model from a widely used repository without thorough checking, and "a compromised model introduces malicious code". The page also mentions that model conversion services have been shown to be open to manipulation, which can slip malicious code into models.

Poisoned training data is a different problem, taught in the Data poisoning lesson. Here the file itself is the weapon.

Safer formats

The best fix is to use a format that cannot hold instructions in the first place. If the file only describes numbers, loading it has nothing to run.

Safetensors is one such format. The Hugging Face documentation introduces it as "a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy)". Remember: tensors are the grids of numbers that hold a model's weights. A format built only to store tensors has no place for "call this function".

The docs also say safetensors is used by many well-known projects, including Hugging Face's own transformers library. Using it does not need any big change: you read tensors by name. This is the docs' own loading example. We are not running it here, because it needs outside packages.

Text
# From the Safetensors docs, shown without its output (not run here)
from safetensors import safe_open

tensors = {}
with safe_open("model.safetensors", framework="pt", device=0) as f:
    for k in f.keys():
        tensors[k] = f.get_tensor(k)

Tip

A safer format helps with this one risk. It does not prove that the numbers inside are good. A model can still be badly trained or tampered with in ways that never run code. That is why the habits below still matter.

Scanning before you load

Sometimes the only file you can get is a pickle. Then you want a way to look inside without opening it the dangerous way.

A scanner is a tool that reads a file's saved steps and flags the suspicious ones, without running them. Python's standard library has a small reader for exactly this, called pickletools. Its dis function prints the steps of a pickle one per line, and never runs them. Here it is on the same demo blob as before.

Python
import pickle
import pickletools


class FruitModel:
    def __reduce__(self):
        return (print, ("code ran during load!",))


blob = pickle.dumps(FruitModel())

# dis only READS the saved steps and prints them. It never runs them.
pickletools.dis(blob)

Output

    0: \x80 PROTO      5
    2: \x95 FRAME      50
   11: \x8c SHORT_BINUNICODE 'builtins'
   21: \x94 MEMOIZE    (as 0)
   22: \x8c SHORT_BINUNICODE 'print'
   29: \x94 MEMOIZE    (as 1)
   30: \x93 STACK_GLOBAL
   31: \x94 MEMOIZE    (as 2)
   32: \x8c SHORT_BINUNICODE 'code ran during load!'
   55: \x94 MEMOIZE    (as 3)
   56: \x85 TUPLE1
   57: \x94 MEMOIZE    (as 4)
   58: R    REDUCE
   59: \x94 MEMOIZE    (as 5)
   60: .    STOP
highest protocol among opcodes = 4

You do not need to understand every line. Look at the quoted names. The steps ask for 'builtins', then 'print', then the message, then a step called REDUCE, which means "call it". Anyone reading this list can see that loading will call something. A scanner looks for that pattern: calls to functions that a plain model has no reason to use.

Scanning is useful, but it is a helper, not a shield. It only knows the patterns it was taught, and OWASP's warning about black boxes still applies. Treat a clean scan as "no alarm rang", never as "this is safe".

Three habits that protect you

You cannot check every byte. You can build habits that make the risk small. The OWASP supply chain page gives a short version: "Only use models from verifiable sources and use third-party model integrity checks with signing and file hashes to compensate for the lack of strong model provenance."

  1. 1

    Pick a source you can name

    Use models from a maker you can identify, such as your own team, a well-known publisher, or a vendor you have a contract with. Check the account name carefully, since lookalike accounts exist.

  2. 2

    Prefer a safe format

    If a model is offered in a format like safetensors, take that one over pickle. If only pickle exists, ask yourself whether you really need that model.

  3. 3

    Check the fingerprint

    Get the publisher's hash from a place you trust, work out the hash of your download, and compare. If they differ, do not load the file.

The third step is easy to try. SHA-256 is a popular way to make a hash. In the demo below, a publisher shares the fingerprint of a good file. Our checker works out the fingerprint of whatever it receives, and refuses to load a file that does not match. It checks before loading, never after.

Python
import hashlib
import pickle

good_bytes = pickle.dumps({"weights": [0.5, 1.25, -0.75]})

# The publisher tells you this fingerprint on their page.
published = hashlib.sha256(good_bytes).hexdigest()
print("published:", published[:16], "...")


def load_if_hash_matches(data, expected):
    actual = hashlib.sha256(data).hexdigest()
    if actual != expected:
        print("rejected:", actual[:16], "...")
        return None
    print("hash ok, safe to load")
    return pickle.loads(data)


print(load_if_hash_matches(good_bytes, published))

changed = bytearray(good_bytes)
changed[-3] = changed[-3] ^ 1
print(load_if_hash_matches(bytes(changed), published))

Output

published: c379fbf9df1d87c4 ...
hash ok, safe to load
{'weights': [0.5, 1.25, -0.75]}
rejected: 03adedaa7ae53e21 ...
None

Walk through it. The first line is the published fingerprint, cut short to 16 characters so it fits. The good file matches, prints hash ok, and loads. Then we flip one tiny part of one byte, a change you could never spot by eye. The fingerprint comes out completely different, 03adedaa7ae53e21 against c379fbf9df1d87c4, and the checker rejects the file and returns None. The altered file was never loaded.

Also note: a hash shows the file was not changed, not that it was safe to begin with. And if someone can change both the file and its fingerprint, the check proves nothing, so get the fingerprint from a separate, trusted place.

For testers: what to check

If you test AI apps, this risk gives you a plain checklist to look at in your own project. You are reading code and settings, not attacking anyone. Remember to work only on systems you own or have written permission to test.

  • Where do models come from?

    List every model your app loads and who published it. Anything you cannot name is a finding.

  • How are they loaded?

    Search the code for pickle.load, pickle.loads and any loader that uses pickle underneath. The OWASP cheat sheet lists these as the first places to review.

  • Is anything checked?

    Look for a hash check or a signature check before load. If a download goes straight into a loader, say so.

The Prompt injection, direct and indirect lesson and the Unauthorized access lesson cover how an attacker may reach your app from the outside. This one is about what you bring in from the outside. Both doors need a lock.

Quick check

A teammate downloads a pickle model file from a stranger's page and loads it on a work laptop. What is the main risk?

Quick check

A hash check on a downloaded file fails. What should you do?

What to remember

A model file can be a list of steps. Pickle follows them on load, so a stranger's file can run code on your machine. The habits: know the source, prefer a safe format, scan if you must use pickle, and check the hash before you load.

Next: the infrastructure around the model.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.