esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Core / 02.11

Infrastructure security

The servers, keys, logs and settings around an AI model. Learn the plain habits that keep them closed, and run a small program that checks a config.

14 min readFree

The house around the model

The last lesson, Insecure model files, ended with a promise: next, the infrastructure around the model. Here it is.

Think of a model as a very clever guest in a house. You can teach the guest good manners, which is what guardrails do. But if the front door is open, the spare key sits under the doormat and the filing cabinet is unlocked, good manners will not help. Infrastructure security is about the house: the doors, the keys, the walls and the cabinets.

A short rule before we start: test only systems you own or have written permission to test. Everything below is about checking and fixing your own setup. The examples use made-up companies and made-up keys.

Key words

Infrastructure
The machines, networks, settings and storage that your app runs on. Everything except the app's own idea.
Secret
A value that proves who you are or lets you in: a password, an API key or a token. An API key is like a long password that one program uses to talk to another.
Network
The wires and rules that let computers talk to each other, inside a company and across the internet.
Server
A computer that waits for requests and answers them. A model server is one that runs a model for other programs.
TLS
A way of scrambling traffic between two computers so nobody in the middle can read it. It is the lock behind the padlock in a browser.
Patch
A fix from the maker of a piece of software for a known problem. Patching means installing those fixes.

By the end you will know the eight habits that keep an AI setup closed, and you will run a tiny program that reads a config and prints what looks risky.

Many small locks beat one big lock

No single habit is enough. People who protect buildings use a door, a lock, a camera and a guard, because any one of them can fail. Security people call this defense in depth. The NIST glossary defines it as an "information security strategy integrating people, technology, and operations capabilities to establish variable barriers across multiple layers and missions of the organization".

In plain words: build several barriers, so one mistake does not open everything. The rest of this lesson is a list of barriers.

  1. 1

    Hide the keys

    Secrets live in a safe place, not in shared files.

  2. 2

    Close the doors

    Only machines that need the model server can reach it.

  3. 3

    Limit each account

    Every part gets only the access it needs.

  4. 4

    Fix and check

    Patch, scan, store less, keep backups.

Secrets: no keys in code or prompts

A secret that sits in your code is like a house key taped to the front door. Code is copied, emailed, pushed to shared folders and pasted into chats. Wherever the code goes, the key goes too.

The OWASP Secrets Management Cheat Sheet notes that many organizations "have them hardcoded within the source code in plaintext, littered throughout configuration files and configuration management tools". Hardcoded means typed straight into the code. Plaintext means anyone who opens the file can read it.

The same guide asks for the opposite: a central, controlled place to store, hand out, check and replace secrets. A program asks that place for the key when it needs it, and the key never sits in the project folder.

Why prompts are a bad place for keys

It can be tempting to write a key inside a system prompt, for example "use this key to call the tool". Do not. A prompt is text the model can read, repeat and sometimes be tricked into showing. The Prompt injection, direct and indirect lesson explains how that trick works. If the key is not in the prompt, there is nothing to leak.

Keys get old: rotate and expire

A key that never changes is a key that keeps working for a thief. The OWASP guide says: "You should regularly rotate secrets so that any stolen credentials will only work for a short time." To rotate a secret means to swap it for a new one on a schedule.

It also says to "create secrets to expire after a defined time where possible", and that secrets you no longer need, or that may have leaked, must be revoked. Revoking means switching a key off so it stops working.

Tip

Found a real key in a shared file? Treat it as leaked. Switch it off and make a new one. Deleting the line is not enough, because old copies of the file may still exist.

Network isolation: keep model servers closed

A model server answers anyone who can reach it. So the first question is: who can reach it? Think of a flat on an upper floor: it should not have its own door on the street.

An address of 0.0.0.0 tells a server to listen on every network address it has, including ones that face the outside world. It is fine for a quick test on a laptop and risky on a real machine. Safer: listen only on the private address your own app uses.

GPU and model servers exposed by mistake

A GPU is the powerful chip that runs models. GPU machines are costly, and teams often set them up in a hurry with a debug flag on. A debug flag is a setting that shows extra detail to help the builder, and that detail also helps strangers.

What can go wrong, as an idea: someone finds the open server and uses your paid power for their own jobs, reads what your users send in, or tampers with the model. Preventing this takes a closed door and a short list of who may enter.

  • Private by default

    A new model server should be reachable only from inside your own network, not from the internet.

  • Allow-list callers

    Name the few machines that may call it. Everyone else is refused.

  • Check from outside

    On your own systems, look at them the way a stranger would. Can you reach the model server without logging in? If yes, that is a finding.

The Protecting AI APIs lesson covers what to do at the front door of an API. This lesson is about not leaving extra doors at the back.

TLS: scramble the traffic

Even a closed network can have an unwanted listener. TLS scrambles traffic so a listener sees nonsense. The OWASP guide is blunt about secrets: "Never transmit secrets via plaintext. In this day and age, there is no excuse given the ubiquitous adoption of TLS."

The same goes for prompts and answers, because they often hold names, plans and private questions. A setting that says TLS is off is a finding, even inside a private network.

Least privilege: only what the job needs

A cleaner gets a key to the office, not to the safe. That is least privilege. Wikipedia describes the principle of least privilege as requiring that every module, such as "a process, a user, or a program", "must be able to access only the information and resources that are necessary for its legitimate purpose".

A service account is a login made for a program, not a person. Give each one a narrow job. The account that reads documents for your chatbot should not be able to delete them. The account that logs usage should not be able to read customer data.

The OWASP guide applies the same idea to people: "engineers should not have access to all secrets in the secrets management system, and the Least Privilege principle should be applied". Fewer people with a key means fewer places a key can leak from.

Tip

When you add a new account, start with nothing and add one permission at a time as something breaks. It is easier to open a door later than to find out who has been through it.

Patching and scanning: fix known holes

Your AI app is built from many parts made by others: the operating system, web tools, libraries and model code. Each part has a maker who finds and fixes holes. A fix does nothing until you install it.

OWASP's LLM03:2025 Supply Chain page lists "outdated or deprecated components, which attackers can exploit to compromise LLM applications" as a risk. Its advice is to apply the mitigations for vulnerable and outdated components, which "includes vulnerability scanning, management, and patching components". It also says to "implement a patching policy".

A scanner is a program that compares what you use against lists of known problems. The same page suggests keeping a list of every component you use, called a Software Bill of Materials, or SBOM. Think of the ingredients list on food. If a bad ingredient is reported, you can check at once whether it is in your dish.

Containers are a handy way to package an app with everything it needs, like a lunchbox. A lunchbox can hold old food, so scan the container images too, and rebuild them when their parts are fixed.

Logs and prompts are personal data

Teams keep logs to fix bugs. With AI apps, a log line often holds the whole prompt, and people type private things into prompts: names, emails, health worries, work plans. That makes your log folder a store of personal data, even if you never meant it to be.

  • Store less

    Ask whether you need the full prompt. Often a length and a timestamp are enough.

  • Hide what you store

    Replace emails, names and keys with labels before a line is written.

  • Delete on time

    Pick a short keeping period and remove old logs on schedule.

Logs can also leak secrets by accident. The OWASP guide warns about secrets ending up in logs, and lists attempts to "dump secrets to logs" as something to alert on. Below is a small program that hides emails and keys in a log line. The email, the name and the key are all made up.

Python
import re

# A made-up prompt log line. The name, email and key are all invented.
line = "user=mina@example.test asked: email my plan, key sk-banana-4f9a8c7d2e1b"

PATTERNS = [
    (re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "[email]"),
    (re.compile(r"sk-[A-Za-z0-9-]{8,}"), "[secret]"),
]


def scrub(text):
    for pattern, label in PATTERNS:
        text = pattern.sub(label, text)
    return text


print("before:", line)
print("after: ", scrub(line))
assert "@" not in scrub(line) and "sk-" not in scrub(line)

Output

before: user=mina@example.test asked: email my plan, key sk-banana-4f9a8c7d2e1b
after:  user=[email] asked: email my plan, key [secret]

The first line is the log line as it arrived. The second is what we would store. The email became [email] and the key became [secret].

Real personal data comes in many shapes, so two patterns catch only some of it. Treat this as a first filter. Storing less is still the stronger move.

Backups: a way back

Things go wrong without any attacker: a deleted folder, a broken update, a full disk. And sometimes an attacker is involved. A backup is a spare copy kept somewhere else, so you can restore what was lost.

Three habits make backups useful. Keep the copy away from the original, so one problem does not hit both. Protect the backup as carefully as the original, because it holds the same data, secrets and prompts. And try a restore now and then, because a backup you never tested is only a hope.

Worked example: a config auditor

Now let us check these habits with a program. It reads a made-up config, which is a dictionary of settings, and a made-up .env text. A .env file is a plain text list of NAME=value settings that apps read at start-up.

The program looks for four kinds of trouble: secret-looking values written in the file, a bind address of 0.0.0.0, debug flags that are on, and TLS switched off. It also notes stored prompts and long log keeping. It prints each finding with a severity: HIGH, MEDIUM or LOW. Severity is just how worried you should be. This is a plain checker, not a real security product, and it only knows the few rules we gave it.

Python
import re

# A made-up config for a made-up company. Nothing here is real.
config = {
    "app": {"name": "fruit-helper", "debug": True},
    "model_server": {"bind": "0.0.0.0", "port": 8000, "tls": False},
    "database": {"host": "10.0.0.5", "password": "mango-secret-123"},
    "logging": {"store_prompts": True, "retention_days": 365},
}

# A made-up .env text.
env_text = """
MODEL_API_KEY=sk-banana-4f9a8c7d2e1b
DB_PASSWORD=papaya-pass-777
REGION=north-1
DEBUG=true
BIND_ADDRESS=0.0.0.0
"""

SECRET_WORDS = ("key", "password", "secret", "token")
RANK = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
findings = []


def add(severity, where, message):
    findings.append((severity, where, message))


def walk(node, path=""):
    for name, value in node.items():
        here = f"{path}.{name}" if path else name
        if isinstance(value, dict):
            walk(value, here)
        else:
            check(here, name, value)


def check(where, name, value):
    low = name.lower()
    if any(w in low for w in SECRET_WORDS) and isinstance(value, str) and value:
        add("HIGH", where, "secret value written in the file")
    if "bind" in low and value == "0.0.0.0":
        add("HIGH", where, "listens on every network address")
    if low == "debug" and str(value).lower() == "true":
        add("MEDIUM", where, "debug mode is on")
    if low == "tls" and value is False:
        add("HIGH", where, "no TLS, traffic is not encrypted")
    if low == "store_prompts" and value is True:
        add("MEDIUM", where, "prompts are stored; they may hold personal data")
    if low == "retention_days" and isinstance(value, int) and value > 90:
        add("LOW", where, f"logs kept {value} days; keep only what you need")


walk(config)
for line in env_text.strip().splitlines():
    name, _, value = line.partition("=")
    check(f".env {name}", name, value)

for severity, where, message in sorted(findings, key=lambda f: RANK[f[0]]):
    print(f"{severity:6} {where}: {message}")
print(f"{len(findings)} findings")

Output

HIGH   model_server.bind: listens on every network address
HIGH   model_server.tls: no TLS, traffic is not encrypted
HIGH   database.password: secret value written in the file
HIGH   .env MODEL_API_KEY: secret value written in the file
HIGH   .env DB_PASSWORD: secret value written in the file
HIGH   .env BIND_ADDRESS: listens on every network address
MEDIUM app.debug: debug mode is on
MEDIUM logging.store_prompts: prompts are stored; they may hold personal data
MEDIUM .env DEBUG: debug mode is on
LOW    logging.retention_days: logs kept 365 days; keep only what you need
10 findings

Read the output from the top. Six HIGH findings come first. The model server listens on every address and has no TLS. The database password and two entries in the .env text are written out in the open. The .env bind address is open too. Then three MEDIUM findings: two debug flags and stored prompts. Last is one LOW finding: logs kept for 365 days. The last line says 10 findings.

The program only looks at names and simple values. A key stored under the name "magic" would slip through, and a harmless value named "token" would be flagged. Real scanners add many more rules. Still, the idea holds: the checks are rules, and rules can be run every time the config changes.

Tip

Print findings with their location, not their value. A scanner that prints the secrets it finds would just copy them into one more log.

For testers: a quick checklist

If you test AI apps, this gives you a plain list to check on your own project or one you have written permission to test. You are reading settings, not attacking anyone.

  • Where are the keys?

    Search code, config and prompts for secret-looking values. Each hit is a finding.

  • What can be reached?

    List every model server and who can reach it. Anything open to the whole internet needs a reason.

  • What do the logs hold?

    Read a few log lines. If you see full prompts, names or keys, say so.

Quick check

A model server is set to listen on 0.0.0.0 on a live machine. What does that mean?

Quick check

You find a real API key inside a shared file. What is the best first step?

Quick check

What does least privilege mean for a service account?

What to remember

Good manners in a model are not enough if the house is open. Keep secrets out of code and prompts, rotate them, close the network and use TLS, give each account only what it needs, patch and scan, store less in logs, and keep backups you have tried restoring. Several small locks together are the plan.

Next: black, white and grey box testing.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.