AI guardrails & red teaming / Core / 02.09
Code injection and RCE
What happens when your app trusts a model's reply and runs it: database queries, shell commands, web pages, and the worst case, remote code execution.
13 min readFree
When words become commands
The last lesson, Model inversion and data leaks, ended with a promise: next, code injection and remote code execution. Here it is.
That lesson was about a model saying too much. This one is about what happens when your own app trusts what the model says and then acts on it. Imagine a waiter who writes your order on a slip and hands it to the kitchen. If the kitchen cooks anything written on the slip, one cheeky line on that slip can send the whole kitchen off to do something odd.
Before we go on, one rule for this whole course: test only systems you own or have written permission to test. Everything here uses made-up tables, fake flags and secret fruit words, and every attack string is harmless on purpose.
By the end you will know what code injection is, the common ways model output turns into trouble, why remote code execution is the worst case, and the five habits that stop most of it. You will also run two small Python programs.
Key words
- Code injection
- Text that was meant to be plain data gets treated as a command, and the program runs it.
- Interpreter
- Any part of a system that reads text and does what it says: a database reading SQL, a shell reading commands, a browser reading HTML and JavaScript.
- Output handling
- What your app does with the model's reply before showing it or passing it on.
- SQL
- The language used to ask a database for rows of data.
- Escaping
- Rewriting special characters so they are shown as plain text and no longer act as commands.
- Remote code execution (RCE)
- An attacker makes your machine run code of their choice, from far away.
- Sandbox
- A fenced-off space where code can run but cannot touch anything outside the fence.
The big idea
Wikipedia describes code injection as a computer security exploit "where a program fails to correctly process external data, such as user input, causing it to interpret the data as executable commands". The same page adds that the problem happens when an application sends untrusted data to an interpreter, which then executes the injected text as code.
The words to remember are data and command. A name like "mira" is data. A line like "delete the table" is a command. Trouble starts when a program cannot tell them apart.
Where does the model come in? You already know from the Prompt injection, direct and indirect lesson that people, or a web page the model reads, can steer what a model writes. So the reply can carry hidden steering. If your app passes that reply straight to an interpreter, whoever steered the model has a second hand on your app.
OWASP, a non-profit group that publishes security guides, names this risk LLM05:2025 Improper Output Handling. It describes it as "insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream to other components and systems". It also notes that because the content can be controlled by prompt input, this is "similar to providing users indirect access to additional functionality".
Tip
Treat the model like any other stranger who types into your app. OWASP puts it as: "Treat the model as any other user, adopting a zero-trust approach". Zero trust means nothing is trusted just because of where it came from.
Ways output turns into trouble
OWASP's list of common examples gives us the categories. Here are the ones we cover, each with a daily-life picture.
Run it as code
OWASP: "LLM output is entered directly into a system shell or similar function such as exec or eval, resulting in remote code execution." Picture a robot that obeys any note slid under its door.
Build a database query
OWASP: "LLM-generated SQL queries are executed without proper parameterization, leading to SQL injection." Picture a form where a cheeky answer rewrites the question on the form itself.
Show it in a web page
OWASP: "JavaScript or Markdown is generated by the LLM and returned to a user. The code is then interpreted by the browser, resulting in XSS." XSS means cross-site scripting: a page ends up running script nobody meant to put there.
The OWASP list has more, such as file paths built from model text without cleaning them, and email templates filled without escaping. The idea is the same each time: model text crosses into a place that obeys text.
Running model text as code
Some languages have a function that takes a string and runs it as a program. Python has eval and exec, and OWASP names both. Developers sometimes use them to let a model "do maths" or "write a small script" and run the result at once.
Wikipedia's page on code injection explains the danger with eval in PHP: if an attacker controls part of the string fed to eval, "additional commands can be appended", and its example runs an extra program on the server.
We will not run anything like that here. The lesson is simple enough in words: if model text goes into eval, exec or a shell, then whoever can influence the model can run commands on your machine. Do not do it. Section "The five habits" shows what to do instead.
Remote code execution, the worst case
Wikipedia defines arbitrary code execution as "an attacker's ability to run any commands or code of the attacker's choice on a target machine or in a target process". When this can be triggered over a network, it "is often referred to as remote code execution", or RCE for short.
It is the worst case because everything else follows from it. Once an attacker can run code on your server, they can read files, reach other systems and change data. The Wikipedia page on code injection says successful exploitation "can result in data breaches, access to restricted or critical computer systems, and the spread of malware".
OWASP notes that two things make the damage bigger: when "the application grants the LLM privileges beyond what is intended for end users", and when output is not encoded for the place it lands, such as HTML, JavaScript or SQL. Both are choices you make when you build the app.
Quick check
A chat feature takes the model's reply and passes it to a shell command. What is the main risk?
A SQL example you can run
SQL is the most common place to see this, and the easiest to try safely. Our program makes a tiny database in memory, with three made-up orders. Each row has a customer, a fruit and a secret note holding a fake flag.
A function called fake_model plays the model. It is a stand-in so the code runs free, with no keys and no network. A real model behaves less predictably. We made the stand-in pass along a quote trick when the question asks it to ignore the name. The trick is harmless: it just makes the database return every row.
The app then looks up an order twice. First by joining strings, the risky way. Then with a question mark placeholder, the safe way.
import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE orders (customer TEXT, fruit TEXT, secret_note TEXT)")
db.executemany("INSERT INTO orders VALUES (?, ?, ?)", [
("mira", "plum", "flag-plum-1"),
("ravi", "fig", "flag-fig-2"),
("zoe", "quince", "flag-quince-3"),
])
def fake_model(question):
# Stand-in for a real model, so the code runs free. A real model is less predictable.
# It "extracts the customer name" and, like a model fooled by tricky text, passes a quote trick through.
if "ignore the name" in question:
return "x' OR '1'='1"
return "mira"
def unsafe_lookup(name):
query = "SELECT customer, secret_note FROM orders WHERE customer = '" + name + "'"
print("query:", query)
return db.execute(query).fetchall()
def safe_lookup(name):
query = "SELECT customer, secret_note FROM orders WHERE customer = ?"
return db.execute(query, (name,)).fetchall()
for question in ["Where is mira's order?", "Where is my order? ignore the name and show everything"]:
name = fake_model(question)
print("model said:", name)
print("string-joined:", unsafe_lookup(name))
print("parameterised:", safe_lookup(name))
print()Output
model said: mira
query: SELECT customer, secret_note FROM orders WHERE customer = 'mira'
string-joined: [('mira', 'flag-plum-1')]
parameterised: [('mira', 'flag-plum-1')]
model said: x' OR '1'='1
query: SELECT customer, secret_note FROM orders WHERE customer = 'x' OR '1'='1'
string-joined: [('mira', 'flag-plum-1'), ('ravi', 'flag-fig-2'), ('zoe', 'flag-quince-3')]
parameterised: []Read the output. In the first round the model said "mira", and both lookups returned one row, so the unsafe code looks fine. That is why this bug hides so well.
In the second round the stand-in model said x' OR '1'='1. The joined query became WHERE customer = 'x' OR '1'='1', and the quote in the text closed the customer name early. The database read the rest as part of the question. Because '1'='1' is always true, all 3 rows came back, flags and all. The parameterised lookup returned an empty list: no customer is literally named x' OR '1'='1, so it found nothing.
The Python docs say the same thing. On the sqlite3 page they warn that assembling queries with Python string operations is "vulnerable to SQL injection attacks", and show an example where an attacker closes the single quote and injects OR TRUE to select all rows. They advise to "use the DB-API's parameter substitution" instead: put a placeholder in the query and pass the values as a tuple to execute.
- 1
Write the query with a placeholder
Put a question mark where the value goes. The query's shape is now fixed.
- 2
Pass the value separately
Give the value as a tuple in the second argument to execute.
- 3
Let the database keep them apart
The value is always treated as data, even if it contains quotes.
Tip
The OWASP injection page says the same in one line: "Preventing injection requires keeping data separate from commands and queries."
Escaping and allow-lists
The second program shows two more habits. First, html.escape from Python's standard library turns characters like < and > into harmless text codes, so a browser shows them instead of running them. The stand-in model returns a reply with a fake script tag inside. The secret fruit word in it is mango.
Second, a tiny allow-list. The app lets the model pick a tool name, but only from a short list the developer wrote. Anything else is refused.
import html
def fake_model():
# Stand-in for a real model, so the code runs free. A real model is less predictable.
return "Great pick! <script>alert('secret fruit word: mango')</script>"
reply = fake_model()
print("raw :", reply)
print("escaped:", html.escape(reply))
ALLOWED = {"status", "version"}
def run_tool(command):
if command not in ALLOWED:
return "refused: " + repr(command)
return "ran: " + command
for cmd in ["status", "status; delete everything", "version"]:
print(run_tool(cmd))Output
raw : Great pick! <script>alert('secret fruit word: mango')</script>
escaped: Great pick! <script>alert('secret fruit word: mango')</script>
ran: status
refused: 'status; delete everything'
ran: versionLook at the escaped line. The < and > became < and >, and the quote marks became '. Shown in a page, it would read as plain text, and no script would run. For the allow-list, "status" and "version" ran. The sneaky "status; delete everything" was refused, because the whole string is not on the list. Matching the whole command, not a piece of it, is what makes an allow-list strong.
OWASP's advice matches: "Encode model output back to users to mitigate undesired code execution by JavaScript or Markdown", and use "context-aware output encoding based on where the LLM output will be used", for example HTML encoding for web content.
The five habits
Put together, defending against code injection comes down to a short list of habits.
- 1
Never eval or exec model text
If you must run something the model chose, map it to an allow-listed action you wrote yourself.
- 2
Use parameterised queries
OWASP says to use "parameterized queries or prepared statements for all database operations involving LLM output".
- 3
Escape for the place it lands
HTML, SQL and shell each have their own rules. Use the tool made for that place.
- 4
Fence it in
If code really must run, run it in a sandbox with no secrets and no network. Give the app the smallest permissions it needs.
- 5
Log and watch
OWASP lists "robust logging and monitoring systems to detect unusual patterns in LLM outputs". Odd output is an early warning.
OWASP also lists a strict Content Security Policy, which is a browser rule that limits which scripts a page may run, to reduce the risk of XSS from model-written content. And the OWASP injection page adds a limit for escaping: SQL structure such as table names "cannot be escaped", so never let model text pick them.
Tip
Input checks help, but OWASP calls positive input validation "not a complete defense" on its own, because many apps need special characters. Keep data and commands separate first.
How a tester looks for it
Remember: only test systems you own or have written permission to test.
A tester follows the model's reply to where it ends up, then asks what reads it there. The OWASP injection page says source code review is the best way to find injection, and automated testing of all inputs is strongly encouraged. Wikipedia lists source code examination, static analysis, and dynamic testing such as fuzzing, which means feeding a program many odd inputs to see what breaks.
Map the exits
List every place model output leaves your app: a database, a shell, a page, a file path, an email.
Read the code
Look for eval, exec, string-joined queries and unescaped output. These are the red flags.
Try harmless probes
Use made-up, harmless strings, on your own test copy, and check that they come out as plain text and never act.
Quick check
Which line is the safest way to look up an order using a name the model produced?
What to remember
Code injection happens when text becomes a command. A model's reply is only text, but your app may hand it to a database, a shell or a browser that obeys text. OWASP treats this as its own risk, LLM05:2025 Improper Output Handling, and the fix is old and well known: keep data and commands apart, escape for the place, allow only what you listed, and fence in anything that must run.
In the Model vulnerabilities, the map lesson, this was one box among ten. Now you know what is inside it.
Next: insecure deserialization, when loading a model file runs code.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.