esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI guardrails & red teaming / Beginner / 01.05

Generative models and LLMs

What a generative model is, how a chat model guesses one word at a time, and why that makes it easy to steer, easy to fool and hard to test.

13 min readFree

From learning to generating

The last lesson, How models learn, for attackers, ended with a promise: next, generative models and large language models. Here it is.

Last time we saw that a model learns from examples, so whoever controls the examples can steer it. Now we meet the kind of model you will test most in this course: the one that writes. It answers your chat message, drafts the email, and fills in the code. It is also the kind that people try to trick with plain words.

One rule before we go further. Test only systems you own or have written permission to test. Everything in this lesson runs on your own computer, and every example text is made up.

By the end you will know what "generative" means, how a chat model picks its words, why a system prompt and a user message end up in one stream, why models make things up, and why the same test can give two answers. You will also run a toy word-guesser and watch one added sentence change what it says.

Key words

Generative model
A model that makes new content (text, pictures, sound, code) instead of only sorting or scoring things.
Large language model (LLM)
A generative model trained on a huge amount of text. It is the engine inside most chatbots.
Token
One small piece of text that a model reads or writes. Often a word, or part of one.
Prompt
The text you give a model to start from.
System prompt
Instructions from the app's builder, placed in front of the user's message. For example: "You are a polite shop assistant."
Hallucination
An answer that sounds sure but is false or made up.
Non-deterministic
Able to give a different result each time, even for the same input.

What "generative" means

Think of two kinds of helpers. One is a mail sorter. It looks at each letter and puts it in a pile: bills, ads, family. The other is a ghostwriter. You give a few notes, and it writes a whole new letter.

The Machine Learning Foundations lessons mostly met the sorter. A generative model is the ghostwriter. Wikipedia describes generative AI as a subfield of AI that uses generative models to produce text, images, videos, audio, computer code or other forms of digital data.

It explains how that works in one sentence. These models learn the patterns and structures of their training data, then use them to generate new data in response to input, which often takes the form of natural language prompts.

  • It learns patterns

    It does not look things up in a book. It repeats the shape of what it saw in training.

  • It answers a prompt

    What it writes depends on the text you hand it. Change the text and the answer changes.

  • It makes new output

    Every answer is built on the spot, so there is no fixed list of answers you can check in advance.

Large language models

A large language model is the generative model for text. The Wikipedia page on large language models defines it as a machine learning model, typically a neural network, trained on a vast amount of text for language tasks, especially language generation.

The same page names one common kind, the generative pre-trained transformer, or GPT. It says GPTs are pre-trained to predict the next word, and are then often fine-tuned to follow instructions and to behave as assistants.

The How LLMs Work lesson in AI engineering explains next-token prediction, and the Tokenization lesson in Machine Learning Foundations explains tokens. Here we only ask what the guessing means for someone who wants to break or protect an app.

Say that again slowly, because the whole lesson hangs on it. Underneath the friendly assistant is a machine that does one thing: it guesses what comes next. The assistant manners are added on top by more training.

A guess, one word at a time

Picture the game where you finish a friend's sentence. "Peanut butter and..." You say "jelly" without thinking, because you have heard it so often. A model plays that game all day. It looks at the text so far, scores every possible next token, picks one, adds it to the text, and goes again.

  1. 1

    Read

    The model takes all the text so far: your prompt plus anything it has already written.

  2. 2

    Score

    It gives every token it knows a score for how well it fits next.

  3. 3

    Pick

    It chooses one token. Sometimes always the top score, sometimes a random pick weighted by the scores.

  4. 4

    Repeat

    The new token joins the text, and the loop starts again until the answer ends.

A toy word-guesser

This program is a toy. It is not an LLM and is nowhere near one. It reads a made-up text of three short sentences and counts, for each word, which word came right after it. That table of counts is called a bigram model: a bigram is just a pair of neighbouring words.

Then it writes by always taking the most common next word. Choosing the top score every time is called greedy output. If two words tie, the one it saw first wins. It runs free, with no keys and no network.

The code runs twice. The first time it learns from the three sentences. The second time it learns from the same three plus one extra sentence, "The cat ate the mouse." For each model it prints what follows the word "the", then the greedy sentence of seven words, starting from "the".

Python
from collections import Counter, defaultdict

def train(text):
    words = text.lower().replace(".", " .").split()
    counts = defaultdict(Counter)
    for a, b in zip(words, words[1:]):
        counts[a][b] += 1
    return counts

def greedy(counts, start, n):
    out = [start]
    for _ in range(n):
        options = counts[out[-1]]
        if not options:
            break
        out.append(options.most_common(1)[0][0])
    return " ".join(out)

story = "The cat sat on the mat. The cat ate the fish. The dog sat on the rug."
model = train(story)
print(dict(model["the"]))
print(greedy(model, "the", 6))

model2 = train(story + " The cat ate the mouse.")
print(dict(model2["the"]))
print(dict(model2["cat"]))
print(greedy(model2, "the", 6))

Output

{'cat': 2, 'mat': 1, 'fish': 1, 'dog': 1, 'rug': 1}
the cat sat on the cat sat
{'cat': 3, 'mat': 1, 'fish': 1, 'dog': 1, 'rug': 1, 'mouse': 1}
{'sat': 1, 'ate': 2}
the cat ate the cat ate the

Read the first two lines. In the three sentences, the word "the" is followed by "cat" twice and by "mat", "fish", "dog" and "rug" once each. Greedy picks "cat". After "cat" it picks "sat" (the first of a tie), and so on, which gives "the cat sat on the cat sat". It is a bit silly, because a toy has no idea what a sentence is. It only follows counts.

Now the extra sentence. One new line of text changed three things. "the" is now followed by "cat" three times. After "cat", "ate" now beats "sat" two to one. So the greedy output flipped to "the cat ate the cat ate the".

That is the security lesson in miniature. The model has no opinion of its own. It repeats what its text taught it, and one added sentence moved the answer. A real model learns from a huge pile of text, so one sentence rarely moves it that far. If an attacker could slip many chosen sentences into the training pile, the move could be real.

Tip

Real models are not this simple and not this predictable. They look at far more than the last word, and many answers are possible at once. The toy only shows the shape of the idea: counts in, next word out.

Why words can steer it

A normal program does what its code says. A model does what its text suggests. Whatever you write becomes part of "the text so far", and the next guess depends on it.

That is why a friendly instruction at the top, like "answer like a pirate", changes the whole style. The pirate-sounding words now make pirate-sounding continuations more likely. It is also why a request that sits in the middle of your input can change what comes next.

OWASP, the group that keeps a widely used list of risks for AI apps, puts it this way for its first risk: a prompt injection vulnerability occurs when user prompts alter the model's behaviour or output in unintended ways. The Prompt injection attacks lesson in AI engineering goes deeper.

One stream for everything

Here is the detail that surprises new builders. In your app, the system prompt and the user's message feel like two different things. One is the rule book, the other is the customer. But by the time the model reads them, they have been joined into a single run of tokens.

The stand-in below shows the idea. It is a toy, written so the code runs free. It follows one trick: it obeys the last line that starts with "say:". Real models are far less tidy and less predictable, but they share the flat-stream problem. The company and the secret word are made up.

Python
# Stand-in, not a real model: it only knows one trick, obey the LAST line that starts with "say:".
# Real models are far less predictable, but they also read one flat stream.
def toy_model(stream):
    said = [line[4:].strip() for line in stream.splitlines() if line.startswith("say:")]
    return said[-1] if said else "..."

system = "say: Welcome to Fruit Co support."
user = "say: The secret fruit word is PEAR."

stream = system + "\n" + user
print(repr(stream))
print(toy_model(stream))

Output

'say: Welcome to Fruit Co support.\nsay: The secret fruit word is PEAR.'
The secret fruit word is PEAR.

The stand-in could not tell that the first line came from the builder and the second from a visitor, so the later line won. A real model has no built-in label saying "trusted" or "untrusted" on each token either. This is the root of a whole family of attacks. Defenders cannot assume the model will keep the two apart. They have to add checks outside the model: limit what the model is allowed to do, filter what goes in and out, and never put a real secret in the prompt.

When it makes things up

Because the model is guessing, it can guess wrong and still sound sure. Wikipedia defines a hallucination as a response generated by AI that contains false or misleading information presented as fact. It gives fabricated citations as an example: plausible-sounding falsehoods inside an otherwise normal answer.

The same page explains why. Pre-training involves predicting the next word, which, in its words, incentivizes the model to "give a guess" even when it lacks information. It also says a hallucination can be a statistically plausible continuation that is not supported by the input, the training data or outside information.

OWASP lists misinformation as a risk of its own: misinformation from LLMs poses a core vulnerability for applications relying on these models. If an app trusts a made-up answer, someone gets hurt or a wrong action is taken.

  • Trusting the answer

    A person or a program acts on a false statement because it was written confidently.

  • Made-up references

    Names, links, case numbers or sources that do not exist, but look right.

  • No warning light

    A wrong answer and a right answer look the same on screen. Nothing flashes red.

Wikipedia lists the ways used to lower the risk: retrieval augmented generation, citation checking, constrained output formats and human review. It adds that none of them guarantees accuracy in every case. So a tester's job is to find where the app trusts the model too much, not to hope the model never errs.

Quick check

Why does a language model sometimes state something false in a confident voice?

Same question, different answer

There is one more habit that matters to testers. A normal function gives the same output for the same input, so a test passes or fails. Many AI apps do not work that way.

Step 3 of the guessing loop may pick a random token, weighted by the scores. Settings such as temperature control how adventurous the pick is; the Sampling Parameters lesson in AI engineering covers them.

The stand-in below is a toy again, with made-up chances for the word after "the". It picks randomly, weighted by those chances: "cat" counts three times, the others once. The random picker gets a seed, a starting number. We run seeds 0 to 7 twice, then seeds 8 to 15 once.

Python
import random

# Stand-in, not a real model: the made-up chances a toy model gives to the word after "the".
chances = {"cat": 3, "mat": 1, "fish": 1, "dog": 1, "rug": 1}

def sample(seed):
    rng = random.Random(seed)
    words, weights = list(chances), list(chances.values())
    return rng.choices(words, weights)[0]

print([sample(seed) for seed in range(8)])
print([sample(seed) for seed in range(8)])
print([sample(seed) for seed in range(8, 16)])

Output

['dog', 'cat', 'rug', 'cat', 'cat', 'fish', 'dog', 'cat']
['dog', 'cat', 'rug', 'cat', 'cat', 'fish', 'dog', 'cat']
['cat', 'mat', 'mat', 'mat', 'mat', 'cat', 'cat', 'rug']

The first two lists are identical: the same seeds give the same picks. The third list, from other seeds, differs. Both runs use the same chances, yet they say different things. A real chat model usually has no fixed seed you can see, so you meet the third case: a different answer each time.

This makes testing harder in a specific way. A bad answer may appear one time in twenty. If you run an attack prompt once and it fails, you have not shown the app is safe. You have shown it was safe once. Careful testers repeat each case many times, record the results, and describe the odds, not a single pass or fail.

What to carry forward

  1. 1

    Generative

    The model writes new content from patterns it learned, answering a text prompt.

  2. 2

    Next-word guesser

    An LLM is a next-token guesser with assistant manners added by extra training.

  3. 3

    Steerable by words

    Whatever text reaches it shapes what it writes next, for builders and attackers alike.

  4. 4

    One flat stream

    System prompt, user text and fetched content are joined, so the model sees no built-in trust labels.

  5. 5

    Fluent, not always right

    Hallucinations look like facts, so apps must check what matters.

  6. 6

    Different each time

    Random picks mean one test run is not proof. Repeat tests and record settings.

Quick check

In the toy word-guesser, what changed when one more sentence was added to its text?

Quick check

Your app joins a system prompt and the user's message into one text for the model. What is the risk?

Next: threat modeling, drawing the map before you attack.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.