esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.03

How LLMs Work

A large language model guesses the next small piece of text, over and over. See how it learns, what you pay for, and what that means for your product.

12 min readFree

One word at a time

The last lesson, AI vs AGI, ended with a promise: next, how large language models work, one word at a time. Here it is, and "one word at a time" is close to the whole secret.

Chat tools feel like magic. You type a question and a full answer appears. Under the hood, the model does one small thing, and it does it again and again. If you understand that one thing, you will understand why these tools are so useful, and why they sometimes get things wrong.

You do not need a maths or machine learning background. You will read a little Python, and you will run two tiny programs that cost nothing and need no internet.

Key words

Large language model (LLM)
A computer program that has learned from a huge amount of text, and can write, summarise, translate and answer in everyday language.
Token
One small piece of text the model works with. It is often a word or part of a word.
Next-token prediction
Guessing which piece of text should come next, given everything so far.
Pre-training
The first, very long stage of learning, where the model reads a huge amount of text.
Fine-tuning
More training afterwards, to teach the model a specific job or style.
Training
The learning stage. It happens once, before you ever use the model.
Inference
Using a trained model to get an answer. It happens every time someone asks.
Pre-trained model
A model someone else already trained, ready for you to reuse.

What a large language model is

Wikipedia defines it like this: a large language model is "a machine learning model (typically a neural network) trained on a vast amount of text for natural language processing tasks, especially language generation". It adds that LLMs can typically generate, summarise, translate and analyse text.

Let's unpack that. A neural network is a program made of many small number-crunching parts that learn patterns from examples. "Language generation" just means writing text. "Large" refers to the size of the network and the amount of text it learned from.

Think of a friend who has read an enormous library but is not allowed to look anything up any more. Ask them a question and they answer from memory, in fluent sentences. That is a fair picture of an LLM.

The one trick: guess the next piece

Here is the core idea. The model reads the text so far and guesses what comes next. Then it adds that guess to the text and guesses again. It repeats until the answer is done.

Your phone's keyboard does a tiny version of this when it suggests the next word. An LLM does the same job, but it has learned from far more text and looks at far more of the sentence.

Wikipedia says that generative pre-trained transformers (GPTs), a type of LLM, are "pre-trained to predict the next word". Its page on GPTs gives a feel for the guess: a model learns whether "I like to eat" is more likely to be followed by "bread" or "rocks".

  1. 1

    Read

    The model takes in all the text so far: your question, plus any words it has already written.

  2. 2

    Score

    It gives every possible next token a chance, like a long list of "how likely is this one?" marks.

  3. 3

    Pick

    One token is chosen from that list. Which one depends on a choice we look at below.

  4. 4

    Add

    The chosen token joins the text. Now the text is one token longer.

  5. 5

    Repeat

    Go back to step 1, until the answer ends.

The pieces are called tokens. The next lesson covers them. For now, think "word or part of a word". The Tokenization lesson in Machine Learning Foundations goes deeper on how text is cut up.

A tiny model you can run

A real LLM is huge. But the idea fits in a few lines. Our tiny model has a small made-up text. It counts which word follows which. That table of counts is the whole "model". It is called a bigram table, because it looks at pairs of words.

It then writes text two ways. Greedy always picks the most common next word. Weighted random picks any next word, but common ones win more often. A fixed seed makes the random picks repeat the same way every run.

Python
import random
from collections import Counter, defaultdict

text = "the cat sat on the mat . the cat ate the fish . the dog sat on the rug . the dog ate the bone ."
words = text.split()

table = defaultdict(Counter)
for a, b in zip(words, words[1:]):
    table[a][b] += 1

print("after 'the':", dict(table["the"]))
print("after 'sat':", dict(table["sat"]))

def greedy(start, n):
    out = [start]
    for _ in range(n):
        nxt = table[out[-1]].most_common(1)[0][0]
        out.append(nxt)
    return " ".join(out)

def weighted(start, n, seed):
    rng = random.Random(seed)
    out = [start]
    for _ in range(n):
        options = table[out[-1]]
        out.append(rng.choices(list(options), weights=list(options.values()))[0])
    return " ".join(out)

print("greedy  :", greedy("the", 8))
print("random 1:", weighted("the", 8, 1))
print("random 2:", weighted("the", 8, 2))

Output

after 'the': {'cat': 2, 'mat': 1, 'fish': 1, 'dog': 2, 'rug': 1, 'bone': 1}
after 'sat': {'on': 2}
greedy  : the cat sat on the cat sat on the
random 1: the cat ate the mat . the dog ate
random 2: the bone . the cat ate the dog sat

Read the output. After "the", the table shows "cat" and "dog" twice each, and four other words once. After "sat", only "on" ever followed, twice.

Greedy wrote "the cat sat on the cat sat on the". It picked "cat" after "the" because "cat" and "dog" tied and "cat" came first in the text. Then it got stuck in a loop, because it always takes the top choice. That is a real habit of simple models: always taking the top pick can repeat itself.

The two random runs wrote different text, even though the table is the same. Run 1 gave "the cat ate the mat . the dog ate", which reads quite well. Run 2 gave "the bone . the cat ate the dog sat", which does not. Same model, different luck.

Tip

A real LLM does this same loop, but its "table" is a huge network, it looks at much more than the last word, and it scores every token it knows. Choosing between greedy and random is a setting you will meet later, in the lessons on Temperature and Sampling Parameters.

The design inside: the transformer

Inside most LLMs sits a design called the transformer. It comes from a 2017 paper whose abstract proposes "a new simple network architecture, the Transformer, based solely on attention mechanisms". Wikipedia says LLMs are typically based on this design.

You do not need its details to build with a model. If you are curious, the Transformers and Self-Attention lessons in Machine Learning Foundations go deeper.

Training: how the model learns

A model does not start out knowing anything. Training is where it learns. Training has two big stages.

  • Pre-training

    The model reads a huge amount of text and practises the guessing game. Wikipedia describes the idea as training on a large, unlabeled dataset first. No one has to label the text, because the text itself provides the answers: hide the next word, and the right answer is right there.

  • Fine-tuning

    Wikipedia describes fine-tuning as adapting a model trained for one task to perform a different, usually more specific, task. The model gets more training, reusing what it already learned as the starting point.

  • Instruction tuning

    A kind of fine-tuning that teaches the model to follow what people ask. Wikipedia notes that before fine-tuning, most LLMs are next-token predictors.

Why the second stage? A model that has only practised guessing the next word is good at continuing text. It is not yet a helpful assistant.

Teaching it to follow instructions

A famous example is a 2022 paper from OpenAI. Its abstract says that making language models bigger "does not inherently make them better at following a user's intent", and that large models can produce answers that are untruthful, toxic or simply not helpful.

The fix was fine-tuning with human feedback. The abstract describes two steps. First, people wrote examples of the behaviour they wanted, and the team fine-tuned GPT-3 on them. Then people ranked model answers, and the model was tuned again using reinforcement learning from human feedback.

The result is striking. In human evaluations, outputs from the 1.3 billion parameter model were preferred to outputs from the 175 billion parameter GPT-3, "despite having 100x fewer parameters". A parameter is one of the many numbers inside a model that training adjusts.

For a builder, the point is simple. The chat feel of today's assistants owes a lot to this second stage, not to size alone.

Quick check

What does pre-training teach a model?

Training happens once, inference every time

These two words matter for your budget and your plans, so keep them apart.

Training is the learning stage. It runs once, takes a huge amount of computer power, and is done by the company or group that builds the model. You, as a builder, usually never do it.

Inference is using the finished model. Wikipedia defines it as the process of providing input to a trained large language model and receiving generated output. Every time someone presses send, that is one inference. It is what you pay for when you use a hosted model, usually counted in tokens.

Inference: the part you pay for

Wikipedia gives a feel for the gap in one example. It says training costs about 6 floating-point operations (FLOPs, tiny sums the computer does) per parameter for each token, while inference costs only 1 to 2 FLOPs per parameter per token. So one answer is cheap next to training. But it happens for every user, every day.

Let's do the maths with made-up numbers. Pretend one answer costs 0.002 of some currency, which is a made-up price. If 1,000 people each ask 10 questions a day, that is 10,000 answers. At the made-up price that is 20 a day. Small per answer, and it adds up as your product grows.

Wikipedia also says hosted inference runs a model on remote data centres and makes the output available over the Internet. Some open-weight models can instead be downloaded and run on a personal computer.

Pre-trained models: reuse, don't rebuild

Because training is so big, almost nobody starts from zero. You pick a model someone already trained and reuse it. That is a pre-trained model.

Wikipedia's page on fine-tuning says models pre-trained on large, general text are usually fine-tuned by reusing their parameters as a starting point. It also notes that fine-tuning the whole model is common and often gives better results, but costs more computing.

Wikipedia adds that model names often show the training stage. A model straight out of pre-training is marked "Base". One tuned to follow instructions is marked "Instruct" or "Chat". When you browse models, check which one you are picking.

What this means when you build

Now the useful part. Because a model only guesses the next token, three things follow.

  • Sure-sounding guesses

    Wikipedia says hallucinations are fluent text that appears factually sound, but is factually incorrect. The model is built to sound right, not to check. Plan to verify anything important.

  • No lookup

    A model writes from what it learned in training and from the text you send. It does not go and check a source unless your product gives it the facts.

  • No memory of past chats

    Each request is a fresh start. If you want the model to know what was said before, your product must send that text again.

That last point is easy to see in code. The next program uses a stand-in, a tiny fake model written inside the code so it runs free. A chat is a list of messages, and the stand-in only knows what is in the list you give it.

Python
def fake_model(messages):
    # Stand-in: it only "knows" what is inside the messages it is given.
    seen = " ".join(m["content"] for m in messages)
    if "Venkat" in seen:
        return "Your name is Venkat."
    return "I do not know your name."

chat = [{"role": "user", "content": "My name is Venkat."}]
print(fake_model(chat))

new_chat = [{"role": "user", "content": "What is my name?"}]
print(fake_model(new_chat))

full = chat + [{"role": "assistant", "content": "Nice to meet you."}] + new_chat
print(fake_model(full))
print("messages sent:", len(new_chat), "then", len(full))

Output

Your name is Venkat.
I do not know your name.
Your name is Venkat.
messages sent: 1 then 3

Remember, this is a stand-in so the code runs free. It is not a real model. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse.

Read the output. The first call saw the name, so it answered. The second call sent only the new question, and the stand-in knew nothing. The third call sent the whole conversation, and the answer came back. Notice the last line: sending history means sending more messages, 3 instead of 1, and more tokens means more cost.

Tip

Chat apps feel like they remember you because the app keeps your earlier messages and sends them along with each new question. The model itself is not keeping anything between calls.

Quick check

A user asks your chat feature a question that depends on their last message. What must your product do?

Quick recap

  • What it does

    An LLM guesses the next token, adds it, and repeats until the answer is done.

  • How it learns

    Pre-training on huge amounts of text, then fine-tuning to follow instructions. Training happens once.

  • What you use

    Inference, every time someone asks, usually counted in tokens. Most builders reuse a pre-trained model.

Keep these three facts in your pocket: answers are confident guesses, the model does not look things up for you, and it does not remember unless you send the history.

Next: tokens, the pieces a model reads and the unit you pay for.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.