esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.06

Sampling Parameters

How a model picks its next word from a list of chances, and the four dials that make answers safe, varied or less repetitive.

13 min readFree

The dials behind every answer

The last lesson, Context Window, ended with a promise: next, sampling parameters, the dials that make answers safe or surprising. Here they are.

Ask a chat model the same question twice and you may get two different answers. That is not a bug. It is a dial. Behind every reply, the model makes a long string of small choices, one word piece at a time, and a few settings decide how bold or how careful each choice is.

By the end you will know what temperature, top-k, top-p and repetition penalties do, when to turn each one up or down, and why a setting on one service may not mean the same on another. You will also run a small program that shows all four on a toy example, with no model and no cost.

Key words

Token
One piece of text a model works with: a word, part of a word, or a mark like a comma. The Tokenization lesson in Machine Learning Foundations goes deeper.
Score
A number the model gives each possible next token. A higher score means the model likes that token more.
Probability
A chance between 0 and 1. A probability of 0.25 means about one time in four.
Softmax
A small formula that turns a list of scores into chances that add up to 1.
Sampling
Picking the next token at random, using those chances like a weighted dice roll.
Temperature
A dial that makes the chances flatter (more surprising) or sharper (more predictable).
Seed
A starting number for a random generator. The same seed gives the same "random" picks again.

How a model picks the next word

A language model does not write a whole answer in one go. It writes one token, adds it to the text, then writes the next. Each time, it gives a score to every token it knows.

Think of a vending machine with a huge number of buttons. The model lights each button with a different brightness, and something must press one. That choice is what these dials control.

  1. 1

    Score

    The model gives every possible next token a score.

  2. 2

    Turn scores into chances

    Softmax changes the scores into probabilities that add up to 1. The Wikipedia page on the softmax function says it "normalizes it into a probability distribution".

  3. 3

    Filter (optional)

    Top-k or top-p throw away tokens that are very unlikely.

  4. 4

    Pick

    Choose one token. Either always take the best one, or roll a weighted dice.

  5. 5

    Repeat

    Add the token to the text and start again for the next one.

The Hugging Face blog post "How to generate text" calls the "always take the best" way greedy search: "Greedy search is the simplest decoding method. It selects the word with the highest probability as its next word". Greedy sounds safe. But the paper The Curious Case of Neural Text Degeneration says that using likelihood as a decoding objective "leads to text that is bland and strangely repetitive".

So most products sample instead. The same blog says that with sampling, generation "is not deterministic anymore": run it twice, and you can get two different texts. The paper adds an important point: "decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model". The model did not change. Only the way it picks did.

The four dials

Each dial below changes step 2, 3 or 4 of the list above. You can use one, or mix them.

Temperature

Temperature is the most famous dial. Picture a room full of people voting for the next word. At a low temperature, everyone follows the loudest voice. At a high temperature, the quiet voices get heard too.

The trick is simple. Before softmax, divide every score by the temperature. Then run softmax as usual. Here is the formula, with T as the temperature, z as the scores and e as roughly 2.718, the base of the exponential function.

pi=ezi/T∑jezj/Tp_i = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}

In words: divide each score by T, raise e to that power, then divide each result by the total of all of them, so the chances add up to 1. Wikipedia describes the effect: "A higher temperature results in a more uniform output distribution (i.e. with higher entropy; it is "more random"), while a lower temperature results in a sharper output distribution, usually with one value dominating."

  • Low (under 1)

    Scores are divided by a small number, so gaps grow. The top word dominates. Safe, steady, and often repetitive.

  • Temperature 1

    Scores stay as they are. The model's own chances are used with no change.

  • High (over 1)

    Scores are squeezed together. Odd words get a real chance. Varied, and riskier.

The Hugging Face blog adds the edge case: when temperature goes towards 0, "temperature scaled sampling becomes equal to greedy decoding". The same post makes the distribution sharper by "lowering the so-called temperature of the softmax".

Top-K

Top-K cuts the list. Keep only the K best tokens, throw away the rest, then pick from those. If K is 3, only the top three tokens can ever be chosen.

The Hugging Face blog puts it this way: "In Top-K sampling, the K most likely next words are filtered and the probability mass is redistributed among only those K next words." "Redistributed" just means the survivors' chances are scaled up so they add up to 1 again.

The catch is that K is fixed. Sometimes the model is very sure and only one or two words make sense, yet K still lets in K choices. Other times many words are fine, and K cuts good ones off. In its Hugging Face docs, the library's default top_k is 50 when the model's config does not set it.

Top-P

Top-P fixes that. It is also called nucleus sampling. The idea comes from the paper arXiv 1904.09751, which describes sampling "from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution".

In plain words: line up the tokens from most to least likely. Add their chances one by one until the total reaches P, for example 0.9. Keep exactly those tokens and drop the rest. The Hugging Face blog says Top-p "chooses from the smallest possible set of words whose cumulative probability exceeds the probability p".

"Cumulative" means a running total. And because the set is the smallest one that reaches P, its size changes by itself: few tokens when the model is sure, many when it is not. The blog's own example, with p set to 0.92, kept 9 words in one step and only 3 in another.

Repetition Penalties

Models can get stuck in loops, saying the same phrase again and again. A repetition penalty lowers the score of tokens that already appeared, so the model is nudged towards something new.

In the Hugging Face docs, the repetition_penalty setting works like this: 1.0 means no penalty, and values above 1.0 penalize previously generated tokens. A higher value penalizes more strongly. The docs also say the penalty "is applied at most once per token", and that for decoder-only models, like most large language models, the tokens counted include the prompt by default. They note that the original paper suggests a penalty of around 1.2 for "a good balance between truthful generation and lack of repetition".

Tip

Raise a repetition penalty a little at a time. Too strong, and the model avoids words it truly needs, like a name or a number you asked it to repeat.

Which setting for which job

There is no one right setting. It depends on whether you want the same good answer every time, or many different ideas.

  • Facts, code, data extraction

    Use a low temperature, or a small top-p. You want the likeliest answer, not a creative one. A wrong surprise here is a bug.

  • Chat and everyday writing

    Stay near the defaults. They are a middle road, and most products start there.

  • Brainstorming, names, stories

    Go higher. You want a spread of ideas. Add a mild repetition penalty if the ideas start to sound alike.

Two cautions. First, the dials stack: temperature, top-k and top-p can all run together, so change one at a time and test.

Second, services name and size these dials differently. In the Hugging Face docs the names are temperature, top_k, top_p and repetition_penalty, with defaults of 1.0, 50, 1.0 and 1.0 (no penalty) when nothing is set. Another provider may leave a dial out, call it something else, use another range, or set another default. Before you copy a setting from a tutorial, check the docs for the exact service and model you use.

Quick check

You are building a tool that pulls dates out of invoices. Which setting fits best?

Try it: a toy model in plain Python

Now let us see the dials in action. The program below is a stand-in so the code runs free: it has no model, no key and no network. It just has six candidate words for the end of "I drink a cup of ___" and a made-up score for each. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. Its list would also hold thousands of tokens, not six.

The first block turns scores into chances at three temperatures, then shows which words survive top-k=3 and top-p=0.9.

Python
import math

words = ["tea", "coffee", "water", "milk", "soup", "gravel"]
scores = [3.0, 2.6, 2.0, 1.2, 0.3, -1.0]  # the model's raw score for each word


def softmax(scores, temperature=1.0):
    scaled = [s / temperature for s in scores]
    exps = [math.exp(s) for s in scaled]
    total = sum(exps)
    return [e / total for e in exps]


def show(label, chances):
    print(label.ljust(14), "  ".join(f"{w}={c:.2f}" for w, c in zip(words, chances)))


for t in (0.5, 1.0, 1.5):
    show(f"temp {t}", softmax(scores, t))

chances = softmax(scores)
ranked = sorted(zip(words, chances), key=lambda pair: pair[1], reverse=True)

top_k = [w for w, c in ranked[:3]]
print("top-k=3 keeps:", top_k)

kept, running = [], 0.0
for w, c in ranked:
    kept.append(w)
    running += c
    if running >= 0.9:
        break
print("top-p=0.9 keeps:", kept, f"(adds up to {running:.2f})")

Output

temp 0.5       tea=0.62  coffee=0.28  water=0.08  milk=0.02  soup=0.00  gravel=0.00
temp 1.0       tea=0.44  coffee=0.29  water=0.16  milk=0.07  soup=0.03  gravel=0.01
temp 1.5       tea=0.36  coffee=0.27  water=0.18  milk=0.11  soup=0.06  gravel=0.02
top-k=3 keeps: ['tea', 'coffee', 'water']
top-p=0.9 keeps: ['tea', 'coffee', 'water', 'milk'] (adds up to 0.96)

Read the table row by row. At temperature 0.5, tea jumps to 0.62 and gravel drops to 0.00. At 1.5, the gaps shrink: tea falls to 0.36 and gravel rises to 0.02. The order of the words never changes. Only the gaps do.

The last two lines show the filters. Top-k=3 keeps the first three words, tea, coffee and water. Top-p=0.9 needs a fourth. Tea, coffee and water add up to only about 0.89, just under 0.9, so milk joins and the total reaches 0.96. That is the "smallest set" rule at work.

The second block is a full program, so it repeats the first few lines. It draws 10 words for each setting, using a fixed seed so the run can be repeated. Then it applies a repetition penalty to "tea", as if the word had just been used. This is one simple version I wrote for the demo: it divides a positive score by the penalty. Real libraries do the maths in their own way, so read their docs.

Python
import math
import random

words = ["tea", "coffee", "water", "milk", "soup", "gravel"]
scores = [3.0, 2.6, 2.0, 1.2, 0.3, -1.0]


def softmax(scores, temperature=1.0):
    exps = [math.exp(s / temperature) for s in scores]
    return [e / sum(exps) for e in exps]


def keep_top(chances, how_many):
    # zero out everything except the best few, then share the chances out again
    best = sorted(range(len(chances)), key=lambda i: chances[i], reverse=True)[:how_many]
    cut = [c if i in best else 0.0 for i, c in enumerate(chances)]
    return [c / sum(cut) for c in cut]


def keep_nucleus(chances, p):
    best = sorted(range(len(chances)), key=lambda i: chances[i], reverse=True)
    kept, running = [], 0.0
    for i in best:
        kept.append(i)
        running += chances[i]
        if running >= p:
            break
    cut = [c if i in kept else 0.0 for i, c in enumerate(chances)]
    return [c / sum(cut) for c in cut]


settings = {
    "temp 0.5": softmax(scores, 0.5),
    "temp 1.0": softmax(scores, 1.0),
    "temp 1.5": softmax(scores, 1.5),
    "top-k 3": keep_top(softmax(scores), 3),
    "top-p 0.9": keep_nucleus(softmax(scores), 0.9),
}
for name, chances in settings.items():
    rng = random.Random(42)  # same seed each time, so the run can be repeated
    print(name.ljust(10), rng.choices(words, weights=chances, k=10))

# repetition penalty: "tea" was already used, so make it less attractive
penalty = 1.5
penalised = [s / penalty if w == "tea" and s > 0 else s for w, s in zip(words, scores)]
before, after = softmax(scores), softmax(penalised)
for w, b, a in zip(words, before, after):
    print(f"{w:7} before={b:.2f} after={a:.2f}")

Output

temp 0.5   ['coffee', 'tea', 'tea', 'tea', 'coffee', 'coffee', 'coffee', 'tea', 'tea', 'tea']
temp 1.0   ['coffee', 'tea', 'tea', 'tea', 'water', 'coffee', 'milk', 'tea', 'tea', 'tea']
temp 1.5   ['water', 'tea', 'tea', 'tea', 'water', 'water', 'milk', 'tea', 'coffee', 'tea']
top-k 3    ['coffee', 'tea', 'tea', 'tea', 'coffee', 'coffee', 'water', 'tea', 'tea', 'tea']
top-p 0.9  ['coffee', 'tea', 'tea', 'tea', 'coffee', 'coffee', 'water', 'tea', 'tea', 'tea']
tea     before=0.44 after=0.22
coffee  before=0.29 after=0.40
water   before=0.16 after=0.22
milk    before=0.07 after=0.10
soup    before=0.03 after=0.04
gravel  before=0.01 after=0.01

Look at the ten draws. At temperature 0.5 only two words appear, tea and coffee. At 1.0 and 1.5, four different words show up, and at 1.5 water turns up three times. Top-k=3 and top-p=0.9 gave the same draws here: the same seed rolls the same dice, and no roll landed on milk.

In the last six lines, tea's chance halves, from 0.44 to 0.22, after the penalty. Coffee rises from 0.29 to 0.40 because the chances must still add up to 1. The penalty does not forbid tea. It only makes it less likely.

Tip

Change the seed from 42 to 7, or the top-p from 0.9 to 0.5, and run it again. Seeing the table move is the fastest way to feel each dial.

Quick checks

Quick check

What does dividing the scores by a low temperature, such as 0.5, do to the chances?

Quick check

Why can top-p work better than a fixed top-k?

What to take away

  • The model scores, the dials choose

    Every next word comes from a list of chances. Temperature reshapes it, top-k and top-p cut it, and a repetition penalty bends it.

  • Match the dial to the job

    Low for facts and code, higher for ideas and stories. Change one dial at a time.

  • Names and ranges differ

    Always read the docs of the exact service you use, and write your settings down.

Next: prompt engineering, writing instructions a model follows.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.