Machine Learning Foundations / Advanced / 03.45
Tokenization
How text is cut into small pieces a model can count: words, letters, and the in-between pieces that modern chat models use.
13 min readFree
Cutting text into pieces
The last lesson, Natural Language Processing, ended with a promise: next, tokenization, cutting text into pieces a model can count. Here it is.
A model cannot read. It only handles numbers. So before any model sees your sentence, something has to chop it into small pieces and give each piece a number. That chopping step is tokenization.
By the end you will know three ways to cut text, how byte-pair encoding builds its pieces, why chat models count tokens, and what special tokens are. You will also run two small programs.
Key words
- Token
- One piece of text that a model treats as a single unit. It can be a whole word, a part of a word, one letter or a punctuation mark.
- Tokenizer
- The program that cuts text into tokens.
- Vocabulary
- The full list of tokens a tokenizer knows. Every token in the list has its own number.
- Unknown word
- A word that is not in the vocabulary, so the tokenizer has no piece for it.
- Subword
- A piece smaller than a word but bigger than a single letter, like "low" inside "lowest".
- Byte-pair encoding (BPE)
- A method that builds subwords by repeatedly joining the pair of pieces that sit side by side most often.
- Special token
- A token that is not ordinary text. It is a marker, such as one for "unknown" or "padding".
What a token is
The word "token" is older than chat models. In computing, the Wikipedia page on lexical analysis describes lexical tokenization as the "conversion of a text into (semantically or syntactically) meaningful lexical tokens". A compiler does this to your code: it cuts a line into names, symbols and numbers before it tries to understand it.
The same page points out how that older kind differs from the kind used in language models. It says lexical tokenization "is usually based on a lexical grammar, whereas LLM tokenizers are usually probability-based". It adds a second difference: language-model tokenizers perform a second step that turns the tokens into numerical values.
In plain words, a language model's tokenizer does two jobs.
- 1
Cut
Split the text into tokens, using a rule or a method learned from lots of text.
- 2
Number
Look each token up in the vocabulary and swap it for its number.
The Large language model page explains why. Learning programs process numbers rather than text, so a vocabulary is decided first. Then each entry gets a whole number, and each number is linked to an embedding, the list of numbers that stands for that token inside the model.
Cutting by words
The first idea most people have is the obvious one: make every word a token. Split on spaces and punctuation, and you are done. This is word-level tokenization.
It is easy to explain, and it matches how we think. But the plain way (split on spaces) leaves punctuation stuck to words. "dollars," would become a different token from "dollars". A regular expression, a short pattern that describes which text to grab, fixes that by pulling punctuation out as its own tokens.
Here is a small tokenizer in plain Python. It tries the space split first so you can see the problem, then the pattern. The pattern grabs either a run of letters and digits (allowing one apostrophe, so "don't" stays whole) or one single punctuation mark.
import re
sentence = "Don't panic: it's 9.50 dollars, not 10!"
by_space = sentence.split(" ")
print(by_space)
pattern = r"\w+(?:'\w+)?|[^\w\s]"
tokens = re.findall(pattern, sentence.lower())
print(tokens)
print(len(tokens), "tokens")
vocab = {"don't": 0, "panic": 1, ":": 2, "it's": 3, "9": 4, ".": 5, "50": 6, "dollars": 7, ",": 8, "not": 9, "10": 10, "!": 11}
new = re.findall(pattern, "Don't panic, tokenizers!".lower())
print([t if t in vocab else "<unk>" for t in new])Output
["Don't", 'panic:', "it's", '9.50', 'dollars,', 'not', '10!'] ["don't", 'panic', ':', "it's", '9', '.', '50', 'dollars', ',', 'not', '10', '!'] 12 tokens ["don't", 'panic', ',', '<unk>', '!']
Read the output line by line. The first line is the space split. It left "panic:", "9.50", "dollars," and "10!" glued to their punctuation. The second line is the pattern at work: colon, comma and exclamation mark are now their own tokens, and "9.50" became three tokens, "9", "." and "50". The sentence cut into 12 tokens in total.
The last line shows the big weakness. We gave the tokenizer a tiny vocabulary of 12 entries, then fed it a new sentence. "tokenizers" was not in the list, so it became "<unk>", the unknown marker. All the meaning of that word is gone.
Problems with whole words
Word-level tokenization has two main problems. A small list gives many unknown words, and a big list is expensive.
Unknown words
Any word the vocabulary never saw turns into "unknown". New names, slang, typos and rare words all vanish. The Sennrich paper calls this a known issue: neural translation models "typically operate with a fixed vocabulary, but translation is an open-vocabulary problem".
Huge vocabularies
To cover many words, the list has to grow. "run", "runs" and "running" each need their own entry. A bigger list is more to store and more to learn.
Cutting by letters
The opposite idea is to make every single character a token. This is character-level tokenization. The vocabulary becomes tiny: letters, digits and punctuation marks.
That nearly solves the unknown-word problem: any word is just a row of known letters.
The cost is length. Every letter is a token, so the same sentence becomes a much longer row, and the model must work out for itself that l-o-w means something together.
So we have two extremes. Words give short rows but a huge, brittle list. Letters give a tiny, safe list but very long rows. The best answer sits in between.
Subwords: the middle road
Subword tokenization keeps common words whole and cuts rare words into familiar pieces. "the" stays one token. A rare word like "tokenizers" might become "token" and "izers".
The Large language model page gives a real example. It says the byte-pair encoding tokenizer used by the legacy version of GPT-3 would split "tokenizer" into "token" and "izer". Two known pieces, no unknown word.
The idea reached language models through a 2015 paper. Its abstract says the approach makes the model "capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units".
The abstract reports that subword models improved over a back-off dictionary baseline for English-German and English-Russian "by 1.1 and 1.3 BLEU, respectively". BLEU is a translation quality score; higher is better.
Byte-pair encoding, step by step
The version used for language models has a different goal. The page says it does not aim to compress text as much as possible. Instead it aims to turn plain text into tokens, which are whole numbers. The tokens it builds match varying amounts of source text, from single characters up to whole words.
The recipe is short. The page describes it like this: start with the unique characters as the first tokens, then merge the most frequent pair of adjacent tokens into a new, longer token, and repeat until a vocabulary of the size you chose is reached.
- 1
Start with letters
Write every word as separate letters, and add an end-of-word mark so the tokenizer knows where a word stops.
- 2
Count pairs
Look at every two neighbouring pieces in all the words. Count how often each pair appears, counting each word as many times as it shows up in your text.
- 3
Merge the winner
Take the most common pair and glue it into one new piece. Write that merge down.
- 4
Repeat
Count the pairs again, merge the new winner, and keep going until you have made as many merges as you wanted.
- 5
Use the merge list
To cut a new word, start from its letters and apply the saved merges in the same order.
The number of merges is the dial you control. More merges mean a bigger vocabulary and shorter rows. The Wikipedia page notes that the vocabulary of GPT-3.5 and GPT-4 has 100,258 entries: 100,000 from the BPE algorithm and 258 added as special tokens.
A tiny trainer you can run
Let's train a tiny tokenizer. Our made-up text has four words, with how often each appears: "low" 5 times, "lower" 2 times, "newest" 6 times and "widest" 3 times. The code uses "_" as the end-of-word mark. It does 6 merges, printing each one, then cuts three new words.
One detail to know. When two pairs tie for first place, this code picks the one that sorts last alphabetically. That is just a way to break ties the same way every run.
from collections import Counter
words = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
corpus = {tuple(w) + ("_",): n for w, n in words.items()}
def count_pairs(corpus):
pairs = Counter()
for symbols, n in corpus.items():
for pair in zip(symbols, symbols[1:]):
pairs[pair] += n
return pairs
def merge(corpus, pair):
out = {}
for symbols, n in corpus.items():
new, i = [], 0
while i < len(symbols):
if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == pair:
new.append(symbols[i] + symbols[i + 1])
i += 2
else:
new.append(symbols[i])
i += 1
out[tuple(new)] = out.get(tuple(new), 0) + n
return out
merges = []
for step in range(1, 7):
pairs = count_pairs(corpus)
best = max(pairs, key=lambda p: (pairs[p], p))
merges.append(best)
corpus = merge(corpus, best)
print(step, best, "seen", pairs[best], "times")
def tokenize(word):
symbols = tuple(word) + ("_",)
for pair in merges:
symbols = merge({symbols: 1}, pair).popitem()[0]
return list(symbols)
print(tokenize("lowest"))
print(tokenize("newer"))
print(tokenize("wide"))Output
1 ('t', '_') seen 9 times
2 ('s', 't_') seen 9 times
3 ('e', 'st_') seen 9 times
4 ('o', 'w') seen 7 times
5 ('l', 'ow') seen 7 times
6 ('w', 'est_') seen 6 times
['low', 'est_']
['n', 'e', 'w', 'e', 'r', '_']
['w', 'i', 'd', 'e', '_']Merges 1 to 3
"t" and the end mark were neighbours 9 times: 6 in "newest" and 3 in "widest". Then "s" joined "t_", and "e" joined "st_". By now the ending "est_" is one piece. Both "newest" and "widest" share it.
Merges 4 and 5
"o" joined "w", seen 7 times (5 in "low" plus 2 in "lower"). Then "l" joined "ow". Now "low" is one piece.
Merge 6
"w" joined "est_", seen 6 times, because "newest" has a "w" right before its ending. "widest" does not.
Now the new words. "lowest" was never in our text, yet it comes out as two pieces, "low" and "est_", both learned from other words. That is the whole trick: a word the tokenizer never saw is built from parts it knows.
"newer" shows the limit. Our tiny text never taught it that "n" and "e" go together, so it stays as single letters: n, e, w, e, r and the end mark, 6 pieces. And "wide" also stays as letters. With only 6 merges and 4 words, that is expected. A real tokenizer is trained on a huge amount of text and makes tens of thousands of merges, so common pieces get covered.
WordPiece and SentencePiece
BPE is not the only way to build subwords. Two other names come up often. Here is what our sources say about each, and no more.
WordPiece
The Large language model page lists it next to BPE as an algorithm for tokenization. It also says BERT, a well-known language model, writes "##" at the start of a piece to show that it continues the word before it.
SentencePiece
Its paper describes it as "a language-independent subword tokenizer and detokenizer". Older tools assumed the text was already split into words. SentencePiece can train "directly from raw sentences".
Byte-pair encoding
The method you just ran. The Sennrich paper used it to cut rare words into subword units.
Languages without spaces
Everything so far leaned on spaces. English puts a space between words, so splitting is easy. Many languages do not.
The Wikipedia page on text segmentation says that the space is a good approximation of a word divider in English and many other languages that use the Latin alphabet. It then lists languages that do not have a trivial word segmentation process: Chinese and Japanese, where sentences but not words are delimited, Thai and Lao, where phrases and sentences but not words are delimited, and Vietnamese, where syllables but not words are delimited.
Quick check
Why is a plain "split on spaces" tokenizer a poor fit for Chinese or Japanese?
Why chat models count tokens
If you have used a chat model, you may have seen limits and prices measured in tokens, not words. Now you can see why.
The model itself works on tokens, one number per piece. So the natural unit for everything is the token.
Limits
A model can only look at so many tokens at once. That is its context window. The Large language model page gives an early example: the small GPT-2 model had "a context window of only 1k tokens". Your question and its answer must both fit.
Price
Companies charge by the token. The same page notes that one model was released at "a much lower price per token for users" than a rival, so price is quoted per token.
Special tokens
Besides ordinary pieces, a vocabulary holds a few special tokens. They are not text from the page. They are markers the system uses to run itself.
Three kinds show up in our sources.
Unknown
"[UNK]" stands for a character not in the vocabulary. It is the safety net, the same idea as the "<unk>" our word tokenizer printed.
Mask
"[MASK]" marks a piece that has been hidden on purpose, as used in BERT.
Padding
Models usually need every input to be the same length, not ragged. The Large language model page says shorter texts must be "padded" until they match the length of the longest one, using filler tokens.
Quick check
What does a padding token do?
Putting it together
Here is the whole story in one place.
Words
Short rows and easy to read, but a huge list, and every new word is unknown.
Letters
A tiny list and nothing unknown, but very long rows and slow work.
Subwords
A middle road. Common words stay whole, rare ones break into known pieces. BPE, WordPiece and SentencePiece are three ways to build them.
Quick check
A word-level tokenizer meets a word that is not in its vocabulary. What happens?
Next: stemming, cutting words back to a rough root.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.