esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.04

Tokens

What a token is, why one word can be several, and how to count, pay for and cap them when you build with an AI model.

13 min readFree

The unit you pay for

The last lesson, How LLMs Work, ended with a promise: next, tokens, the pieces a model reads and the unit you pay for. Here they are.

When you build with an AI model, you will hear one word again and again: token. It decides what a request costs, how much text fits in one request, and how long an answer can be. If you plan a product, you plan in tokens.

A token is one small piece of text that a model treats as a single unit. Think of Lego. You do not hand a model a finished castle. You hand it a pile of bricks, and the brick count is what matters. This lesson looks at tokens from the builder's side: counting them, paying for them, and keeping them under control. The Tokenization lesson in Machine Learning Foundations goes deeper into how the cutting works.

Key words

Token
One piece of text a model treats as a single unit. It can be a whole word, part of a word, one letter or a punctuation mark.
Input tokens
The tokens you send to the model: your instructions, the question, any past messages.
Output tokens
The tokens the model sends back: its answer.
Context window
The most tokens a model can handle in one request, counting what goes in and what comes out. The next lesson covers it.
Max output
A cap you set on how many tokens the answer may use.
Estimate
A quick guess that is close but not exact.

What a token looks like

People often say a token is a word. Sometimes it is. Often it is not. The tiktoken project, an open-source tokenizer from OpenAI, describes it this way: language models don't see text like you and I, instead they see a sequence of numbers, known as tokens.

So before a model reads your message, a program called a tokenizer cuts the text into pieces and swaps each piece for a number. The model only ever works with those numbers.

Why can one word be several tokens? Because the pieces are chosen by how often letters sit together in lots of text, not by where the spaces are. The tiktoken README gives an example: "ing" is a common piece in English, so the word "encoding" will often be split into tokens like "encod" and "ing". Wikipedia's article on large language models gives another: the byte-pair encoding tokenizer of an older GPT-3 version split "tokenizer" into "token" and "izer".

Think of a zip code. Short, common words are like a well-known address and get one piece. A rare word is like a long street name you spell out in parts.

The OpenAI Cookbook adds two more details. In English, tokens commonly range in length from one character to one word. And spaces are usually grouped with the start of a word, so " great" with its leading space is one token.

Tip

A token is not a fixed size. The same word can cost one token in one place and two in another, depending on what comes next to it and which tokenizer is used. Treat any count you work out by hand as a guess.

Different models count differently

Every model family has its own tokenizer, so the same text can give different counts. The OpenAI Cookbook says it plainly: different models use different encodings.

It can even change between versions of one family. The Claude docs say that newer Claude models share a tokenizer that produces roughly 30 percent more tokens for the same text than earlier ones, and that the exact increase depends on the content. They add a clear warning: don't reuse token counts measured on an older model to estimate costs or context window fit. Count again with the model you plan to use.

What about other languages? The Cookbook notes that in some languages, tokens can be shorter than one character or longer than one word. A token shorter than a character means one letter can take more than one token, so the same message can use more tokens in such a language. We could not find a source with exact numbers, so test your own languages before you promise a price.

The same care applies to code and to text full of symbols. Many more punctuation marks and short names often means more pieces. We could not find an official figure, so the estimator below only shows you the pattern on one line of code, and you should measure real examples yourself.

You pay twice: in and out

Model providers charge by the token. The OpenAI Cookbook says usage is priced by token. The Claude pricing page lists one price for Input and a separate price for Output, quoted per million tokens, which the page writes as MTok.

That gives you two meters running at once. One counts everything you send. The other counts everything the model writes back.

  • Tokens in

    Your instructions, the question and any earlier messages you resend. Long prompts cost every single time.

  • Tokens out

    The answer. A chatty model costs more than a short one. Prices for output are often different from input, so look at both.

  • Hidden tokens

    Some models think before they answer. The Claude docs say these thinking tokens are billed as output tokens and count toward the max_tokens setting.

A shop analogy helps. Input is the paper you hand the printer. Output is the paper that comes out. You are charged for both stacks.

Here is the sum in one line, said in words right after it: the cost of a request is the input tokens divided by a million times the input price, plus the output tokens divided by a million times the output price.

cost=nin1,000,000×pin+nout1,000,000×pout\text{cost} = \frac{n_{\text{in}}}{1{,}000{,}000} \times p_{\text{in}} + \frac{n_{\text{out}}}{1{,}000{,}000} \times p_{\text{out}}

In the formula, n_in and n_out are the token counts and p_in and p_out are the prices per million tokens.

Limits are counted in tokens

Prices are not the only thing counted in tokens. Limits are too.

The Claude docs define the context window as all the text a model can reference when generating a response, including the response itself. They also say that everything in the request counts toward it: the system prompt, every message, tool definitions, and the output the model writes. So a long prompt leaves less room for the answer. The next lesson, the context window, looks at this closely.

The second limit is the answer length. In the Claude docs it is the max_tokens setting, which caps how many tokens the model may write in one reply.

Both limits grow and change from model to model, so we do not list numbers here. Look them up in your provider's docs on the day you build.

Tip

If a model stops in the middle of a sentence, the answer probably hit your cap. Raise the cap or ask for a shorter answer.

Count before you send

The best habit is simple: count first, send second. The Claude docs say you can count the tokens in a message before you send it, to manage rate limits and costs, make model routing decisions, and fit prompts to a target length. They also say the token counting feature is free to use, with its own request limits, and that the count is an estimate that may differ by a small amount from what is used when you create a message.

You do not always have a counter. So here is a rough one you can write yourself. It uses a rule from the tiktoken README: on average, in practice, each token corresponds to about 4 bytes of text. In plain English, one letter is one byte, so that works out to about four letters a token.

  1. 1

    Turn the text into bytes

    A byte is the small chunk a computer uses to store one plain English letter. Python gives it to you with encode.

  2. 2

    Divide by four

    This is the rough rule from the README. It is a guess, not a measurement.

  3. 3

    Try a second guess

    Count every word and every punctuation mark as one piece. If both guesses are close, you can trust the size of the answer a little more.

The program below applies both guesses to three texts: one short sentence, one longer paragraph and one line of code.

Python
import re

def guess_by_bytes(text):
    # Rough rule of thumb: about 4 bytes of text per token.
    return round(len(text.encode("utf-8")) / 4)

def guess_by_pieces(text):
    # Rough second guess: every word and every punctuation mark is one piece.
    return len(re.findall(r"\w+|[^\w\s]", text))

texts = {
    "sentence": "The cat sat on the mat.",
    "paragraph": (
        "Our shop ships orders within two days. If your parcel is late, "
        "reply to the confirmation email with your order number and we "
        "will check it for you. Returns are free for thirty days."
    ),
    "code": "total = sum(price * qty for price, qty in cart)",
}

for name, text in texts.items():
    print(name, len(text), guess_by_bytes(text), guess_by_pieces(text))

Output

sentence 23 6 7
paragraph 181 45 38
code 47 12 14

The output has four columns: the name, the number of characters, the guess by bytes, and the guess by words and punctuation. For the sentence, 23 characters give a byte guess of 6 tokens and a piece guess of 7. For the paragraph, 181 characters give 45 and 38. For the line of code, 47 characters give 12 and 14.

Notice how the two guesses disagree a little. That is the point. Both are rough, and neither is a real tokenizer. The code line shows the pattern from earlier: it has fewer letters than the paragraph, but the symbols (brackets, equals signs) make the word-and-punctuation guess higher than the byte guess.

Quick check

Your byte-based guess says a prompt is 500 tokens. What is the safest way to use that number?

A cost sum for 1,000 requests

Now let's price a small feature: a shop helper that answers one question about a parcel. The code below uses the same guess, then adds up the cost for 1,000 requests.

Two notes first. The prices are made up. They are not real prices, because real prices change, so look yours up. And the code calls a stand-in, fake_model, which returns the same fixed line every time so the program runs free. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse.

Python
# Made-up prices, not real ones. Dollars per one million tokens.
PRICE_IN = 2.00
PRICE_OUT = 8.00

def fake_model(messages):
    # Stand-in so this runs free. A real model is not like this.
    return "Thanks for asking. Your parcel should arrive within two days."

def guess(text):
    return round(len(text.encode("utf-8")) / 4)

prompt = (
    "You are a friendly shop helper. Answer in two short sentences. "
    "Customer question: Where is my parcel? It was due yesterday."
)
messages = [{"role": "user", "content": prompt}]
answer = fake_model(messages)

tokens_in = guess(prompt)
tokens_out = guess(answer)
requests = 1000

cost_in = requests * tokens_in / 1_000_000 * PRICE_IN
cost_out = requests * tokens_out / 1_000_000 * PRICE_OUT

print("tokens in per request :", tokens_in)
print("tokens out per request:", tokens_out)
print("input cost  :", round(cost_in, 4))
print("output cost :", round(cost_out, 4))
print("total for", requests, "requests:", round(cost_in + cost_out, 4))

Output

tokens in per request : 31
tokens out per request: 15
input cost  : 0.062
output cost : 0.12
total for 1000 requests: 0.182

Each request used about 31 tokens in and 15 tokens out. With the made-up prices of 2 dollars and 8 dollars per million tokens, input cost 0.062 and output cost 0.12, for a total of 0.182 dollars for 1,000 requests.

Look at the split. The answer is shorter than the prompt, but each output token costs four times more in this made-up example, so output was about two thirds of the bill. Which part is bigger in your own product depends on your prices and your prompts. Always check both.

A real prompt would be far longer, because you would add instructions, examples and earlier messages. That is why the sum is worth doing before you launch, not after.

Habits that keep tokens under control

Three habits save most of the money and most of the surprises.

  1. 1

    Count before you send

    Use your provider's counter if it has one, or a rough guess like the one above. Reject or shorten a prompt that is too big.

  2. 2

    Keep prompts short

    Every word you send is paid for on every request. Cut greetings, repeated rules and old messages the model no longer needs.

  3. 3

    Cap the answer

    Set a max output so one runaway reply cannot drain your budget or hit the context window limit.

The next program puts habits one and three in code. It refuses a prompt over a made-up limit, and it caps the answer. The model is again a stand-in called fake_model, so it is free to run. Remember that a real model is slower, costs money for every token, can answer differently each time, and can fail or refuse.

Python
MAX_PROMPT_TOKENS = 40   # made-up limit for this demo
MAX_ANSWER_TOKENS = 12   # made-up cap on the answer

def guess(text):
    return round(len(text.encode("utf-8")) / 4)

def fake_model(prompt, max_answer_tokens):
    # Stand-in so this runs free. A real model would stop at the cap itself.
    reply = "Your parcel should arrive within two days, and we will email you."
    words = reply.split()
    while guess(" ".join(words)) > max_answer_tokens:
        words.pop()
    return " ".join(words)

def ask(prompt):
    size = guess(prompt)
    if size > MAX_PROMPT_TOKENS:
        return "Too long: " + str(size) + " tokens, limit " + str(MAX_PROMPT_TOKENS)
    return fake_model(prompt, MAX_ANSWER_TOKENS)

print(ask("Where is my parcel?"))
print(ask("Where is my parcel? " * 12))

Output

Your parcel should arrive within two days, and we
Too long: 60 tokens, limit 40

The first call went through, and the answer was cut off at the cap, in the middle of a sentence. The second call was refused before any model was called, because the prompt was 60 tokens against a limit of 40. Cutting an answer short like this can look broken, so in a real product ask the model for a short answer in your instructions as well as setting the cap.

Tip

Keep a small table of the token counts you see for your real prompts. After a month you will know your average, your worst case, and your cost per user.

Quick check

Your feature runs 1,000 times a day. Which change usually cuts the bill most?

What to remember

  • Tokens are pieces

    They are not always words. One word can be several tokens, and each tokenizer cuts a little differently.

  • Two meters

    You pay for tokens in and tokens out, often at two different prices. Limits for the context window and the answer are counted in tokens too.

  • Count, trim, cap

    Count before you send, keep prompts short, and cap the answer length.

Next: the context window, everything the model can see at once.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.