esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.16

Watching cost and speed

How to measure tokens, time and money on every model call, so a surprise bill or a slow answer shows up before your users feel it.

13 min readFree

Keeping speed and spend in check

The last lesson, Tracing & logging, ended with a promise: next, cost and latency monitoring, keeping speed and spend in check. Here it is.

Think of a phone plan. If you never look at your usage, the bill comes as a shock. If you check it every week, you notice early when something is eating your data. A product built on an AI model has the same problem, with two meters running at once: money and time. Every call costs a little money and takes a little time. A thousand calls a day turns "a little" into a real number.

This lesson shows you how to read both meters. You will record what every call used, turn it into money, look at speed the honest way, and set up a warning before a bill gets out of hand.

Key words

Token
A small piece of text a model counts. Models read and write in tokens, and they bill by tokens. The Tokenization lesson in Machine Learning Foundations goes deeper.
Input and output tokens
Input tokens are what you send in (your prompt, history, documents). Output tokens are what the model writes back. They usually have different prices.
Latency
How long you wait. For a model call, the time from sending the request to getting the answer back.
Time to first token
When a reply streams in word by word, how long you wait before the very first piece shows up. It is how fast the answer feels like it starts.
Percentile
A way to describe a pile of numbers by position. The 95th percentile is the value that 95 out of 100 requests were faster than.
Budget
A spending limit you choose in advance, for example per user per day.
Alert
A message that fires when a number crosses a line you set.

Record every call

You cannot watch what you did not write down. The first job is small: every time your code calls a model, save a short note about it. Who asked, which feature, how many tokens went in and out, and how long it took.

You do not have to count tokens yourself. Providers send the counts back with the answer. The Anthropic Messages API reference, for example, lists usage fields named input_tokens and output_tokens, plus cache_creation_input_tokens and cache_read_input_tokens for prompt caching. Other providers name their fields differently, so read your provider's page and use what it gives you.

Time you measure yourself. Note the clock before the call, note it after, and subtract.

The code below does exactly that. It uses a tiny stand-in called fake_model, so it runs free with no key and no network. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The stand-in always returns the same text and the same token counts, so do not read anything into its numbers.

Python
import time

def fake_model(messages):
    # Stand-in so this runs free. A real model is slower and costs money.
    time.sleep(0.05)
    return {"text": "Sure, here is a summary.",
            "usage": {"input_tokens": 120, "output_tokens": 18}}

log = []

def call_and_record(feature, user, messages):
    start = time.perf_counter()
    reply = fake_model(messages)
    seconds = time.perf_counter() - start
    log.append({
        "feature": feature,
        "user": user,
        "input_tokens": reply["usage"]["input_tokens"],
        "output_tokens": reply["usage"]["output_tokens"],
        "seconds": round(seconds, 1),
    })
    return reply["text"]

call_and_record("summarize", "ana", [{"role": "user", "content": "Summarize this."}])
print(log)

Output

[{'feature': 'summarize', 'user': 'ana', 'input_tokens': 120, 'output_tokens': 18, 'seconds': 0.1}]

Each call leaves one small record in the log. In a real app you would save it to a file or a database instead of a list. The Tracing & logging lesson showed how to keep such records so you can look back at them.

Tip

Record the user and the feature on every call, even when it feels like extra work. Later you will want to ask "who spent this?" and "which part of the product costs the most?". You cannot answer if the note was never saved.

  1. 1

    Start a timer

    Read the clock just before you send the request.

  2. 2

    Call the model

    Send the request and wait for the reply.

  3. 3

    Stop the timer

    Read the clock again. The gap is your latency.

  4. 4

    Copy the usage counts

    Take the input and output token numbers from the reply.

  5. 5

    Save one record

    Write feature, user, tokens and time together, so you can group them later.

Turning tokens into money

Tokens are not money yet. To get money you need a price table: how much one input token costs and how much one output token costs. Our sum below uses a price per million tokens.

Keep that table in config, not scattered through your code. Prices change, and new models arrive with new prices. When it lives in one place, you update one line and every report is right again. Never copy a price from a lesson, a blog post or a chat answer. Read it from your provider's own pricing page on the day you set it.

The sum itself is plain. In words: take the input tokens times the input price, add the output tokens times the output price, and divide by one million.

cost=input tokens×input price+output tokens×output price1,000,000\text{cost} = \frac{\text{input tokens} \times \text{input price} + \text{output tokens} \times \text{output price}}{1{,}000{,}000}

The prices in this lesson are made up. They exist only so the arithmetic has something to chew on. Do not treat them as real.

Once you can price one request, you can add the prices up in three useful ways.

  • Per request

    The cost of one call. Good for spotting a single huge call.

  • Per user

    What each person costs you. Good for budgets and fair limits.

  • Per feature

    What each part of the product costs. Good for deciding where to save effort.

That is why the record from the last section kept the user and the feature. Group by one or the other and add up.

Watching speed the honest way

The usual first idea is to track the average speed. It is a trap, and the Google SRE book explains why. In its words, "When building a monitoring system from scratch, it’s tempting to design a system based upon the mean of some quantity". It then gives an example: with "an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds."

Picture a restaurant that says dinner takes 20 minutes on average. That sounds fine. But if one table in a hundred waits an hour, those guests do not care about the average. They remember the hour. An average hides them.

So look at percentiles. Wikipedia describes the idea like this: "the 97th percentile (P97) is the value such that 97% of the data points are less than it." The 50th percentile is the median, the middle value.

  • p50

    The typical request. Half were faster, half were slower.

  • p95

    Nineteen out of twenty were faster than this. It shows what a fairly unlucky user sees.

  • p99

    Ninety-nine out of a hundred were faster than this. It shows the real slow tail.

The SRE book also warns that tails add up: if your users depend on several services, "the 99th percentile of one backend can easily become the median response of your frontend." An agent that makes many model calls in a row feels this strongly, because it waits for the slow one over and over.

Time to first token

When your app streams the reply, as in the Streaming responses lesson, two times matter. The total time is how long until the last word. The time to first token is how long until the first word appears.

For a chat screen, the second one is what the person feels. A reply that starts after half a second and then keeps flowing feels quick, even if it takes ten seconds to finish. A reply that shows nothing for five seconds feels broken. So measure both, and track percentiles for both.

To record it, note the clock when you send the request, note it again when the first piece arrives, and note it a third time when the last piece arrives.

A worked example: a small cost report

Now let us put it together. The program below makes up 30 requests with a seeded random generator, so you get the same numbers every time. No model is called. The requests are invented records, shaped like the ones you would log. Four made-up users use three made-up features, and one request is a runaway on purpose: a huge prompt and a very slow answer.

It does four things. It prices every request with a made-up price table. It totals the cost per feature. It prints p50, p95 and p99 latency using the statistics module. And it raises an alert when a user passes a daily budget.

One detail on percentiles. The Python docs give the function as statistics.quantiles(data, *, n=4, method='exclusive'). Asking for n=100 cuts the data into 100 slices, and the cut points are the percentiles. We pass method="inclusive" so the answers stay inside the range of the real data.

Python
import random
import statistics

PRICE_PER_MILLION = {"input": 2.00, "output": 8.00}  # made-up numbers, in dollars
DAILY_BUDGET = 0.20  # made-up, dollars per user per day

rng = random.Random(7)
features = ["chat", "summarize", "search"]
users = ["ana", "ben", "caro", "dev"]

requests = []
for i in range(30):
    feature = rng.choice(features)
    user = rng.choice(users)
    input_tokens = rng.randint(200, 1500)
    output_tokens = rng.randint(50, 400)
    latency = round(rng.uniform(0.6, 2.5), 2)
    requests.append((feature, user, input_tokens, output_tokens, latency))

# One runaway: a long prompt and a very slow answer.
requests[11] = ("chat", "dev", 90000, 3000, 14.8)

def cost(input_tokens, output_tokens):
    return (input_tokens * PRICE_PER_MILLION["input"]
            + output_tokens * PRICE_PER_MILLION["output"]) / 1_000_000

per_feature = {}
per_user = {}
for feature, user, tin, tout, lat in requests:
    c = cost(tin, tout)
    per_feature[feature] = per_feature.get(feature, 0) + c
    per_user[user] = per_user.get(user, 0) + c

latencies = [r[4] for r in requests]
cuts = statistics.quantiles(latencies, n=100, method="inclusive")
total = sum(per_feature.values())

print("requests:", len(requests), " total cost: $%.4f" % total)
for name, value in sorted(per_feature.items()):
    print("  feature %-9s $%.4f" % (name, value))
print("mean latency: %.2f s" % statistics.mean(latencies))
print("p50: %.2f s  p95: %.2f s  p99: %.2f s" % (cuts[49], cuts[94], cuts[98]))

for user, value in sorted(per_user.items()):
    if value > DAILY_BUDGET:
        print("ALERT: %s spent $%.4f, over the $%.2f budget" % (user, value, DAILY_BUDGET))

priciest = max(requests, key=lambda r: cost(r[2], r[3]))
print("most expensive request:", priciest[0], priciest[1], priciest[2], "input tokens, $%.4f" % cost(priciest[2], priciest[3]))

Output

requests: 30  total cost: $0.3138
  feature chat      $0.2316
  feature search    $0.0376
  feature summarize $0.0446
mean latency: 1.81 s
p50: 1.42 s  p95: 2.32 s  p99: 11.19 s
ALERT: dev spent $0.2154, over the $0.20 budget
most expensive request: chat dev 90000 input tokens, $0.2040

Read the report from the top. The 30 requests cost $0.3138 in total. The chat feature costs $0.2316 of it, about three quarters. If you wanted to save money, chat is where to look first.

Now look at speed. The mean latency is 1.81 seconds, which sounds calm. The median (p50) is 1.42 seconds, so a typical request is fast. The p95 is 2.32 seconds. But the p99 is 11.19 seconds. One slow request in thirty was enough to drag the p99 up, and a plain average would have hidden it.

The alert fired for dev, who spent $0.2154 against a $0.20 budget. And the most expensive single request was a chat call with 90,000 input tokens that cost $0.2040. That one request is about two thirds of the whole day's cost.

Tip

A report this small already answers three real questions: where does the money go, how slow is the slow end, and who needs a limit. Start with a report like this before you buy any tool.

Budgets and alerts

A report you read once a month tells you about a bill after you have paid it. A budget and an alert tell you while it is happening.

Pick a limit for each thing you care about. For example, a daily spend per user, a daily spend for the whole product, and a p95 latency you will not go above. Check the numbers on a timer, and send a message when one crosses its line.

Do not make alerts noisy. The Google SRE book lists questions to ask about each alert, including "Can I take action in response to this alert?" An alert nobody can act on teaches people to ignore alerts. Warn at a soft line (say 80 percent of the budget) so someone can look, and add a hard stop only where a runaway would be truly harmful.

Alerts also protect you from your own users. A budget per user means one person, or one script, cannot quietly run up a bill for everyone.

Finding the expensive outliers

In our report, one request caused most of the cost. Real systems look the same: a few calls cost far more than the rest. Sort your records by cost, look at the top ten, and ask why. Three causes show up again and again.

  • Long prompts

    Someone pasted a whole book, or your code stuffs too much history or too many documents into every call. Input tokens pile up. The Context window lesson explains the limit.

  • Runaway agent loops

    An agent keeps calling the model and its tools, again and again, and never decides it is finished. Each turn adds more history, so each call gets bigger than the last.

  • Retries

    A call fails, your code tries again, and the retry fails too. Each attempt that ran may be billed, so a bug can multiply the bill.

The fixes are simple to say. Cap the length of what you send. Cap how many steps an agent may take. Cap how many times you retry. A limit that is too small is easy to raise. A bill that is too large is hard to give back.

The levers you can pull

Once you see where money and time go, you can change them. You have met each lever in an earlier lesson, so here is a short reminder.

  1. 1

    Cache what repeats

    If the start of every prompt is the same, caching can avoid paying full price to process it again. See Prompt caching.

  2. 2

    Use a smaller model where it is enough

    Not every job needs the biggest model. Test a smaller one on your own examples. See Choosing the right model.

  3. 3

    Shorten the context

    Send only what the model needs. Fewer input tokens cost less and are read faster.

  4. 4

    Cap the reply

    Ask for short answers where short is fine. Output tokens cost money and take time.

  5. 5

    Measure again

    After each change, check the same report. A change that does not move the numbers was not worth keeping.

Quick check

Your average latency looks fine, but some users say the app is very slow. What should you look at?

Quick check

Why keep your price table in config instead of inside the code?

Quick check

One request out of 30 took 14.8 seconds. Which number shows it best?

What to remember

  • Record every call

    Feature, user, input tokens, output tokens and time, saved together.

  • Price it from config

    Tokens times your price table. Prices change, so never hard-code them.

  • Watch the tail

    Use p50, p95 and p99, not just the average, and set budgets with alerts.

Next: LLM observability tools, dashboards for all of this.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.