esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.40

Inference Engineering

How to serve a model's answers quickly, reliably and cheaply: what to measure, and the simple levers that help.

13 min readFree

Making the answer arrive fast

The last lesson, Hugging Face, ended with a promise: next, inference engineering, making a model answer fast and cheap. Here it is.

By the end you will know six numbers worth watching, seven simple ways to improve them, and how to retry a failed call without making things worse. You will run two small programs, plus one tiny sum.

Key words

Inference
Using a trained model to produce an answer. Training teaches the model. Inference is every time it answers.
Latency
How long you wait. Total latency is the time from sending a request to getting the full answer.
Time to first token
How long until the first small piece of the answer shows up. A token is one piece of text; the Tokens lesson explains it.
Throughput
How much work gets done per second across all users together, for example requests finished per second.
Percentile
A cut point in a sorted list. The 95th percentile is the time that 95 out of 100 requests beat.
Retry
Trying a failed call again.
Timeout
A limit on how long you will wait before giving up on a call.

Six numbers to watch

You cannot improve what you do not measure. Six numbers cover most of what users and owners feel.

  • Time to first token

    How long before the first word appears. This is the pause a user feels most. It is like the time before the waiter says "coming right up".

  • Tokens per second

    How fast the rest of the answer arrives once it starts. A slow stream feels like someone typing with one finger.

  • Total latency

    Time to first token plus the time to deliver every token. This is the full wait for the whole answer.

  • Throughput

    How many requests your setup finishes per second for everyone at once. A busy day needs more of it.

  • Cost per request

    What one answer costs: the tokens you send in plus the tokens that come out, times the price for each.

  • Error rate

    The share of requests that fail or come back empty. If 2 in 100 fail, the error rate is 2 percent.

Why averages hide the pain

Say you time 20 requests. The average is easy to compute, but it can mislead. Most requests can be quick while a few are very slow, and the average sits somewhere in between, describing nobody's real experience.

Two better summaries help. The median is the middle value when you sort the times: half the requests are faster, half are slower. The 95th percentile (often written p95) tells you how bad the slow end is. If p95 is 6 seconds, then 1 request in 20 takes about that long or more.

Python's statistics module has both. Its docs describe median as the "middle value" of numeric data and say the median is "less affected by the presence of outliers". An outlier is a value far from the rest.

For percentiles it offers quantiles. Per the docs, it divides data into n continuous intervals with equal probability and returns a list of n - 1 cut points. Setting n to 100 "gives the 99 cuts points that separate data into 100 equal sized groups". The 95th percentile is therefore the item at index 94 of that list.

Here are 20 response times. They are made up, not measured from any real model or server. Two of them (4.8 and 6.5 seconds) are deliberately slow, like a kitchen that sometimes loses a ticket.

Python
import statistics

# 20 made-up response times in seconds, one per request. Not measured from a real model.
times = [0.9, 1.1, 1.0, 1.3, 0.8, 1.2, 1.0, 0.9, 1.4, 1.1,
         1.0, 1.2, 0.9, 1.1, 1.3, 1.0, 1.2, 4.8, 1.1, 6.5]

median = statistics.median(times)
p95 = statistics.quantiles(times, n=100)[94]
mean = statistics.mean(times)

print("requests:", len(times))
print(f"mean:   {mean:.2f}s")
print(f"median: {median:.2f}s")
print(f"p95:    {p95:.2f}s")
print(f"slowest: {max(times):.1f}s")

Output

requests: 20
mean:   1.54s
median: 1.10s
p95:    6.42s
slowest: 6.5s

The average, 1.54 seconds, looks a little slow. The median, 1.10 seconds, says a typical request is fine. And p95, 6.42 seconds, shows what the unlucky users see. With only 20 samples p95 sits close to the slowest time; real systems use thousands of requests.

Seven levers you can pull

Here is the toolbox. They are ordered roughly from the easiest and cheapest to try, to the ones that take more engineering.

  1. 1

    Pick a smaller model where it is good enough

    Not every request needs the biggest model. Sorting an email into a folder is easier than writing a legal summary. Test a smaller model on your real examples, and use the big one only where the small one fails.

  2. 2

    Shorten prompts and cap the output

    Fewer tokens in means less to read and pay for. A limit on tokens out stops a runaway answer. Both cut latency and cost together.

  3. 3

    Cache what repeats

    If many requests start with the same long text, a cache saves the repeated work. The Prompt Caching lesson covers how.

  4. 4

    Batch requests

    Serve several requests together instead of one by one. Covered in its own part below.

  5. 5

    Stream the answer

    Show words as they arrive so the wait feels shorter. The Streaming Responses lesson covers the code.

  6. 6

    Retry with backoff, and set timeouts

    Recover from short failures without hammering a busy server. We build this below.

  7. 7

    Fall back to another model

    If the first model keeps failing, send the request to a second one instead of showing an error.

Rate limits sit beside these. A rate limit caps how many requests you may send in a given time. Wikipedia says that when a service tells a client it is sending requests too frequently, the client reduces its rate. Retries with backoff are how you do that politely.

Let us look at the plain sum behind the first two levers. The price below is made up, only so the sum has a number: 2 cents per 1,000 tokens. It is not a real price, and real prices change.

Python
# Made-up price, only so the sum has a number: 2 cents per 1,000 tokens (not a real price).
price_per_1000 = 0.02

def cost(tokens):
    return tokens / 1000 * price_per_1000

before = cost(1200 + 400)   # long prompt, long answer
after = cost(500 + 150)     # shorter prompt, capped answer
print(f"before: ${before:.4f} per request")
print(f"after:  ${after:.4f} per request")
print(f"per 100,000 requests: ${before * 100000:,.0f} vs ${after * 100000:,.0f}")

Output

before: $0.0320 per request
after:  $0.0130 per request
per 100,000 requests: $3,200 vs $1,300

Before, the request used 1,200 tokens in and 400 out, 1,600 in total, costing $0.0320. After trimming the prompt to 500 tokens and capping the answer at 150, it uses 650 tokens and costs $0.0130. Across 100,000 requests, that is $3,200 against $1,300. Small savings on one request become real money at scale.

Batching and the KV cache

Batching means handling several requests at the same time. Think of a bus. One bus carrying 40 people is far cheaper per person than 40 taxis. The paper on PagedAttention opens with this idea: "High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time."

A model writes an answer one token at a time. Each time it adds a token, it looks back at the earlier ones. To avoid redoing that work, it keeps notes about them in memory. The paper calls this the key-value cache (KV cache). The Hugging Face docs say it is stored in GPU memory for reuse. A GPU is the kind of chip these models run on.

The catch is size. The paper says this memory "for each request is huge and grows and shrinks dynamically". Managed badly, it is wasted, "limiting the batch size". A smaller batch means fewer people on the bus.

PagedAttention is one fix. The Hugging Face docs explain that it splits the KV cache into blocks reached through a lookup table, so the cache does not have to sit in one unbroken piece of memory. Blocks are given out only as needed. The paper reports that its serving system, vLLM, improves throughput of popular models by 2-4 times at the same level of latency compared to earlier systems.

Another trick is about timing. In a simple batch, the whole group waits for the slowest member. The Orca paper says that with such systems, requests that finish early cannot return to the client, and newly arrived requests have to wait until the current batch is completely done. Its fix is iteration-level scheduling: the scheduler runs just one step of the model at a time on the batch, so finished requests can leave and new ones can join between steps.

Streaming for perceived speed

Streaming does not make the model faster. It changes when the user sees the first word. The Hugging Face docs define token streaming as the server returning "the tokens one by one as the model generates them", and say it "reduces latency, one of the most critical aspects of a smooth experience".

Their own example uses round numbers. A system generating 100 tokens per second, asked for 1,000 tokens, makes a non-streaming user wait 10 seconds. With streaming, the docs say, users get initial results immediately, and although end-to-end latency stays the same, they "can see half of the generation after five seconds".

When a call fails: retries with backoff

Networks drop. Servers get busy. A real model call will fail sometimes, so your code should expect it. The simplest reaction is to try again. But how fast you try again matters a great deal.

Wikipedia gives the plain version: if a mobile app fails to connect to its server, it might try again 1 second later, then if it fails again, 2 seconds later, then 4, and so on. "Each time, the pause is multiplied by a fixed amount (in this case, 2)." The AWS Architecture Blog adds the word capped: capped exponential backoff means clients multiply their backoff by a constant after each attempt, "up to some maximum value".

Jitter: a little randomness

Backoff has one flaw. Suppose 100 users all fail at the same moment. They all wait 1 second, then all try again together, then all wait 2 seconds and try together again. The crowd stays in step, and the server gets hit in waves.

The AWS blog says this directly: "there are still clusters of calls". Its fix may sound strange. "Initially, jitter may appear to be a counter-intuitive idea: trying to improve the performance of a system by adding randomness." Jitter means each client waits a random amount, so the crowd spreads out.

The blog calls its main version "Full Jitter". In its simulation of 100 contending clients, the blog reports that jitter reduced the call count by more than half. It also says the return on the effort is huge, and that jittered backoff "should be considered a standard approach for remote clients".

A retry wrapper you can run

Now the code. This is the stand-in rule in action. The "model" below is a stand-in, a tiny function that fails twice and then answers with fixed text, so the code runs free with no account and no network. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse in ways a stand-in never will. The times are made up too.

To avoid real waiting, we use a fake clock: instead of sleeping, the code just adds the wait to a number. The random choices use a fixed seed, so you get the same output every run. After a failure the wrapper waits a random time between 0 and a ceiling that doubles each attempt (1, 2, 4...) up to a cap of 8 seconds.

Python
import random

random.seed(7)
clock = 0.0  # fake clock, in seconds. Nothing really sleeps.

def fake_sleep(seconds):
    global clock
    clock += seconds

calls = 0

def fake_model(prompt):
    """A stand-in: fails twice, then answers. A real model is not this tidy."""
    global calls
    calls += 1
    if calls <= 2:
        raise RuntimeError("503 busy")
    return "Your order ships on Friday."

def call_with_retries(prompt, max_tries=5, base=1.0, cap=8.0):
    for attempt in range(1, max_tries + 1):
        try:
            answer = fake_model(prompt)
            print(f"attempt {attempt} at t={clock:.2f}s: ok")
            return answer
        except RuntimeError as error:
            if attempt == max_tries:
                raise
            ceiling = min(cap, base * 2 ** (attempt - 1))
            wait = random.uniform(0, ceiling)
            print(f"attempt {attempt} at t={clock:.2f}s: {error}, wait {wait:.2f}s (ceiling {ceiling:.1f}s)")
            fake_sleep(wait)

print(call_with_retries("Where is my order?"))
print(f"total fake time: {clock:.2f}s")

Output

attempt 1 at t=0.00s: 503 busy, wait 0.32s (ceiling 1.0s)
attempt 2 at t=0.32s: 503 busy, wait 0.30s (ceiling 2.0s)
attempt 3 at t=0.63s: ok
Your order ships on Friday.
total fake time: 0.63s

Read the output line by line. Attempt 1 failed with a made-up "503 busy" error. The ceiling was 1.0 seconds, and the random wait came out as 0.32 seconds. Attempt 2 failed too, at fake time 0.32 seconds. The ceiling doubled to 2.0 seconds, and the random wait was 0.30 seconds. Attempt 3 started at 0.63 seconds and succeeded. The whole thing took 0.63 fake seconds.

Quick check

Why add jitter, a random amount, to the wait between retries?

Rules for safe retries

Retries are powerful and easy to misuse. These habits keep them safe. They are practical advice, not from one source.

  • Retry only what can pass

    A busy server or a dropped connection may work next time. A request with bad input will fail every time, so retrying only wastes money. Do not retry those.

  • Limit the tries

    Set a maximum, like 5 attempts, as the code does. Then give up and show a friendly message or use a fallback.

  • Set a timeout on every call

    Without one, a stuck call can wait forever. Pick a limit that matches your goal, and count it as a failure when it passes.

Fallbacks: a second plan

Retries help with short failures. Some failures last longer: a model that is down, or a limit you keep hitting. A fallback is a second route. If model A fails after its retries, send the request to model B.

Fallbacks can also be plain. If no model answers, show a saved answer, or a message that says what happened and when to try again. A calm message beats a blank screen or a crash.

What to remember

  • Measure first

    Watch time to first token, tokens per second, total latency, throughput, cost per request and error rate. Use median and p95, not only the average.

  • Start with the cheap levers

    A smaller model, shorter prompts, capped outputs, caching and streaming come before any heavy engineering.

  • Fail politely

    Retry with backoff and jitter, set timeouts, limit tries, and keep a fallback ready.

Quick check

A user waits a long time before any word appears. Which number points at this?

Next: harness engineering, the code around the model that makes it dependable.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.