AI engineering / Beginner / 01.40
Inference Engineering
How to serve a model's answers quickly, reliably and cheaply: what to measure, and the simple levers that help.
13 min readFree
Making the answer arrive fast
The last lesson, Hugging Face, ended with a promise: next, inference engineering, making a model answer fast and cheap. Here it is.
By the end you will know six numbers worth watching, seven simple ways to improve them, and how to retry a failed call without making things worse. You will run two small programs, plus one tiny sum.
Key words
- Inference
- Using a trained model to produce an answer. Training teaches the model. Inference is every time it answers.
- Latency
- How long you wait. Total latency is the time from sending a request to getting the full answer.
- Time to first token
- How long until the first small piece of the answer shows up. A token is one piece of text; the Tokens lesson explains it.
- Throughput
- How much work gets done per second across all users together, for example requests finished per second.
- Percentile
- A cut point in a sorted list. The 95th percentile is the time that 95 out of 100 requests beat.
- Retry
- Trying a failed call again.
- Timeout
- A limit on how long you will wait before giving up on a call.
Six numbers to watch
You cannot improve what you do not measure. Six numbers cover most of what users and owners feel.
Time to first token
How long before the first word appears. This is the pause a user feels most. It is like the time before the waiter says "coming right up".
Tokens per second
How fast the rest of the answer arrives once it starts. A slow stream feels like someone typing with one finger.
Total latency
Time to first token plus the time to deliver every token. This is the full wait for the whole answer.
Throughput
How many requests your setup finishes per second for everyone at once. A busy day needs more of it.
Cost per request
What one answer costs: the tokens you send in plus the tokens that come out, times the price for each.
Error rate
The share of requests that fail or come back empty. If 2 in 100 fail, the error rate is 2 percent.
Why averages hide the pain
Say you time 20 requests. The average is easy to compute, but it can mislead. Most requests can be quick while a few are very slow, and the average sits somewhere in between, describing nobody's real experience.
Two better summaries help. The median is the middle value when you sort the times: half the requests are faster, half are slower. The 95th percentile (often written p95) tells you how bad the slow end is. If p95 is 6 seconds, then 1 request in 20 takes about that long or more.
Python's statistics module has both. Its docs describe median as the "middle value" of numeric data and say the median is "less affected by the presence of outliers". An outlier is a value far from the rest.
For percentiles it offers quantiles. Per the docs, it divides data into n continuous intervals with equal probability and returns a list of n - 1 cut points. Setting n to 100 "gives the 99 cuts points that separate data into 100 equal sized groups". The 95th percentile is therefore the item at index 94 of that list.
Here are 20 response times. They are made up, not measured from any real model or server. Two of them (4.8 and 6.5 seconds) are deliberately slow, like a kitchen that sometimes loses a ticket.
import statistics
# 20 made-up response times in seconds, one per request. Not measured from a real model.
times = [0.9, 1.1, 1.0, 1.3, 0.8, 1.2, 1.0, 0.9, 1.4, 1.1,
1.0, 1.2, 0.9, 1.1, 1.3, 1.0, 1.2, 4.8, 1.1, 6.5]
median = statistics.median(times)
p95 = statistics.quantiles(times, n=100)[94]
mean = statistics.mean(times)
print("requests:", len(times))
print(f"mean: {mean:.2f}s")
print(f"median: {median:.2f}s")
print(f"p95: {p95:.2f}s")
print(f"slowest: {max(times):.1f}s")Output
requests: 20 mean: 1.54s median: 1.10s p95: 6.42s slowest: 6.5s
The average, 1.54 seconds, looks a little slow. The median, 1.10 seconds, says a typical request is fine. And p95, 6.42 seconds, shows what the unlucky users see. With only 20 samples p95 sits close to the slowest time; real systems use thousands of requests.
Seven levers you can pull
Here is the toolbox. They are ordered roughly from the easiest and cheapest to try, to the ones that take more engineering.
- 1
Pick a smaller model where it is good enough
Not every request needs the biggest model. Sorting an email into a folder is easier than writing a legal summary. Test a smaller model on your real examples, and use the big one only where the small one fails.
- 2
Shorten prompts and cap the output
Fewer tokens in means less to read and pay for. A limit on tokens out stops a runaway answer. Both cut latency and cost together.
- 3
Cache what repeats
If many requests start with the same long text, a cache saves the repeated work. The Prompt Caching lesson covers how.
- 4
Batch requests
Serve several requests together instead of one by one. Covered in its own part below.
- 5
Stream the answer
Show words as they arrive so the wait feels shorter. The Streaming Responses lesson covers the code.
- 6
Retry with backoff, and set timeouts
Recover from short failures without hammering a busy server. We build this below.
- 7
Fall back to another model
If the first model keeps failing, send the request to a second one instead of showing an error.
Rate limits sit beside these. A rate limit caps how many requests you may send in a given time. Wikipedia says that when a service tells a client it is sending requests too frequently, the client reduces its rate. Retries with backoff are how you do that politely.
Let us look at the plain sum behind the first two levers. The price below is made up, only so the sum has a number: 2 cents per 1,000 tokens. It is not a real price, and real prices change.
# Made-up price, only so the sum has a number: 2 cents per 1,000 tokens (not a real price).
price_per_1000 = 0.02
def cost(tokens):
return tokens / 1000 * price_per_1000
before = cost(1200 + 400) # long prompt, long answer
after = cost(500 + 150) # shorter prompt, capped answer
print(f"before: ${before:.4f} per request")
print(f"after: ${after:.4f} per request")
print(f"per 100,000 requests: ${before * 100000:,.0f} vs ${after * 100000:,.0f}")Output
before: $0.0320 per request after: $0.0130 per request per 100,000 requests: $3,200 vs $1,300
Before, the request used 1,200 tokens in and 400 out, 1,600 in total, costing $0.0320. After trimming the prompt to 500 tokens and capping the answer at 150, it uses 650 tokens and costs $0.0130. Across 100,000 requests, that is $3,200 against $1,300. Small savings on one request become real money at scale.
Batching and the KV cache
Batching means handling several requests at the same time. Think of a bus. One bus carrying 40 people is far cheaper per person than 40 taxis. The paper on PagedAttention opens with this idea: "High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time."
A model writes an answer one token at a time. Each time it adds a token, it looks back at the earlier ones. To avoid redoing that work, it keeps notes about them in memory. The paper calls this the key-value cache (KV cache). The Hugging Face docs say it is stored in GPU memory for reuse. A GPU is the kind of chip these models run on.
The catch is size. The paper says this memory "for each request is huge and grows and shrinks dynamically". Managed badly, it is wasted, "limiting the batch size". A smaller batch means fewer people on the bus.
PagedAttention is one fix. The Hugging Face docs explain that it splits the KV cache into blocks reached through a lookup table, so the cache does not have to sit in one unbroken piece of memory. Blocks are given out only as needed. The paper reports that its serving system, vLLM, improves throughput of popular models by 2-4 times at the same level of latency compared to earlier systems.
Another trick is about timing. In a simple batch, the whole group waits for the slowest member. The Orca paper says that with such systems, requests that finish early cannot return to the client, and newly arrived requests have to wait until the current batch is completely done. Its fix is iteration-level scheduling: the scheduler runs just one step of the model at a time on the batch, so finished requests can leave and new ones can join between steps.
Streaming for perceived speed
Streaming does not make the model faster. It changes when the user sees the first word. The Hugging Face docs define token streaming as the server returning "the tokens one by one as the model generates them", and say it "reduces latency, one of the most critical aspects of a smooth experience".
Their own example uses round numbers. A system generating 100 tokens per second, asked for 1,000 tokens, makes a non-streaming user wait 10 seconds. With streaming, the docs say, users get initial results immediately, and although end-to-end latency stays the same, they "can see half of the generation after five seconds".
When a call fails: retries with backoff
Networks drop. Servers get busy. A real model call will fail sometimes, so your code should expect it. The simplest reaction is to try again. But how fast you try again matters a great deal.
Wikipedia gives the plain version: if a mobile app fails to connect to its server, it might try again 1 second later, then if it fails again, 2 seconds later, then 4, and so on. "Each time, the pause is multiplied by a fixed amount (in this case, 2)." The AWS Architecture Blog adds the word capped: capped exponential backoff means clients multiply their backoff by a constant after each attempt, "up to some maximum value".
Jitter: a little randomness
Backoff has one flaw. Suppose 100 users all fail at the same moment. They all wait 1 second, then all try again together, then all wait 2 seconds and try together again. The crowd stays in step, and the server gets hit in waves.
The AWS blog says this directly: "there are still clusters of calls". Its fix may sound strange. "Initially, jitter may appear to be a counter-intuitive idea: trying to improve the performance of a system by adding randomness." Jitter means each client waits a random amount, so the crowd spreads out.
The blog calls its main version "Full Jitter". In its simulation of 100 contending clients, the blog reports that jitter reduced the call count by more than half. It also says the return on the effort is huge, and that jittered backoff "should be considered a standard approach for remote clients".
A retry wrapper you can run
Now the code. This is the stand-in rule in action. The "model" below is a stand-in, a tiny function that fails twice and then answers with fixed text, so the code runs free with no account and no network. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse in ways a stand-in never will. The times are made up too.
To avoid real waiting, we use a fake clock: instead of sleeping, the code just adds the wait to a number. The random choices use a fixed seed, so you get the same output every run. After a failure the wrapper waits a random time between 0 and a ceiling that doubles each attempt (1, 2, 4...) up to a cap of 8 seconds.
import random
random.seed(7)
clock = 0.0 # fake clock, in seconds. Nothing really sleeps.
def fake_sleep(seconds):
global clock
clock += seconds
calls = 0
def fake_model(prompt):
"""A stand-in: fails twice, then answers. A real model is not this tidy."""
global calls
calls += 1
if calls <= 2:
raise RuntimeError("503 busy")
return "Your order ships on Friday."
def call_with_retries(prompt, max_tries=5, base=1.0, cap=8.0):
for attempt in range(1, max_tries + 1):
try:
answer = fake_model(prompt)
print(f"attempt {attempt} at t={clock:.2f}s: ok")
return answer
except RuntimeError as error:
if attempt == max_tries:
raise
ceiling = min(cap, base * 2 ** (attempt - 1))
wait = random.uniform(0, ceiling)
print(f"attempt {attempt} at t={clock:.2f}s: {error}, wait {wait:.2f}s (ceiling {ceiling:.1f}s)")
fake_sleep(wait)
print(call_with_retries("Where is my order?"))
print(f"total fake time: {clock:.2f}s")Output
attempt 1 at t=0.00s: 503 busy, wait 0.32s (ceiling 1.0s) attempt 2 at t=0.32s: 503 busy, wait 0.30s (ceiling 2.0s) attempt 3 at t=0.63s: ok Your order ships on Friday. total fake time: 0.63s
Read the output line by line. Attempt 1 failed with a made-up "503 busy" error. The ceiling was 1.0 seconds, and the random wait came out as 0.32 seconds. Attempt 2 failed too, at fake time 0.32 seconds. The ceiling doubled to 2.0 seconds, and the random wait was 0.30 seconds. Attempt 3 started at 0.63 seconds and succeeded. The whole thing took 0.63 fake seconds.
Quick check
Why add jitter, a random amount, to the wait between retries?
Rules for safe retries
Retries are powerful and easy to misuse. These habits keep them safe. They are practical advice, not from one source.
Retry only what can pass
A busy server or a dropped connection may work next time. A request with bad input will fail every time, so retrying only wastes money. Do not retry those.
Limit the tries
Set a maximum, like 5 attempts, as the code does. Then give up and show a friendly message or use a fallback.
Set a timeout on every call
Without one, a stuck call can wait forever. Pick a limit that matches your goal, and count it as a failure when it passes.
Fallbacks: a second plan
Retries help with short failures. Some failures last longer: a model that is down, or a limit you keep hitting. A fallback is a second route. If model A fails after its retries, send the request to model B.
Fallbacks can also be plain. If no model answers, show a saved answer, or a message that says what happened and when to try again. A calm message beats a blank screen or a crash.
What to remember
Measure first
Watch time to first token, tokens per second, total latency, throughput, cost per request and error rate. Use median and p95, not only the average.
Start with the cheap levers
A smaller model, shorter prompts, capped outputs, caching and streaming come before any heavy engineering.
Fail politely
Retry with backoff and jitter, set timeouts, limit tries, and keep a fallback ready.
Quick check
A user waits a long time before any word appears. Which number points at this?
Next: harness engineering, the code around the model that makes it dependable.
Sources
- Wikipedia: Exponential backoff
- AWS Architecture Blog: Exponential Backoff and Jitter
- arXiv: Efficient Memory Management for Large Language Model Serving with PagedAttention (abstract)
- USENIX OSDI '22: Orca, A Distributed Serving System for Transformer-Based Generative Models (abstract)
- Hugging Face text-generation-inference docs: Streaming
- Hugging Face text-generation-inference docs: PagedAttention
- Python docs: statistics
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.