Latency vs Throughput: Time for One Request vs Work Done Over Time
Latency is how long one request takes. Throughput is how much work a system finishes in a period. Learn why they differ and how to choose which one to favor.
In short
Latency is how long one request takes, and throughput is how much work a system finishes in a period, so they answer different questions.
12 min readFree
What you will learn
Every system that serves people has a speed, and that speed can be measured in two different ways. This lesson explains the two measures, latency and throughput, and why improving one can make the other worse. By the end you will be able to tell them apart, follow a worked example, and pick the measure that matters for a product.
Key words
- Request
- One ask sent to a system, such as tapping a button to load a page or sending a message.
- Latency
- The time one request takes, from the moment it starts until the answer is back.
- Throughput
- The amount of work a system finishes in a set period of time.
- Operation
- One unit of work that the system performs, such as saving one record.
- Batching
- Grouping several requests together so the system handles them in one go.
- Queue
- A waiting line where requests stand until a worker is free.
- Parallelism
- Doing several pieces of work at the same time.
- Bottleneck
- The slowest step in a chain, which limits how much the whole chain can handle.
What is latency?
Latency measures the wait for a single request. The clock starts when the request is sent and stops when the result arrives. Imagine you order a coffee and wait six minutes for it. Those six minutes are the latency of that order. The coffee shop could be busy or quiet, but your own wait is what latency describes. (The six minutes is a made-up example.)
Lower latency means a faster response for the person waiting. Latency is a property of one request, so it tells you how one user feels, not how the whole system is doing.
Units of latency are units of time. Web and app calls are usually counted in milliseconds, where one millisecond is one thousandth of a second. Slower jobs are counted in seconds, and very slow ones in minutes.
What is throughput?
Throughput counts finished work across a period. It does not care how long any single request took. It asks how many requests the system completes each second, or how much data it moves each second.
A system with high throughput can serve many people at once. A system with low throughput runs out of room sooner, even if every single request is fast.
Units of throughput are a count of work divided by time. Common examples are requests per second, often written as RPS, and megabytes per second, written as MB/s, when the job is moving large amounts of data. Higher is better.
Throughput and latency are both part of the same story, but they are not the same number. Keep the two units apart in your head: milliseconds for latency, and something per second for throughput.
Why are they two different questions?
Latency answers the question: how long did one request wait? Throughput answers the question: how many requests can the system finish in a period? These questions are related, but neither answer decides the other.
Different scope
Latency looks at one request. Throughput looks at the whole system over a period of time.
Different units
Latency uses time, such as milliseconds. Throughput uses a count per time, such as requests per second.
Highway picture
A narrow road with a fast speed limit gets one car to its target quickly, but few cars pass in a minute. A wide, slow highway lets many cars pass, though each car takes longer.
Because the two are independent, you can improve one and leave the other unchanged, or even make it worse. Good design starts by asking which of the two the product needs.
Learners often ask whether high throughput means low latency. It does not. A system can finish a huge number of requests per second while each request still waits a long time.
Latency is a per-request number, while throughput is a system-wide number. Some sources put it in these short words: latency is the property of each request, and throughput is the property of the system.
Throughput and latency are linked by a simple rule called Little's law. In words, it says the average number of requests in progress equals the throughput multiplied by the average latency. It holds for a system that is steady over time, not one that is growing without limit.
In that formula, C is the number of requests in progress at one time (concurrency), T is throughput in requests per second, and L is latency in seconds. Here is a made-up check: if 50 requests arrive each second and each takes 0.4 seconds, then about 20 requests are in progress at any moment. Notice that the formula ties the three together, but it does not tell you which one to improve first.
Throughput and latency are two views of one flow of work. You can use the formula to plan capacity, which means estimating how many requests are in progress before you add more workers.
How do you measure both numbers?
The code below uses one worker that handles requests one at a time. Each request needs 200 milliseconds of work. The program prints the latency of one request and the throughput of the whole run. Both numbers come from simple arithmetic, so you can check them by hand.
# One worker handles requests one at a time.
# Each request needs 200 ms of work (a made-up number).
work_ms = 200
count = 5
latency_ms = work_ms
total_seconds = (work_ms * count) / 1000
throughput = count / total_seconds
print('Latency per request (ms):', latency_ms)
print('Throughput (requests per second):', throughput)Output
Latency per request (ms): 200 Throughput (requests per second): 5.0
Read the output carefully. Each request took 200 milliseconds from start to finish, so its latency is 200 milliseconds. The five requests together took one second, so the system finished five requests per second. Those are two different answers to two different questions.
How do latency and throughput trade off?
Many techniques that raise throughput also make each request wait longer. Engineers call this a trade-off: you gain on one measure and pay on the other. The three techniques below are the most common ones.
Batching
Batching bundles several requests and handles them together. This saves fixed costs for each item, so more items finish per second. The price is that the first item must wait for the bundle to fill.
Queuing
A queue holds requests during a spike, so workers stay busy. The queue raises throughput, but every request that waits in line adds to its own latency.
Parallelism
Parallelism runs several pieces of work at once. More workers can finish more requests per second. A single request, though, does not finish faster just because others run beside it, and shared resources can slow everyone down.
The sections below walk through batching in more detail, because it shows the trade clearly.
Batching is a useful picture of the trade. It groups requests so the system does less overhead per item. That is why throughput rises. But items that arrive early must wait for later items before the group leaves.
# Five requests arrive at these times (ms). They wait until the last one arrives.
arrivals_ms = [0, 40, 80, 120, 160]
# One batch takes 100 ms of work, whatever its size (a made-up number).
batch_work_ms = 100
send_ms = arrivals_ms[-1]
finish_ms = send_ms + batch_work_ms
latencies = [finish_ms - t for t in arrivals_ms]
print('Latency of first request (ms):', latencies[0])
print('Latency of last request (ms):', latencies[-1])
print('Throughput (requests per second):', round(len(arrivals_ms) / (finish_ms / 1000), 1))Output
Latency of first request (ms): 260 Latency of last request (ms): 100 Throughput (requests per second): 19.2
In the output, the first request waited 260 milliseconds, while the last request waited only 100. The group finished five requests in 0.26 seconds. The first request paid for the batch, and that is the trade-off in one example.
Queuing and parallelism show the same pattern in different ways. Each one keeps workers busy, which helps throughput, while each request spends more time in line or competing for shared resources.
Every system has a maximum throughput, set by its CPUs, memory, disks, network, and similar limits. Well below that maximum, latency stays fairly steady. As the load moves close to the limit, requests begin to queue, and latency climbs quickly. Past the limit, waits grow without a clear end.
The idea that latency rises close to a system's limit is the reason a busy service feels slow before it fails. The users at the back of the line feel the most delay.
Tip
When a system is busy, check the queue first. Long waits often come from requests waiting for a free worker, connection, or disk, not from slow code.
Worked example: a video app
Video apps such as YouTube show both measures clearly. The example below uses the public definitions that video companies publish. It does not use YouTube's internal numbers, which are not public.
- 1
Pick the request
You tap a video. The app asks the service for the video data. The clock for latency starts at that tap.
- 2
Measure the wait for you
The time from the tap until the first picture appears is the startup time. This wait is the latency of the play request.
- 3
Check the patience limit
So a startup latency that is too long costs viewers.
- 4
Count the work the service does
Throughput for streaming can be counted as how many viewers the service can serve at the same time. It can also be counted as the bitrate each stream sustains.
- 5
See how the two can conflict
If the service packs more viewers onto each server, more streams finish in a period, but each new viewer may wait longer in a queue to start. If it lowers the bitrate of each stream, more streams fit, but picture quality drops.
- 6
Choose the focus
For the play button, the viewer feels startup latency most. For planning how many viewers the whole service can carry, throughput matters most. Both numbers appear in the same product, but they answer different questions.
Video also has a second wait. When a viewer jumps to another point in the video, the time until playback resumes is called seek latency. It is still latency, because it measures one viewer's wait.
How to choose a goal
A common goal is to get as much work through the system as possible while keeping each wait within a limit that users accept. In plain words, aim for high throughput, but only as far as latency stays acceptable.
Interactive screens
Screens where a person waits for each answer need low latency. Examples are a search box, a chat message, or a play button. A slow reply here is the problem users notice first.
Bulk pipelines
Jobs that move large amounts of data need high throughput. An example is copying millions of log lines overnight. Nobody waits for one line, so the job can accept higher latency for each item.
Mixed products
Many products have both kinds of work. Set a latency limit for the user-facing part, then push throughput as high as possible inside that limit.
Percentile latency is a way to see the slow end of the range. A p95 value is the time that 95 percent of requests finish within. An average can hide a few very slow requests, so many teams watch p95 or p99 alongside throughput.
Tip
Never judge a system by one number. Track throughput, average latency, and a high percentile such as p95 or p99 together. The slow requests usually reveal the problem first.
Check your understanding
Try these two short scenarios. Each one needs the ideas from this lesson to answer.
Quick check
A photo-sharing app's upload server handles 1,000 uploads per second. Yet each user waits 8 seconds to see their own upload finish. Which statement is best?
Quick check
A logging service starts grouping incoming messages into batches of 50. The number of messages written per second rises about tenfold. But a message can now wait up to 2 seconds for its batch to fill. What is the right way to describe this change?
Key takeaways
Two measures
Latency is the time for one request, in units of time such as milliseconds. Throughput is the work finished per period, such as requests per second.
Independent but linked
The two can move in different directions. Batching, queuing, and parallelism often raise throughput while each request waits longer.
Choose by product
Screens that people use interactively favor low latency. Bulk pipelines favor high throughput. Most products aim for maximal throughput with acceptable latency.
The next step is to ask which of your product's requests a user waits for, and which work runs in the background. Measure both numbers for each, and decide which one to protect first.
Sources
- Design Gurus: Throughput vs Latency, What Each Measures and the Trade-Off
- Design Gurus course notes: Latency vs Throughput
- cs.fyi: System Design, Latency vs Throughput
- The Computer System Trail (arXiv paper 2603.28777), Latency v/s Throughput
- Anand Rochlani: Latency vs Throughput in System Design
- System Design Handbook: Difference Between Latency and Throughput in System Design
- Estuary: Throughput vs Latency, Understanding the Key Difference in Data Engineering
- GPU-to-Grid paper (arXiv 2602.05116), Tradeoff between Latency and Throughput
- Mux Docs: Startup Time metric
- Mux Blog: The four elements of video performance
- Mux Blog: Video Startup Time, the crucial video metric explained
- Wikipedia: Online video analytics
Last updated on 17 November 2026
Related lessons
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.