# Latency vs Throughput: Time for One Request vs Work Done Over Time

> Latency is how long one request takes. Throughput is how much work a system finishes in a period. Learn why they differ and how to choose which one to favor.

**In short:** Latency is how long one request takes, and throughput is how much work a system finishes in a period, so they answer different questions.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/latency-vs-throughput

## What you will learn

Every system that serves people has a speed, and that speed can be measured in two different ways. This lesson explains the two measures, latency and throughput, and why improving one can make the other worse. By the end you will be able to tell them apart, follow a worked example, and pick the measure that matters for a product.

- **Request:** One ask sent to a system, such as tapping a button to load a page or sending a message.
- **Latency:** The time one request takes, from the moment it starts until the answer is back.
- **Throughput:** The amount of work a system finishes in a set period of time.
- **Operation:** One unit of work that the system performs, such as saving one record.
- **Batching:** Grouping several requests together so the system handles them in one go.
- **Queue:** A waiting line where requests stand until a worker is free.
- **Parallelism:** Doing several pieces of work at the same time.
- **Bottleneck:** The slowest step in a chain, which limits how much the whole chain can handle.

## What is latency?

Latency measures the wait for a single request. The clock starts when the request is sent and stops when the result arrives. Imagine you order a coffee and wait six minutes for it. Those six minutes are the latency of that order. The coffee shop could be busy or quiet, but your own wait is what latency describes. (The six minutes is a made-up example.)

Lower latency means a faster response for the person waiting. Latency is a property of one request, so it tells you how one user feels, not how the whole system is doing.

Units of latency are units of time. Web and app calls are usually counted in milliseconds, where one millisecond is one thousandth of a second. Slower jobs are counted in seconds, and very slow ones in minutes.

## What is throughput?

Throughput counts finished work across a period. It does not care how long any single request took. It asks how many requests the system completes each second, or how much data it moves each second.

A system with high throughput can serve many people at once. A system with low throughput runs out of room sooner, even if every single request is fast.

Units of throughput are a count of work divided by time. Common examples are requests per second, often written as RPS, and megabytes per second, written as MB/s, when the job is moving large amounts of data. Higher is better.

Throughput and latency are both part of the same story, but they are not the same number. Keep the two units apart in your head: milliseconds for latency, and something per second for throughput.

## Why are they two different questions?

Latency answers the question: how long did one request wait? Throughput answers the question: how many requests can the system finish in a period? These questions are related, but neither answer decides the other.

- **Different scope:** Latency looks at one request. Throughput looks at the whole system over a period of time.
- **Different units:** Latency uses time, such as milliseconds. Throughput uses a count per time, such as requests per second.
- **Highway picture:** A narrow road with a fast speed limit gets one car to its target quickly, but few cars pass in a minute. A wide, slow highway lets many cars pass, though each car takes longer.

Because the two are independent, you can improve one and leave the other unchanged, or even make it worse. Good design starts by asking which of the two the product needs.

Learners often ask whether high throughput means low latency. It does not. A system can finish a huge number of requests per second while each request still waits a long time.

Latency is a per-request number, while throughput is a system-wide number. Some sources put it in these short words: latency is the property of each request, and throughput is the property of the system.

Throughput and latency are linked by a simple rule called Little's law. In words, it says the average number of requests in progress equals the throughput multiplied by the average latency. It holds for a system that is steady over time, not one that is growing without limit.

$$
C = T \times L
$$

In that formula, C is the number of requests in progress at one time (concurrency), T is throughput in requests per second, and L is latency in seconds. Here is a made-up check: if 50 requests arrive each second and each takes 0.4 seconds, then about 20 requests are in progress at any moment. Notice that the formula ties the three together, but it does not tell you which one to improve first.

Throughput and latency are two views of one flow of work. You can use the formula to plan capacity, which means estimating how many requests are in progress before you add more workers.

## How do you measure both numbers?

The code below uses one worker that handles requests one at a time. Each request needs 200 milliseconds of work. The program prints the latency of one request and the throughput of the whole run. Both numbers come from simple arithmetic, so you can check them by hand.

```python
# One worker handles requests one at a time.
# Each request needs 200 ms of work (a made-up number).
work_ms = 200
count = 5

latency_ms = work_ms
total_seconds = (work_ms * count) / 1000
throughput = count / total_seconds

print('Latency per request (ms):', latency_ms)
print('Throughput (requests per second):', throughput)
```

Output:

```text
Latency per request (ms): 200
Throughput (requests per second): 5.0

```

Read the output carefully. Each request took 200 milliseconds from start to finish, so its latency is 200 milliseconds. The five requests together took one second, so the system finished five requests per second. Those are two different answers to two different questions.

## How do latency and throughput trade off?

Many techniques that raise throughput also make each request wait longer. Engineers call this a trade-off: you gain on one measure and pay on the other. The three techniques below are the most common ones.

- **Batching:** Batching bundles several requests and handles them together. This saves fixed costs for each item, so more items finish per second. The price is that the first item must wait for the bundle to fill.
- **Queuing:** A queue holds requests during a spike, so workers stay busy. The queue raises throughput, but every request that waits in line adds to its own latency.
- **Parallelism:** Parallelism runs several pieces of work at once. More workers can finish more requests per second. A single request, though, does not finish faster just because others run beside it, and shared resources can slow everyone down.

The sections below walk through batching in more detail, because it shows the trade clearly.

Batching is a useful picture of the trade. It groups requests so the system does less overhead per item. That is why throughput rises. But items that arrive early must wait for later items before the group leaves.

```python
# Five requests arrive at these times (ms). They wait until the last one arrives.
arrivals_ms = [0, 40, 80, 120, 160]
# One batch takes 100 ms of work, whatever its size (a made-up number).
batch_work_ms = 100
send_ms = arrivals_ms[-1]
finish_ms = send_ms + batch_work_ms
latencies = [finish_ms - t for t in arrivals_ms]

print('Latency of first request (ms):', latencies[0])
print('Latency of last request (ms):', latencies[-1])
print('Throughput (requests per second):', round(len(arrivals_ms) / (finish_ms / 1000), 1))
```

Output:

```text
Latency of first request (ms): 260
Latency of last request (ms): 100
Throughput (requests per second): 19.2

```

In the output, the first request waited 260 milliseconds, while the last request waited only 100. The group finished five requests in 0.26 seconds. The first request paid for the batch, and that is the trade-off in one example.

Queuing and parallelism show the same pattern in different ways. Each one keeps workers busy, which helps throughput, while each request spends more time in line or competing for shared resources.

Every system has a maximum throughput, set by its CPUs, memory, disks, network, and similar limits. Well below that maximum, latency stays fairly steady. As the load moves close to the limit, requests begin to queue, and latency climbs quickly. Past the limit, waits grow without a clear end.

The idea that latency rises close to a system's limit is the reason a busy service feels slow before it fails. The users at the back of the line feel the most delay.

> **Tip:** When a system is busy, check the queue first. Long waits often come from requests waiting for a free worker, connection, or disk, not from slow code.

## Worked example: a video app

Video apps such as YouTube show both measures clearly. The example below uses the public definitions that video companies publish. It does not use YouTube's internal numbers, which are not public.

1. **Pick the request:** You tap a video. The app asks the service for the video data. The clock for latency starts at that tap.
2. **Measure the wait for you:** The time from the tap until the first picture appears is the startup time. This wait is the latency of the play request.
3. **Check the patience limit:** So a startup latency that is too long costs viewers.
4. **Count the work the service does:** Throughput for streaming can be counted as how many viewers the service can serve at the same time. It can also be counted as the bitrate each stream sustains.
5. **See how the two can conflict:** If the service packs more viewers onto each server, more streams finish in a period, but each new viewer may wait longer in a queue to start. If it lowers the bitrate of each stream, more streams fit, but picture quality drops.
6. **Choose the focus:** For the play button, the viewer feels startup latency most. For planning how many viewers the whole service can carry, throughput matters most. Both numbers appear in the same product, but they answer different questions.

Video also has a second wait. When a viewer jumps to another point in the video, the time until playback resumes is called seek latency. It is still latency, because it measures one viewer's wait.

## How to choose a goal

A common goal is to get as much work through the system as possible while keeping each wait within a limit that users accept. In plain words, aim for high throughput, but only as far as latency stays acceptable.

- **Interactive screens:** Screens where a person waits for each answer need low latency. Examples are a search box, a chat message, or a play button. A slow reply here is the problem users notice first.
- **Bulk pipelines:** Jobs that move large amounts of data need high throughput. An example is copying millions of log lines overnight. Nobody waits for one line, so the job can accept higher latency for each item.
- **Mixed products:** Many products have both kinds of work. Set a latency limit for the user-facing part, then push throughput as high as possible inside that limit.

Percentile latency is a way to see the slow end of the range. A p95 value is the time that 95 percent of requests finish within. An average can hide a few very slow requests, so many teams watch p95 or p99 alongside throughput.

> **Tip:** Never judge a system by one number. Track throughput, average latency, and a high percentile such as p95 or p99 together. The slow requests usually reveal the problem first.

## Check your understanding

Try these two short scenarios. Each one needs the ideas from this lesson to answer.

**Quick check:** A photo-sharing app's upload server handles 1,000 uploads per second. Yet each user waits 8 seconds to see their own upload finish. Which statement is best?

A. The problem is throughput, because 8 seconds means the server cannot handle enough uploads per second.
B. The problem is latency, because each upload takes 8 seconds from start to finish. More servers add capacity for more uploads per second, but they do not automatically shorten the steps one upload must go through.
C. The problem is both, so adding more servers will fix both numbers at the same time.
D. There is no problem, because 1,000 uploads per second is already a fast rate for every user.

**Answer:** B. The 1,000 per second is throughput, and it looks healthy. The 8-second wait belongs to one user's upload, so it is latency. Extra servers can lift throughput, and they can cut queue waits, but they do not by themselves shorten each upload's own steps.

**Quick check:** A logging service starts grouping incoming messages into batches of 50. The number of messages written per second rises about tenfold. But a message can now wait up to 2 seconds for its batch to fill. What is the right way to describe this change?

A. The change made the service faster in every way, so the 2-second wait is a bug that should be removed right away.
B. Throughput improved and per-message latency got worse. This is the usual trade-off of batching, so the team must decide whether a 2-second wait is acceptable for this product.
C. Latency improved, because the messages now leave the service in larger groups.
D. Nothing changed for users, because batching only affects the database and not the messages.

**Answer:** B. Batching raises throughput by handling items together, and each item then waits for its group. The 2-second wait is the cost of the higher throughput. Whether that cost is acceptable depends on the product, which is the goal-setting step from this lesson.

## Key takeaways

- **Two measures:** Latency is the time for one request, in units of time such as milliseconds. Throughput is the work finished per period, such as requests per second.
- **Independent but linked:** The two can move in different directions. Batching, queuing, and parallelism often raise throughput while each request waits longer.
- **Choose by product:** Screens that people use interactively favor low latency. Bulk pipelines favor high throughput. Most products aim for maximal throughput with acceptable latency.

The next step is to ask which of your product's requests a user waits for, and which work runs in the background. Measure both numbers for each, and decide which one to protect first.

## Sources

- [Design Gurus: Throughput vs Latency, What Each Measures and the Trade-Off](https://www.designgurus.io/blog/throughput-vs-latency)
- [Design Gurus course notes: Latency vs Throughput](https://www.designgurus.io/course-play/grokking-the-system-design-interview/doc/latency-vs-throughput)
- [cs.fyi: System Design, Latency vs Throughput](https://cs.fyi/guide/latency-vs-throughput)
- [The Computer System Trail (arXiv paper 2603.28777), Latency v/s Throughput](https://arxiv.org/pdf/2603.28777)
- [Anand Rochlani: Latency vs Throughput in System Design](https://anandrochlani.com/blog/latency-vs-throughput-system-design)
- [System Design Handbook: Difference Between Latency and Throughput in System Design](https://www.systemdesignhandbook.com/blog/difference-between-latency-and-throughput-in-system-design/)
- [Estuary: Throughput vs Latency, Understanding the Key Difference in Data Engineering](https://estuary.dev/blog/latency-vs-throughput/)
- [GPU-to-Grid paper (arXiv 2602.05116), Tradeoff between Latency and Throughput](https://arxiv.org/pdf/2602.05116)
- [Mux Docs: Startup Time metric](https://www.mux.com/docs/guides/data-startup-time-metric)
- [Mux Blog: The four elements of video performance](https://www.mux.com/blog/the-four-elements-of-video-performance)
- [Mux Blog: Video Startup Time, the crucial video metric explained](https://www.mux.com/blog/the-video-startup-time-metric-explained)
- [Wikipedia: Online video analytics](https://en.wikipedia.org/wiki/Online_video_analytics)
