esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Latency vs Throughput: Time for One Request vs Work Done Over Time

Latency is how long one request takes. Throughput is how much work a system finishes in a period. Learn why they differ and how to choose which one to favor.

In short

Latency is how long one request takes, and throughput is how much work a system finishes in a period, so they answer different questions.

12 min readFree

What you will learn

Every system that serves people has a speed, and that speed can be measured in two different ways. This lesson explains the two measures, latency and throughput, and why improving one can make the other worse. By the end you will be able to tell them apart, follow a worked example, and pick the measure that matters for a product.

Key words

Request
One ask sent to a system, such as tapping a button to load a page or sending a message.
Latency
The time one request takes, from the moment it starts until the answer is back.
Throughput
The amount of work a system finishes in a set period of time.
Operation
One unit of work that the system performs, such as saving one record.
Batching
Grouping several requests together so the system handles them in one go.
Queue
A waiting line where requests stand until a worker is free.
Parallelism
Doing several pieces of work at the same time.
Bottleneck
The slowest step in a chain, which limits how much the whole chain can handle.

What is latency?

Latency measures the wait for a single request. The clock starts when the request is sent and stops when the result arrives. Imagine you order a coffee and wait six minutes for it. Those six minutes are the latency of that order. The coffee shop could be busy or quiet, but your own wait is what latency describes. (The six minutes is a made-up example.)

Lower latency means a faster response for the person waiting. Latency is a property of one request, so it tells you how one user feels, not how the whole system is doing.

Units of latency are units of time. Web and app calls are usually counted in milliseconds, where one millisecond is one thousandth of a second. Slower jobs are counted in seconds, and very slow ones in minutes.

What is throughput?

Throughput counts finished work across a period. It does not care how long any single request took. It asks how many requests the system completes each second, or how much data it moves each second.

A system with high throughput can serve many people at once. A system with low throughput runs out of room sooner, even if every single request is fast.

Units of throughput are a count of work divided by time. Common examples are requests per second, often written as RPS, and megabytes per second, written as MB/s, when the job is moving large amounts of data. Higher is better.

Throughput and latency are both part of the same story, but they are not the same number. Keep the two units apart in your head: milliseconds for latency, and something per second for throughput.

Why are they two different questions?

Latency answers the question: how long did one request wait? Throughput answers the question: how many requests can the system finish in a period? These questions are related, but neither answer decides the other.

  • Different scope

    Latency looks at one request. Throughput looks at the whole system over a period of time.

  • Different units

    Latency uses time, such as milliseconds. Throughput uses a count per time, such as requests per second.

  • Highway picture

    A narrow road with a fast speed limit gets one car to its target quickly, but few cars pass in a minute. A wide, slow highway lets many cars pass, though each car takes longer.

Because the two are independent, you can improve one and leave the other unchanged, or even make it worse. Good design starts by asking which of the two the product needs.

Learners often ask whether high throughput means low latency. It does not. A system can finish a huge number of requests per second while each request still waits a long time.

Latency is a per-request number, while throughput is a system-wide number. Some sources put it in these short words: latency is the property of each request, and throughput is the property of the system.

Throughput and latency are linked by a simple rule called Little's law. In words, it says the average number of requests in progress equals the throughput multiplied by the average latency. It holds for a system that is steady over time, not one that is growing without limit.

C=T×LC = T \times L

In that formula, C is the number of requests in progress at one time (concurrency), T is throughput in requests per second, and L is latency in seconds. Here is a made-up check: if 50 requests arrive each second and each takes 0.4 seconds, then about 20 requests are in progress at any moment. Notice that the formula ties the three together, but it does not tell you which one to improve first.

Throughput and latency are two views of one flow of work. You can use the formula to plan capacity, which means estimating how many requests are in progress before you add more workers.

How do you measure both numbers?

The code below uses one worker that handles requests one at a time. Each request needs 200 milliseconds of work. The program prints the latency of one request and the throughput of the whole run. Both numbers come from simple arithmetic, so you can check them by hand.

Python
# One worker handles requests one at a time.
# Each request needs 200 ms of work (a made-up number).
work_ms = 200
count = 5

latency_ms = work_ms
total_seconds = (work_ms * count) / 1000
throughput = count / total_seconds

print('Latency per request (ms):', latency_ms)
print('Throughput (requests per second):', throughput)

Output

Latency per request (ms): 200
Throughput (requests per second): 5.0

Read the output carefully. Each request took 200 milliseconds from start to finish, so its latency is 200 milliseconds. The five requests together took one second, so the system finished five requests per second. Those are two different answers to two different questions.

How do latency and throughput trade off?

Many techniques that raise throughput also make each request wait longer. Engineers call this a trade-off: you gain on one measure and pay on the other. The three techniques below are the most common ones.

  • Batching

    Batching bundles several requests and handles them together. This saves fixed costs for each item, so more items finish per second. The price is that the first item must wait for the bundle to fill.

  • Queuing

    A queue holds requests during a spike, so workers stay busy. The queue raises throughput, but every request that waits in line adds to its own latency.

  • Parallelism

    Parallelism runs several pieces of work at once. More workers can finish more requests per second. A single request, though, does not finish faster just because others run beside it, and shared resources can slow everyone down.

The sections below walk through batching in more detail, because it shows the trade clearly.

Batching is a useful picture of the trade. It groups requests so the system does less overhead per item. That is why throughput rises. But items that arrive early must wait for later items before the group leaves.

Python
# Five requests arrive at these times (ms). They wait until the last one arrives.
arrivals_ms = [0, 40, 80, 120, 160]
# One batch takes 100 ms of work, whatever its size (a made-up number).
batch_work_ms = 100
send_ms = arrivals_ms[-1]
finish_ms = send_ms + batch_work_ms
latencies = [finish_ms - t for t in arrivals_ms]

print('Latency of first request (ms):', latencies[0])
print('Latency of last request (ms):', latencies[-1])
print('Throughput (requests per second):', round(len(arrivals_ms) / (finish_ms / 1000), 1))

Output

Latency of first request (ms): 260
Latency of last request (ms): 100
Throughput (requests per second): 19.2

In the output, the first request waited 260 milliseconds, while the last request waited only 100. The group finished five requests in 0.26 seconds. The first request paid for the batch, and that is the trade-off in one example.

Queuing and parallelism show the same pattern in different ways. Each one keeps workers busy, which helps throughput, while each request spends more time in line or competing for shared resources.

Every system has a maximum throughput, set by its CPUs, memory, disks, network, and similar limits. Well below that maximum, latency stays fairly steady. As the load moves close to the limit, requests begin to queue, and latency climbs quickly. Past the limit, waits grow without a clear end.

The idea that latency rises close to a system's limit is the reason a busy service feels slow before it fails. The users at the back of the line feel the most delay.

Tip

When a system is busy, check the queue first. Long waits often come from requests waiting for a free worker, connection, or disk, not from slow code.

Worked example: a video app

Video apps such as YouTube show both measures clearly. The example below uses the public definitions that video companies publish. It does not use YouTube's internal numbers, which are not public.

  1. 1

    Pick the request

    You tap a video. The app asks the service for the video data. The clock for latency starts at that tap.

  2. 2

    Measure the wait for you

    The time from the tap until the first picture appears is the startup time. This wait is the latency of the play request.

  3. 3

    Check the patience limit

    So a startup latency that is too long costs viewers.

  4. 4

    Count the work the service does

    Throughput for streaming can be counted as how many viewers the service can serve at the same time. It can also be counted as the bitrate each stream sustains.

  5. 5

    See how the two can conflict

    If the service packs more viewers onto each server, more streams finish in a period, but each new viewer may wait longer in a queue to start. If it lowers the bitrate of each stream, more streams fit, but picture quality drops.

  6. 6

    Choose the focus

    For the play button, the viewer feels startup latency most. For planning how many viewers the whole service can carry, throughput matters most. Both numbers appear in the same product, but they answer different questions.

Video also has a second wait. When a viewer jumps to another point in the video, the time until playback resumes is called seek latency. It is still latency, because it measures one viewer's wait.

How to choose a goal

A common goal is to get as much work through the system as possible while keeping each wait within a limit that users accept. In plain words, aim for high throughput, but only as far as latency stays acceptable.

  • Interactive screens

    Screens where a person waits for each answer need low latency. Examples are a search box, a chat message, or a play button. A slow reply here is the problem users notice first.

  • Bulk pipelines

    Jobs that move large amounts of data need high throughput. An example is copying millions of log lines overnight. Nobody waits for one line, so the job can accept higher latency for each item.

  • Mixed products

    Many products have both kinds of work. Set a latency limit for the user-facing part, then push throughput as high as possible inside that limit.

Percentile latency is a way to see the slow end of the range. A p95 value is the time that 95 percent of requests finish within. An average can hide a few very slow requests, so many teams watch p95 or p99 alongside throughput.

Tip

Never judge a system by one number. Track throughput, average latency, and a high percentile such as p95 or p99 together. The slow requests usually reveal the problem first.

Check your understanding

Try these two short scenarios. Each one needs the ideas from this lesson to answer.

Quick check

A photo-sharing app's upload server handles 1,000 uploads per second. Yet each user waits 8 seconds to see their own upload finish. Which statement is best?

Quick check

A logging service starts grouping incoming messages into batches of 50. The number of messages written per second rises about tenfold. But a message can now wait up to 2 seconds for its batch to fill. What is the right way to describe this change?

Key takeaways

  • Two measures

    Latency is the time for one request, in units of time such as milliseconds. Throughput is the work finished per period, such as requests per second.

  • Independent but linked

    The two can move in different directions. Batching, queuing, and parallelism often raise throughput while each request waits longer.

  • Choose by product

    Screens that people use interactively favor low latency. Bulk pipelines favor high throughput. Most products aim for maximal throughput with acceptable latency.

The next step is to ask which of your product's requests a user waits for, and which work runs in the background. Measure both numbers for each, and decide which one to protect first.

Sources

How is this lesson?

Last updated on 17 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.