esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Throttling: Slow or Refuse Excess Requests to Keep Systems Healthy

How services cap work per client, per key, or per time window, and how a well-behaved caller should respond to a 429 reply.

In short

Throttling slows or refuses excess requests by capping work per client, key, or time window, and a well-behaved caller backs off when it gets a 429 reply.

12 min readFree

What you will learn

Throttling means controlling how much work a service accepts in a given time. When requests arrive faster than the service can handle, the service slows them down, delays them, or turns some of them away. The goal is to keep the system healthy during heavy load, and to stop one heavy client from hurting everyone else.

This lesson explains why teams throttle, the main strategies they use, what a client sees when it hits a limit, and how a careful client should respond. A worked example follows one API key through a limit, so you can watch each step happen.

Key words

Request
One message a client sends to a server, asking it to do something or send back data.
Client
The program or person that sends requests, such as a web browser or a script.
Throttling
Controlling how much work a system takes on in a set time, by slowing, delaying, or rejecting extra requests.
Rate limit
A cap on how many requests one client may send inside a set window of time.
Quota
A larger budget for usage, often counted per day or per month, for one customer or account.
Tenant
One customer or organisation that shares a system with other customers.
API key
A secret code that tells the server which program is calling its API.
Shared resource
Something many clients use at once, such as processor time, memory, a database, or network bandwidth.
HTTP 429
The status code a server sends when a client has sent too many requests in a set time.
Retry-After
A response header that tells the client how long to wait before trying again.

Why do teams throttle?

A service has limited parts. It has a fixed amount of processor time, memory, database connections, and network bandwidth. When too many requests arrive together, every user waits longer and some requests fail. Teams often promise a target, called a service-level objective or SLO, such as answering most requests within a set time. Throttling helps keep that promise when the load rises.

  • Protect shared resources

    Many clients share the same servers and database. One client that floods them can use up the room everyone else needs.

  • Keep service fair

    A cap stops one client from taking most of the capacity, so smaller clients still get answers.

  • Control cost

    Cloud services often charge for usage. A cap stops a runaway script from creating a large bill.

  • Survive sudden spikes

    A news story or a flash sale can cause a burst of traffic. Adding capacity takes time and money, so a cap can hold the line while the system grows.

Throttling is not the same as adding servers. Autoscaling adds resources to match demand. Throttling is a limit that applies while those resources are still arriving, or when adding them would cost too much. Azure's guidance describes it as a cap on resource use that the system enforces on itself.

Tip

Put the limit on whatever runs out first. Counting requests per minute is the easiest number to see, but the real bottleneck is often how many requests are running at the same moment.

Quotas and limits per user, tenant, or API key

A limit has to be attached to an owner. Per user means each person gets a share. Per tenant means each customer organisation gets its own allowance, so a large customer cannot starve a small one. Per API key means each program that calls the API has its own counter. A quota is usually a longer budget, such as a daily total, while a rate limit is a short window such as one minute. Limits by IP address are also common, but many people can share one IP address behind a company proxy, so keys and accounts are usually more exact.

What are common ways to throttle?

Teams combine several strategies. Some count requests over time. Some limit how many requests run at once. Some turn off extra features when the load is high. The sub-topics below explain each one in turn.

What are fixed and sliding windows?

A fixed window counts requests inside set blocks of time, such as each clock minute. It is simple to build. Its weak point is the edge of a block. A client can use its full allowance at the end of one minute and again at the start of the next, so about twice the allowance can pass across that boundary. A sliding window instead counts requests in the most recent stretch of time, such as the last 60 seconds. That removes most of the edge problem, but the server must remember more data about each client.

What are token and leaky buckets?

Think of a token bucket as a container that holds up to a set number of tokens. Tokens drip in at a steady rate. Each request takes one token, and a request that finds no token is refused. A full bucket lets a short burst through, but the long-run average cannot go above the refill speed. A leaky bucket works differently. Requests join a queue that drains at a fixed pace, and when the queue is full, new arrivals are dropped. Its output is very even, which suits a slow back end that needs a steady flow.

Python
class TokenBucket:
    def __init__(self, capacity, refill_per_second):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_per_second = refill_per_second
        self.last_time = 0

    def allow(self, now):
        elapsed = now - self.last_time
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_second)
        self.last_time = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

bucket = TokenBucket(3, 1)
for second in [0, 0, 0, 0, 1, 1]:
    result = 'allowed' if bucket.allow(second) else 'rejected'
    print(second, result)

Output

0 allowed
0 allowed
0 allowed
0 rejected
1 allowed
1 rejected

Tip

Token bucket suits most public APIs because it allows short bursts. Leaky bucket suits a pipeline that must never see a spike. Names also vary between sources. Some call a delaying limit a throttle and a refusing limit a rate limit, so read the documentation of the tool you use.

How do you limit concurrency?

Concurrency means how many requests are in progress at the same moment. A service might allow 20 in progress, so the 21st must wait or be refused. This matters because a slow report or a large upload can hold a slot for a long time. Even a modest request rate can exhaust threads, connections, and memory when each request runs long.

Turning off extra features under load

Graceful degradation means switching off or simplifying nonessential features when the system is busy, so essential features keep working. A video app might lower stream quality. A shop might hide product recommendations while keeping checkout running. The answer is less complete, but the service stays up and keeps its targets.

Worked example: a per-key limit on a public API

The example below uses a hypothetical public API. It gives every API key 600 requests per minute. The numbers are invented for teaching. Large public APIs publish their own limits in their documentation, so check those pages for real figures.

  1. 1

    Find the caller

    The server reads the API key from each request. It then finds the counter for that key. A script using key A never touches the count for key B.

  2. 2

    Count inside the window

    For the current minute, the server adds one to the key's count for each accepted request, and it knows how many requests remain.

  3. 3

    Serve requests under the limit

    A request that fits under 600 gets the normal answer. The reply can also carry headers that show how many requests are left.

  4. 4

    Refuse the excess

    The script sends its 601st request in that minute. The server does not run the work. It returns HTTP 429 and a Retry-After header of 30 seconds.

  5. 5

    The script waits

    A well-behaved script reads the header, pauses for 30 seconds, and then sends again. It does not loop and resend at once.

  6. 6

    Everyone else keeps going

    Because the counter is per key, the other clients were never slowed. The server kept its capacity for them, so the one noisy client did not slow everyone down.

Text
HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: text/plain

Too many requests for this API key. Try again in 30 seconds.

What the client experiences

When a request is throttled, the client sees one of a few outcomes. The most common is an immediate refusal. Other systems hold the request for a while instead. Each outcome tells the client something different.

  • Refused with 429

    The server says the request was not wrong but came too soon. The status code is defined in RFC 6585, which is the document that standardised it.

  • Retry-After header

    This header gives a number of seconds, or a date, that marks when the client may try again. It is optional in the spec, but it is the most useful part of the reply.

  • Delayed or queued

    The server holds the request briefly and answers later. The client waits longer, and the holding area can fill up.

What the 429 status tells the client

RFC 6585 says the 429 code means the user has sent too many requests in a given amount of time. The spec does not say how to count requests or how to identify a user, so each API makes its own rules. A 429 reply must not be stored by a cache, so each retry gets a fresh decision. Servers are not forced to use 429. Under a heavy attack, a server may simply drop connections instead. A 429 is also different from a 403. A 403 says the request is not allowed at all, while a 429 says the request is allowed but must wait.

Delaying or queuing instead of refusing

Some systems hold a request for a short time instead of refusing it. This smooths small spikes, but it makes the client wait longer. A queue has a size limit, so when it fills, the system must refuse anyway. Designing that queue is a topic for a separate lesson.

How a well-behaved client responds

  1. 1

    Notice the 429

    Treat the reply as a timing signal. The request was valid, so do not change the request itself.

  2. 2

    Honor Retry-After

    If the header is present, wait at least that long. Many guides say this value should be followed exactly.

  3. 3

    Back off when no header is given

    Wait a while before trying again, and wait longer after each new refusal. Adding a small random amount to each wait keeps many clients from returning at the same instant.

  4. 4

    Stop after a limit

    Set a maximum number of attempts or a total time. When that runs out, report the failure instead of looping forever.

  5. 5

    Slow down before the wall

    Read any rate-limit headers the API sends, and pace requests so the count never reaches zero.

The full rules for repeating failed calls, including exact backoff formulas, belong to the Retry lesson. For now, the habit is enough: honor the server's timing and never hammer it.

Tip

Do not retry a 429 in a tight loop. A script that resends the moment it is refused keeps the limit used up, and some APIs block such clients for longer.

What the API team should add

On the server side, the team must decide who owns each request, pick an algorithm from the list above, and answer refusals clearly. It must also watch the results, because limits that are too tight hurt real users and limits that are too loose do nothing.

  • Identify each caller

    Attach every request to a user, tenant, or API key so that its count goes to the right owner.

  • Choose an algorithm on purpose

    Use a token bucket for bursty public traffic, a sliding window for smoother counts, and a concurrency cap for slow work.

  • Answer clearly

    Return 429 with Retry-After, not a vague 403 or 500, so the client knows what happened and when to try again.

Track how often 429 replies happen, broken down by endpoint and by client. A client that keeps getting refused while ignoring Retry-After is a sign that it needs help, and a sudden jump in 429s may show that a limit is set too low.

Check your understanding

Quick check

A photo-sharing app's API slows down every evening. One partner's script sends thousands of requests per second, and that hurts all the other users. What should you add, and what should the API return to the script when it goes over the limit?

Quick check

A payment API must let a partner send a burst of 50 requests when a job starts, but the average over time must stay at 5 requests per second. Which design fits best?

Key takeaways

Throttling keeps a service within the load it can really carry. It caps work per client, per tenant, or per key, and it uses strategies such as windows, token buckets, leaky buckets, concurrency caps, and feature shut-offs. When a request is refused, the server should say so with 429 and a Retry-After header, and a well-behaved client should wait, slow down, and stop after a set number of tries.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.