esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Retry Storm: How Retries Can Take Down a Struggling System

When many clients retry failed requests at the same time, the extra load can stop a weak service from recovering. Here is how that forms and how to stop it.

In short

A retry storm happens when many clients retry failed requests at the same time, adding load that stops a struggling service from recovering.

12 min readFree

What You Will Learn

Retries are meant to make software more reliable. If a request fails, sending it again may work. But when a service is already struggling, many retries at the same moment can make things worse. This lesson explains that problem, which system designers call a retry storm. In this course it is treated as a performance antipattern, which means a common design mistake that looks sensible but harms the system. You will see how a small slowdown grows into a large outage, how to stop that growth, and how to tell which errors are worth retrying.

Key words

Retry
Sending the same request again after it fails.
Retry storm
Many clients or services retrying at the same time, adding load to a system that is already struggling, so it cannot recover.
Load
The amount of work a system must handle in a set time, such as requests per second.
Layer or tier
One part of a chain of software, such as a phone app, an API, an order service, or a payments service.
Timeout
A time limit on how long a caller waits for an answer before it gives up.
Transient error
A temporary failure, such as a timeout or a busy server, that may pass on its own.
Permanent error
A failure that will happen again on every try, such as a request with a wrong format.
Exponential backoff
Waiting longer after each failed try, for example a wait that doubles each time.
Jitter
A small random change added to each wait, so many clients do not act at the same moment.
Circuit breaker
A guard that stops calls to a service that keeps failing, and returns an error at once instead.
Retry budget
A limit on how many retries are allowed, usually as a share of normal traffic.
Idempotent
Safe to repeat. Doing the same request twice has the same result as doing it once.

How Does a Retry Storm Form?

A retry storm usually starts small. A service becomes slow, or returns errors for a short time. The cause might be a busy database, a new software release, or a brief network problem. Callers see timeouts, because their requests did not finish in time. Each caller then sends the request again. Retries are meant to help, because a short fault often passes. But the extra attempts arrive while the service is still busy, so it receives even more work than before.

  • The spark

    A small slowdown or a short burst of errors makes some requests fail or time out.

  • The multiplier

    Each failed request is sent again, so the same work arrives several times. Load rises just when the service has the least spare room to handle it.

  • The feedback loop

    The service slows further under the extra load, more requests fail, and more retries follow. The loop can keep going even after the original fault has ended.

The last step is the dangerous one. A retry storm is retries generating more load than the original traffic, which keeps a service down after the fault that started it has passed.

Tip

A retry is not free. Each retry is extra work sent to a service that may already be overloaded. Ask whether the extra work will help the service recover, or hurt it.

How Do Retries Stack Across Layers?

Most real systems have several layers. A phone app may call an API gateway, which calls an order service, which calls a payments service. If every layer retries when the layer below it fails, the retries multiply. Retrying at all layers multiplies the number of retries rather than adding to them. Suppose each of three tiers tries a call three times. One user action then reaches the bottom service 27 times.

Python
tiers = 3
attempts_per_tier = 3
print(attempts_per_tier ** tiers)

Output

27

The same arithmetic works for any number of tiers. Each extra layer that retries multiplies the traffic again. This is why a designer should count the layers before adding any retry logic, and choose one layer to do the retrying.

Tip

Retry at one layer only. The layer closest to the failing call is usually the best place, and the other layers should pass the error up without retrying it again.

Why Does the Storm Keep Going?

Clients that fail at the same moment tend to retry at the same moment too. If a thousand clients all wait exactly one second and then send again, the service gets a wave of a thousand calls, then silence, then another wave. Each wave can knock the service down again. Immediate retries make this worse, because they return while the service is still in trouble.

Quick check

A service handles 500 requests per second. For 20 seconds it returns errors. Each client retries immediately, up to 2 times, and every request fails during that window. About how many requests per second reach the service during the outage?

Worked Example: A Food-Delivery Order Screen

Consider a mobile food-delivery app. When a customer taps Place order, the app calls a payments service to charge the card. Follow what happens when payments slows down, then see how two fixes change the result.

  1. 1

    Payments slows down

    The payments service starts answering slowly. Many calls still finish, but some pass the order screen's timeout.

  2. 2

    Phones retry every second

    The app treats each timeout as a failure and sends the charge request again. Every phone retries about once a second, with no wait that grows over time.

  3. 3

    Customers tap again

    The screen shows no result, so thousands of users tap Place order again. Each tap adds another request on top of the retries already running.

  4. 4

    Payments goes fully down

    Payments now receives several times its normal traffic. It slows further, more calls time out, and more retries follow. The service stops answering altogether.

  5. 5

    Add backoff with jitter

    Each phone now waits longer after each failure, and a random jitter spreads those waits apart. The retries no longer hit payments in one synchronized wave.

  6. 6

    Add a circuit breaker

    After repeated failures, the app stops calling payments for a cooldown period. The order screen shows a clear message at once. When the cooldown ends, a few trial calls are allowed through. If they succeed, normal traffic resumes.

  7. 7

    Payments recovers

    With fewer and spaced-out calls, payments has room to catch up. The app no longer makes its own outage larger.

Tip

Before adding retries to Place order, make sure a repeated tap cannot charge a card twice. That is what idempotent design is for, and it has its own lesson.

How Do You Prevent a Retry Storm?

No single setting stops a retry storm. Designers combine several habits, so that each one covers a gap left by the others.

  • Timeouts

    Set a time limit on every remote call. A caller that waits forever holds memory, threads, and connections, and a stalled service can take the caller down with it.

  • Capped retry counts

    Allow only a few attempts per request. After the limit, stop and return a clear failure to the user.

  • Retry budgets

    Limit retries to a share of normal traffic. During an outage, the extra load then stays bounded instead of doubling or tripling.

  • Circuit breakers

    Stop calling a service that keeps failing, so that it receives no traffic while it recovers.

The timeout tells a caller when to give up. The retry count and budget decide how often to try again. The circuit breaker decides when to stop altogether. Each one is needed, because a retry rule alone cannot tell when a service is down for good.

What Are Backoff and Jitter?

Exponential backoff makes the wait grow after each failure. The wait should also have a cap, which is a maximum value. One write-up notes that if the wait is not capped, even ten doublings produce a delay of almost half an hour. Jitter then adds randomness, so clients that failed together do not retry together. Full jitter picks a random wait between zero and the capped delay for that attempt.

Python
BASE_MS = 100
CAP_MS = 500
MAX_ATTEMPTS = 4

for attempt in range(MAX_ATTEMPTS):
    ceiling = min(CAP_MS, BASE_MS * 2 ** attempt)
    print(f'retry {attempt + 1}: wait up to {ceiling} ms')

Output

retry 1: wait up to 100 ms
retry 2: wait up to 200 ms
retry 3: wait up to 400 ms
retry 4: wait up to 500 ms

The sample prints the ceiling for each wait. It stays fixed so its output is always the same. A real client would pick a random wait below each ceiling, which is what full jitter does. The cap stops the fourth wait from growing to 800 milliseconds.

Tip

Jitter must be large enough to matter. A few milliseconds of randomness added to a wait of many seconds spreads almost nothing. The random part should be similar in size to the wait itself.

Circuit Breakers

A circuit breaker sits in front of a remote call and watches recent results. It works like an electrical breaker and moves between three states. Its job is to stop the caller from repeatedly trying an operation that is likely to fail.

  • Closed

    Calls go through normally. The breaker counts recent failures, and the count resets on a schedule, so occasional errors do not trip it.

  • Open

    The failure count has passed the set limit. Calls now fail at once, without reaching the service, so the service gets room to recover.

  • Half-open

    After a timer ends, a limited number of trial calls are let through. If they succeed, the breaker closes again. If one fails, the breaker opens again.

A breaker also protects the caller. Without one, each call waits for a timeout, and those waiting requests hold resources. Microsoft's guidance warns that this can exhaust memory, threads, and database connections, and that unrelated parts of the system can fail too. The breaker fails fast instead. The half-open state matters as well, because a service that is recovering can be flooded and fail again if every waiting request is sent at once.

When Retrying Is Safe

Retries help only when the failure is likely to pass. Before retrying, sort the error. Then decide whether a second try has any chance of working.

  • Retry transient errors

    Timeouts, connection resets, 503 Service Unavailable (the server is busy or down for now), and 429 Too Many Requests (you are sending too much) may pass, so a later attempt can help.

  • Skip permanent errors

    A 400 Bad Request means the request itself is wrong, so it fails the same way every time. A 404 Not Found means the target is missing, and waiting will not bring it back.

  • Respect Retry-After

    When a server sends a Retry-After header, it says how long to wait. Use that time instead of your own guess.

  • Make the repeat safe

    Retry only operations that are safe to repeat. A first attempt may have succeeded even though the caller saw a timeout, so a repeated payment could charge twice.

The last point matters most for writes. A timeout does not tell the caller whether the work happened. Only retry a write when the operation is idempotent, meaning a second attempt cannot do harm. Designing that is covered in the Idempotent Operations lesson.

Tip

A 429 response with a Retry-After header is a clear signal to slow down. Treat it as a rule to follow, not as a cue to try again straight away.

Classify Before You Retry

A client can keep the retry rule in one place. The sample below decides, from the status code alone, whether a response is worth another try. Timeouts are not status codes, so a real client would also treat them as retryable.

Python
RETRYABLE = {429, 503}

def should_retry(status):
    return status in RETRYABLE

for status in [400, 404, 429, 503]:
    print(status, should_retry(status))

Output

400 False
404 False
429 True
503 True

The rule is short on purpose. A request that is wrong or points at something missing should fail once, and the caller should show the error. Only temporary failures earn a retry, and even then the retry should wait, as the earlier sections explained.

Quick Check on Error Rules

Quick check

A photo upload app gets four errors. 400: the file is too large. 404: the album was deleted. 503: the service is busy. 429: too many requests, with Retry-After set to 30 seconds. Which plan is best?

Key Takeaways

A retry storm happens when retries add load to a service that is already in trouble. It forms when a small fault triggers retries, retries stack across layers, and the overloaded service fails further in a feedback loop.

To prevent it, set timeouts, cap the number of attempts, use retry budgets, and wait with exponential backoff plus jitter. Put a circuit breaker in front of failing services. Before any retry, check that the error is transient and the operation is safe to repeat, and respect Retry-After.

Tip

Remember the rule in one line: a retry is a request for more work at the worst possible time. Retry rarely, wait longer each time, add random spread, and stop when the service says it is down.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.