# Retry Storm: How Retries Can Take Down a Struggling System

> When many clients retry failed requests at the same time, the extra load can stop a weak service from recovering. Here is how that forms and how to stop it.

**In short:** A retry storm happens when many clients retry failed requests at the same time, adding load that stops a struggling service from recovering.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/retry-storm

## What You Will Learn

Retries are meant to make software more reliable. If a request fails, sending it again may work. But when a service is already struggling, many retries at the same moment can make things worse. This lesson explains that problem, which system designers call a retry storm. In this course it is treated as a performance antipattern, which means a common design mistake that looks sensible but harms the system. You will see how a small slowdown grows into a large outage, how to stop that growth, and how to tell which errors are worth retrying.

- **Retry:** Sending the same request again after it fails.
- **Retry storm:** Many clients or services retrying at the same time, adding load to a system that is already struggling, so it cannot recover.
- **Load:** The amount of work a system must handle in a set time, such as requests per second.
- **Layer or tier:** One part of a chain of software, such as a phone app, an API, an order service, or a payments service.
- **Timeout:** A time limit on how long a caller waits for an answer before it gives up.
- **Transient error:** A temporary failure, such as a timeout or a busy server, that may pass on its own.
- **Permanent error:** A failure that will happen again on every try, such as a request with a wrong format.
- **Exponential backoff:** Waiting longer after each failed try, for example a wait that doubles each time.
- **Jitter:** A small random change added to each wait, so many clients do not act at the same moment.
- **Circuit breaker:** A guard that stops calls to a service that keeps failing, and returns an error at once instead.
- **Retry budget:** A limit on how many retries are allowed, usually as a share of normal traffic.
- **Idempotent:** Safe to repeat. Doing the same request twice has the same result as doing it once.

## How Does a Retry Storm Form?

A retry storm usually starts small. A service becomes slow, or returns errors for a short time. The cause might be a busy database, a new software release, or a brief network problem. Callers see timeouts, because their requests did not finish in time. Each caller then sends the request again. Retries are meant to help, because a short fault often passes. But the extra attempts arrive while the service is still busy, so it receives even more work than before.

- **The spark:** A small slowdown or a short burst of errors makes some requests fail or time out.
- **The multiplier:** Each failed request is sent again, so the same work arrives several times. Load rises just when the service has the least spare room to handle it.
- **The feedback loop:** The service slows further under the extra load, more requests fail, and more retries follow. The loop can keep going even after the original fault has ended.

The last step is the dangerous one. A retry storm is retries generating more load than the original traffic, which keeps a service down after the fault that started it has passed.

> **Tip:** A retry is not free. Each retry is extra work sent to a service that may already be overloaded. Ask whether the extra work will help the service recover, or hurt it.

### How Do Retries Stack Across Layers?

Most real systems have several layers. A phone app may call an API gateway, which calls an order service, which calls a payments service. If every layer retries when the layer below it fails, the retries multiply. Retrying at all layers multiplies the number of retries rather than adding to them. Suppose each of three tiers tries a call three times. One user action then reaches the bottom service 27 times.

```python
tiers = 3
attempts_per_tier = 3
print(attempts_per_tier ** tiers)
```

Output:

```text
27

```

The same arithmetic works for any number of tiers. Each extra layer that retries multiplies the traffic again. This is why a designer should count the layers before adding any retry logic, and choose one layer to do the retrying.

> **Tip:** Retry at one layer only. The layer closest to the failing call is usually the best place, and the other layers should pass the error up without retrying it again.

## Why Does the Storm Keep Going?

Clients that fail at the same moment tend to retry at the same moment too. If a thousand clients all wait exactly one second and then send again, the service gets a wave of a thousand calls, then silence, then another wave. Each wave can knock the service down again. Immediate retries make this worse, because they return while the service is still in trouble.

**Quick check:** A service handles 500 requests per second. For 20 seconds it returns errors. Each client retries immediately, up to 2 times, and every request fails during that window. About how many requests per second reach the service during the outage?

A. 500 requests per second, because retries replace the first attempt
B. 1,000 requests per second, because each request is sent twice
C. 1,500 requests per second, because each request makes three attempts
D. 10,000 requests per second, because each retry doubles the load

**Answer:** C. Each original request makes a first try and two retries, so three attempts arrive for every request. Three times 500 is 1,500. The retries are added to the original traffic, not swapped in for it.

## Worked Example: A Food-Delivery Order Screen

Consider a mobile food-delivery app. When a customer taps Place order, the app calls a payments service to charge the card. Follow what happens when payments slows down, then see how two fixes change the result.

1. **Payments slows down:** The payments service starts answering slowly. Many calls still finish, but some pass the order screen's timeout.
2. **Phones retry every second:** The app treats each timeout as a failure and sends the charge request again. Every phone retries about once a second, with no wait that grows over time.
3. **Customers tap again:** The screen shows no result, so thousands of users tap Place order again. Each tap adds another request on top of the retries already running.
4. **Payments goes fully down:** Payments now receives several times its normal traffic. It slows further, more calls time out, and more retries follow. The service stops answering altogether.
5. **Add backoff with jitter:** Each phone now waits longer after each failure, and a random jitter spreads those waits apart. The retries no longer hit payments in one synchronized wave.
6. **Add a circuit breaker:** After repeated failures, the app stops calling payments for a cooldown period. The order screen shows a clear message at once. When the cooldown ends, a few trial calls are allowed through. If they succeed, normal traffic resumes.
7. **Payments recovers:** With fewer and spaced-out calls, payments has room to catch up. The app no longer makes its own outage larger.

> **Tip:** Before adding retries to Place order, make sure a repeated tap cannot charge a card twice. That is what idempotent design is for, and it has its own lesson.

## How Do You Prevent a Retry Storm?

No single setting stops a retry storm. Designers combine several habits, so that each one covers a gap left by the others.

- **Timeouts:** Set a time limit on every remote call. A caller that waits forever holds memory, threads, and connections, and a stalled service can take the caller down with it.
- **Capped retry counts:** Allow only a few attempts per request. After the limit, stop and return a clear failure to the user.
- **Retry budgets:** Limit retries to a share of normal traffic. During an outage, the extra load then stays bounded instead of doubling or tripling.
- **Circuit breakers:** Stop calling a service that keeps failing, so that it receives no traffic while it recovers.

The timeout tells a caller when to give up. The retry count and budget decide how often to try again. The circuit breaker decides when to stop altogether. Each one is needed, because a retry rule alone cannot tell when a service is down for good.

### What Are Backoff and Jitter?

Exponential backoff makes the wait grow after each failure. The wait should also have a cap, which is a maximum value. One write-up notes that if the wait is not capped, even ten doublings produce a delay of almost half an hour. Jitter then adds randomness, so clients that failed together do not retry together. Full jitter picks a random wait between zero and the capped delay for that attempt.

```python
BASE_MS = 100
CAP_MS = 500
MAX_ATTEMPTS = 4

for attempt in range(MAX_ATTEMPTS):
    ceiling = min(CAP_MS, BASE_MS * 2 ** attempt)
    print(f'retry {attempt + 1}: wait up to {ceiling} ms')
```

Output:

```text
retry 1: wait up to 100 ms
retry 2: wait up to 200 ms
retry 3: wait up to 400 ms
retry 4: wait up to 500 ms

```

The sample prints the ceiling for each wait. It stays fixed so its output is always the same. A real client would pick a random wait below each ceiling, which is what full jitter does. The cap stops the fourth wait from growing to 800 milliseconds.

> **Tip:** Jitter must be large enough to matter. A few milliseconds of randomness added to a wait of many seconds spreads almost nothing. The random part should be similar in size to the wait itself.

### Circuit Breakers

A circuit breaker sits in front of a remote call and watches recent results. It works like an electrical breaker and moves between three states. Its job is to stop the caller from repeatedly trying an operation that is likely to fail.

- **Closed:** Calls go through normally. The breaker counts recent failures, and the count resets on a schedule, so occasional errors do not trip it.
- **Open:** The failure count has passed the set limit. Calls now fail at once, without reaching the service, so the service gets room to recover.
- **Half-open:** After a timer ends, a limited number of trial calls are let through. If they succeed, the breaker closes again. If one fails, the breaker opens again.

A breaker also protects the caller. Without one, each call waits for a timeout, and those waiting requests hold resources. Microsoft's guidance warns that this can exhaust memory, threads, and database connections, and that unrelated parts of the system can fail too. The breaker fails fast instead. The half-open state matters as well, because a service that is recovering can be flooded and fail again if every waiting request is sent at once.

## When Retrying Is Safe

Retries help only when the failure is likely to pass. Before retrying, sort the error. Then decide whether a second try has any chance of working.

- **Retry transient errors:** Timeouts, connection resets, 503 Service Unavailable (the server is busy or down for now), and 429 Too Many Requests (you are sending too much) may pass, so a later attempt can help.
- **Skip permanent errors:** A 400 Bad Request means the request itself is wrong, so it fails the same way every time. A 404 Not Found means the target is missing, and waiting will not bring it back.
- **Respect Retry-After:** When a server sends a Retry-After header, it says how long to wait. Use that time instead of your own guess.
- **Make the repeat safe:** Retry only operations that are safe to repeat. A first attempt may have succeeded even though the caller saw a timeout, so a repeated payment could charge twice.

The last point matters most for writes. A timeout does not tell the caller whether the work happened. Only retry a write when the operation is idempotent, meaning a second attempt cannot do harm. Designing that is covered in the Idempotent Operations lesson.

> **Tip:** A 429 response with a Retry-After header is a clear signal to slow down. Treat it as a rule to follow, not as a cue to try again straight away.

### Classify Before You Retry

A client can keep the retry rule in one place. The sample below decides, from the status code alone, whether a response is worth another try. Timeouts are not status codes, so a real client would also treat them as retryable.

```python
RETRYABLE = {429, 503}

def should_retry(status):
    return status in RETRYABLE

for status in [400, 404, 429, 503]:
    print(status, should_retry(status))
```

Output:

```text
400 False
404 False
429 True
503 True

```

The rule is short on purpose. A request that is wrong or points at something missing should fail once, and the caller should show the error. Only temporary failures earn a retry, and even then the retry should wait, as the earlier sections explained.

## Quick Check on Error Rules

**Quick check:** A photo upload app gets four errors. 400: the file is too large. 404: the album was deleted. 503: the service is busy. 429: too many requests, with Retry-After set to 30 seconds. Which plan is best?

A. Retry all four right away, up to five times each
B. Retry the 503 and the 429 later, waiting 30 seconds for the 429, and do not retry the 400 or the 404
C. Retry only the 400, because a smaller file might work
D. Retry nothing, because any retry adds load to the service

**Answer:** B. The 400 and 404 describe a request that is wrong or aimed at something gone, so they fail the same way each time. The 503 and 429 are temporary, and the Retry-After header tells the app how long to wait.

## Key Takeaways

A retry storm happens when retries add load to a service that is already in trouble. It forms when a small fault triggers retries, retries stack across layers, and the overloaded service fails further in a feedback loop.

To prevent it, set timeouts, cap the number of attempts, use retry budgets, and wait with exponential backoff plus jitter. Put a circuit breaker in front of failing services. Before any retry, check that the error is transient and the operation is safe to repeat, and respect Retry-After.

> **Tip:** Remember the rule in one line: a retry is a request for more work at the worst possible time. Retry rarely, wait longer each time, add random spread, and stop when the service says it is down.

## Sources

- [Microsoft Learn: Circuit Breaker pattern (Azure Architecture Center)](https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker)
- [Lumigo: Amazon Builders' Library in focus, timeouts, retries, and backoff with jitter](https://lumigo.io/blog/amazon-builders-library-in-focus-1-timeouts-retries-and-backoff-with-jitter/)
- [OneUptime: How to Fix 'Retry Storm' Issues in Microservices](https://oneuptime.com/blog/post/2026-01-24-retry-storm-microservices/view)
- [DEV Community: Retries without a retry storm, timeouts, backoff, jitter and circuit breakers](https://dev.to/ayushwing/retries-without-a-retry-storm-timeouts-backoff-jitter-and-circuit-breakers-48i2)
- [DEV Community: Circuit Breakers in Production, Timeouts, Retry Storms, and Failure Recovery](https://dev.to/zeeadil/circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery-4cj1)
- [DEV Community: Your First Production Incident, on Localhost (a retry storm lab)](https://dev.to/soaresdiogo/your-first-production-incident-on-localhost-an-open-source-lab-1l1n)
- [Vetora Labs: Retry with Exponential Backoff and Jitter](https://www.vetoralabs.com/system-design/concepts/reliability/retry-backoff-jitter)
- [Nexgismo: Retry Storm Prevention, Backoff, Jitter and Retry Budgets](https://www.nexgismo.com/blog/retry-storm-prevention-backoff-jitter-retry-budgets)
- [ediverse: Pattern, Retry With Jitter (anti retry storm)](https://www.ediverse.io/en/patterns/retry-with-jitter/)
- [Sujeet Jaiswal: Exponential Backoff and Retry Strategy](https://sujeet.pro/articles/exponential-backoff-retry-strategy)
- [Microsoft Learn: Application resiliency patterns (.NET)](https://learn.microsoft.com/en-us/dotnet/architecture/cloud-native/application-resiliency-patterns)
- [The Backend Tech Lead: The Best System Design Decisions Are Often Invisible](https://thebackendtechlead.substack.com/p/the-best-system-design-decisions)
- [JP Blogs: Timeouts, Retries & Backoff with Jitter (system design lesson)](https://jp-blogs.vercel.app/system-design/lesson-41)
- [CodeReliant: Retries, Backoff and Jitter](https://www.codereliant.io/p/retries-backoff-jitter)
