Reliability Patterns: Expect, Detect, Contain, and Recover From Failure
Failures are normal in distributed systems. Learn the four families of reliability patterns, how to choose among them, and how they work together.
In short
Reliability patterns help a distributed system expect, detect, contain, and recover from failure, since failures are normal and four families of patterns cover them.
10 min readFree
What You Will Learn
Computers fail, and networks fail too. In a system design, many parts run on many machines and talk over a network, so something is always at risk of breaking. Reliability patterns are the design techniques that keep a system working correctly when parts of it fail.
In this lesson you will learn the four families of these patterns, the problem each family solves, and how to choose and combine them without making things worse. This is an overview. Each pattern has its own lesson with the full details, so here you learn what each one is for and how they fit together.
Key words
- Failure
- A part of the system stops doing its job correctly, either fully or in part.
- Distributed system
- An app made of several parts, on several machines, that talk to each other over a network.
- Reliability pattern
- A tested design technique that limits harm when something fails.
- Health check
- A simple test that asks a part whether it is still working.
- Timeout
- The longest time you wait for a reply before you give up on it.
- Retry
- Sending the same request again after a failure, in case the problem was brief.
- Circuit breaker
- A switch that stops calls to a service that keeps failing, for a while.
- Bulkhead
- A wall between parts, so one failing part cannot sink the others. The name comes from the sealed rooms in a ship.
- Throttling
- Limiting how many requests a caller may send in a given period.
- Load leveling
- Using a queue to turn a sudden burst of work into a steady flow.
- Compensating transaction
- An action that undoes the earlier steps of a job that could not finish.
- Fallback
- A simpler, backup answer shown when the normal answer is not available.
- Blast radius
- How much of the system one failure can touch.
Why Are Failures Normal?
A common beginner belief is that a system can be made fail-proof. Distributed systems do not work that way. Azure's cloud design guidance lists the belief that the network is reliable among the common false assumptions that lead to weak designs. It says patterns do not remove these false beliefs, but they help teams see them and offer ways to compensate.
Network drops
A message can get lost, or a connection can hang with no reply at all.
Overloaded services
A service that gets too many requests slows down, and then starts failing for every user.
Bad deploys
A new release with a bug can break one feature, or the whole app, for every user at once.
So the goal of reliability is not to prevent every failure. You cannot do that. Azure's reliability guidance says that a dependable design focuses on keeping the system available, protecting itself, recovering, and keeping data correct when parts go wrong.
The four stages below are the simplest way to hold the whole topic in your head.
- 1
Expect
Assume that any part can fail, and ask which failures would hurt the most.
- 2
Detect
Notice the failure quickly, with checks and measurements, so the team knows it happened.
- 3
Contain
Stop the failure from spreading into healthy parts of the system.
- 4
Recover
Bring the job back, by retrying, switching to a backup, or undoing partial work.
What Are the Four Pattern Families?
Reliability patterns fall into four families. Each family answers a different question. Azure's reliability pattern list describes its goals as workload availability, self-preservation, recovery, data and processing integrity, and containment of malfunctions. The families below group those goals into four simple jobs.
Detect
Notice that a part is unhealthy. Health checks are the usual tool.
Recover
Bring the work back after a failure. Retry and compensating transactions are examples.
Contain
Keep one failure from taking down the rest. Bulkhead and circuit breaker are examples.
Protect capacity
Stop demand from going past what a part can handle. Throttling and load leveling are examples.
The next four sub-topics explain each family in more detail.
How Do You Detect Failure with Health Checks?
Detecting problems with health checks. A health check is a question a part answers about its own state, often through a dedicated address. Azure's Health Endpoint Monitoring pattern exposes such an endpoint so the health of a system can be watched.
How Do You Recover with Retry and Compensation?
Recovering with retries and compensation. Many failures are brief. A packet is lost, or a server restarts in seconds. A retry sends the same request again, which can clear a brief problem. Retries need limits, because endless retries add load to a part that is already struggling.
What Do You Do When Retry Is Not Enough?
A compensating transaction solves a different problem. Sometimes a job has several steps, and step three fails after steps one and two already worked. A compensating transaction runs actions that reverse those earlier steps. Azure describes it as a way to recover from failure by reversing the effects of earlier actions. Its full workflow has its own lesson.
Contain With Bulkheads and Circuit Breakers
Containing damage with bulkheads and circuit breakers. A bulkhead separates resources, such as connection pools, memory, and processor time, so that each service has its own share. Azure says this stops one service from using up everything and causing cascading failures across the system. A circuit breaker watches calls to a dependency. When the dependency keeps failing, the breaker stops sending calls for a while, which protects the failing part from more load. Azure notes that breakers are often paired with automatic recovery. Azure's microservices pattern guidance describes the Retry pattern and the Circuit Breaker pattern for resilient service-to-service calls.
Protect Capacity With Throttling and Load Leveling
Protecting capacity with throttling and load leveling. Throttling limits how many requests a caller can send, so a burst cannot swamp a service. Load leveling places a queue between the sender and the worker. The full queue patterns belong to the messaging lessons.
Choosing and Combining Patterns
Choose patterns by the problem, not by the tool you like. Azure's guidance says to start with a specific risk, such as a service that fails under load or a dependency you cannot fully trust. Then check that the pattern's trade-offs are ones your team can accept. Each pattern has trade-offs, so no pattern is free.
- 1
Ask what fails
Name the part that could break, such as a payment service or a database. A fix for a slow service is not always the right fix for a dead one.
- 2
Ask how badly it hurts users
A slow suggestion box is annoying. A failed checkout loses money. Spend the most effort on the work that matters most to the business, as Azure's reliability principles suggest.
- 3
Ask what the fix costs
Every pattern adds code, moving parts, and new ways to go wrong. Add only what the problem needs.
- 4
Test the fix under real failure
A pattern that looks right on paper can behave badly under real load, so test it while the failure is happening on purpose.
Watch Out for Misuse
Watching out for misuse. A reliability pattern used badly can make a failure worse. The classic case is the retry storm. Many callers retry at the same moment against a service that is already overloaded, so the extra requests keep it down.
Tip
Before you add a retry, ask what happens if a thousand callers do the same thing at the same moment. If the answer is that the service falls over, add a retry limit and a circuit breaker first. The course lesson Retry Storm covers this failure in depth.
Worked Example: The Request Ride Screen
This example uses a made-up ride-hailing app, not a real company's code. A rider taps Request ride. Before the booking is confirmed, the app asks the payment service to check the rider's card. Today the payment service is slow, so the app has to decide what to do. Here is how the four ideas work together on that one screen.
- 1
Set a timeout
The app stops waiting after a fixed, short time. Without a timeout, the rider's screen would spin forever and the waiting request would tie up resources.
- 2
Retry once for a brief blip
If the first call times out, the app tries the payment call one more time. Brief problems often clear, so the rider never notices. The app stops after that one retry, because the limit keeps a blip from turning into a storm.
- 3
Open the circuit breaker
After repeated timeouts, the breaker opens. The app stops calling the payment service and fails fast, so riders no longer wait on a service that is clearly down.
- 4
Show a fallback
The screen offers a pay later option. The ride is booked now, and payment is collected afterward. The main job, booking a ride, keeps working while the payment step is degraded.
Notice how the work is divided. The timeout and the single retry handle the brief blip. The breaker handles the longer outage. The fallback keeps the rider's main goal alive. A health check would tell the team that payment is down, and a bulkhead would keep payment calls from using up the whole app. The small program below models the breaker idea. It makes the risky call only until it has failed twice, then answers with the fallback without calling again.
class CircuitBreaker:
def __init__(self, limit):
self.limit = limit
self.failures = 0
def call(self, fn):
if self.failures >= self.limit:
return 'fallback: pay later'
try:
result = fn()
self.failures = 0
return result
except TimeoutError:
self.failures += 1
return 'fallback: pay later'
calls = []
def slow_payment():
calls.append('call')
raise TimeoutError
breaker = CircuitBreaker(limit=2)
for attempt in range(4):
print('attempt', attempt, breaker.call(slow_payment))
print('payment calls made:', len(calls))
Output
attempt 0 fallback: pay later attempt 1 fallback: pay later attempt 2 fallback: pay later attempt 3 fallback: pay later payment calls made: 2
Tip
A fallback is only as good as the rule behind it. Before you ship pay later, agree with the business team what it means, for example when the rider is charged and what happens if the charge fails. Engineers should not decide that alone. Also note that this small model never tries the service again. Real breakers test the service after a wait, which is covered in the circuit breaker lesson.
Check Your Understanding
Quick check
A shopping site's recommendation service goes down, and every product page hangs for 30 seconds waiting on it. The whole site slows to a crawl. Which pattern would you add first, and what should the page show instead?
Quick check
An app retries failed uploads right away, up to ten times, with no limit on total attempts. During a database outage, the retries pile up, and the database never recovers after it comes back. What change would help most?
Where to Go Next
This overview gives you the map. Each of the following topics goes deeper into one part of it.
Health Monitoring
How to catch failing parts before users notice them.
Retry Storm
How retries can take down a struggling system, and how to prevent that.
Back Pressure
How an overloaded system tells its callers to slow down.
Queue-Based Load Leveling
The full pattern for smoothing bursts of traffic, covered in the messaging lessons.
Sources
- Architecture design patterns that support reliability - Microsoft Azure Well-Architected Framework
- Cloud Design Patterns - Azure Architecture Center
- Reliability design principles - Microsoft Azure Well-Architected Framework
- Best Practices in Cloud Applications - Azure Architecture Center
- Design Patterns for Microservices - Azure Architecture Center
- Queue-based load leveling summary in the Azure reliability patterns table (MicrosoftDocs well-architected on GitHub)
- Azure reliability, resiliency, and recoverability: Build continuity by design
- Design principles - AWS Well-Architected Framework (Reliability)
- The 6 Pillars of the AWS Well-Architected Framework (AWS Partner Network blog)
- Well-Architected Reliability: Best practices you can use today (AWS slide deck)
Last updated on 18 November 2026
Related lessons
- Design systemsPattern: Combining Components Into Reusable User Solutions
- System designConsistency Patterns: Weak, Eventual, and Strong Consistency Explained
- System designAvailability Patterns: Fail-Over and Replication to Keep Systems Running
- System designCloud Design Patterns: Reusable Fixes, With Strangler Fig and Sidecar
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.