esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Reliability Patterns: Expect, Detect, Contain, and Recover From Failure

Failures are normal in distributed systems. Learn the four families of reliability patterns, how to choose among them, and how they work together.

In short

Reliability patterns help a distributed system expect, detect, contain, and recover from failure, since failures are normal and four families of patterns cover them.

10 min readFree

What You Will Learn

Computers fail, and networks fail too. In a system design, many parts run on many machines and talk over a network, so something is always at risk of breaking. Reliability patterns are the design techniques that keep a system working correctly when parts of it fail.

In this lesson you will learn the four families of these patterns, the problem each family solves, and how to choose and combine them without making things worse. This is an overview. Each pattern has its own lesson with the full details, so here you learn what each one is for and how they fit together.

Key words

Failure
A part of the system stops doing its job correctly, either fully or in part.
Distributed system
An app made of several parts, on several machines, that talk to each other over a network.
Reliability pattern
A tested design technique that limits harm when something fails.
Health check
A simple test that asks a part whether it is still working.
Timeout
The longest time you wait for a reply before you give up on it.
Retry
Sending the same request again after a failure, in case the problem was brief.
Circuit breaker
A switch that stops calls to a service that keeps failing, for a while.
Bulkhead
A wall between parts, so one failing part cannot sink the others. The name comes from the sealed rooms in a ship.
Throttling
Limiting how many requests a caller may send in a given period.
Load leveling
Using a queue to turn a sudden burst of work into a steady flow.
Compensating transaction
An action that undoes the earlier steps of a job that could not finish.
Fallback
A simpler, backup answer shown when the normal answer is not available.
Blast radius
How much of the system one failure can touch.

Why Are Failures Normal?

A common beginner belief is that a system can be made fail-proof. Distributed systems do not work that way. Azure's cloud design guidance lists the belief that the network is reliable among the common false assumptions that lead to weak designs. It says patterns do not remove these false beliefs, but they help teams see them and offer ways to compensate.

  • Network drops

    A message can get lost, or a connection can hang with no reply at all.

  • Overloaded services

    A service that gets too many requests slows down, and then starts failing for every user.

  • Bad deploys

    A new release with a bug can break one feature, or the whole app, for every user at once.

So the goal of reliability is not to prevent every failure. You cannot do that. Azure's reliability guidance says that a dependable design focuses on keeping the system available, protecting itself, recovering, and keeping data correct when parts go wrong.

The four stages below are the simplest way to hold the whole topic in your head.

  1. 1

    Expect

    Assume that any part can fail, and ask which failures would hurt the most.

  2. 2

    Detect

    Notice the failure quickly, with checks and measurements, so the team knows it happened.

  3. 3

    Contain

    Stop the failure from spreading into healthy parts of the system.

  4. 4

    Recover

    Bring the job back, by retrying, switching to a backup, or undoing partial work.

What Are the Four Pattern Families?

Reliability patterns fall into four families. Each family answers a different question. Azure's reliability pattern list describes its goals as workload availability, self-preservation, recovery, data and processing integrity, and containment of malfunctions. The families below group those goals into four simple jobs.

  • Detect

    Notice that a part is unhealthy. Health checks are the usual tool.

  • Recover

    Bring the work back after a failure. Retry and compensating transactions are examples.

  • Contain

    Keep one failure from taking down the rest. Bulkhead and circuit breaker are examples.

  • Protect capacity

    Stop demand from going past what a part can handle. Throttling and load leveling are examples.

The next four sub-topics explain each family in more detail.

How Do You Detect Failure with Health Checks?

Detecting problems with health checks. A health check is a question a part answers about its own state, often through a dedicated address. Azure's Health Endpoint Monitoring pattern exposes such an endpoint so the health of a system can be watched.

How Do You Recover with Retry and Compensation?

Recovering with retries and compensation. Many failures are brief. A packet is lost, or a server restarts in seconds. A retry sends the same request again, which can clear a brief problem. Retries need limits, because endless retries add load to a part that is already struggling.

What Do You Do When Retry Is Not Enough?

A compensating transaction solves a different problem. Sometimes a job has several steps, and step three fails after steps one and two already worked. A compensating transaction runs actions that reverse those earlier steps. Azure describes it as a way to recover from failure by reversing the effects of earlier actions. Its full workflow has its own lesson.

Contain With Bulkheads and Circuit Breakers

Containing damage with bulkheads and circuit breakers. A bulkhead separates resources, such as connection pools, memory, and processor time, so that each service has its own share. Azure says this stops one service from using up everything and causing cascading failures across the system. A circuit breaker watches calls to a dependency. When the dependency keeps failing, the breaker stops sending calls for a while, which protects the failing part from more load. Azure notes that breakers are often paired with automatic recovery. Azure's microservices pattern guidance describes the Retry pattern and the Circuit Breaker pattern for resilient service-to-service calls.

Protect Capacity With Throttling and Load Leveling

Protecting capacity with throttling and load leveling. Throttling limits how many requests a caller can send, so a burst cannot swamp a service. Load leveling places a queue between the sender and the worker. The full queue patterns belong to the messaging lessons.

Choosing and Combining Patterns

Choose patterns by the problem, not by the tool you like. Azure's guidance says to start with a specific risk, such as a service that fails under load or a dependency you cannot fully trust. Then check that the pattern's trade-offs are ones your team can accept. Each pattern has trade-offs, so no pattern is free.

  1. 1

    Ask what fails

    Name the part that could break, such as a payment service or a database. A fix for a slow service is not always the right fix for a dead one.

  2. 2

    Ask how badly it hurts users

    A slow suggestion box is annoying. A failed checkout loses money. Spend the most effort on the work that matters most to the business, as Azure's reliability principles suggest.

  3. 3

    Ask what the fix costs

    Every pattern adds code, moving parts, and new ways to go wrong. Add only what the problem needs.

  4. 4

    Test the fix under real failure

    A pattern that looks right on paper can behave badly under real load, so test it while the failure is happening on purpose.

Watch Out for Misuse

Watching out for misuse. A reliability pattern used badly can make a failure worse. The classic case is the retry storm. Many callers retry at the same moment against a service that is already overloaded, so the extra requests keep it down.

Tip

Before you add a retry, ask what happens if a thousand callers do the same thing at the same moment. If the answer is that the service falls over, add a retry limit and a circuit breaker first. The course lesson Retry Storm covers this failure in depth.

Worked Example: The Request Ride Screen

This example uses a made-up ride-hailing app, not a real company's code. A rider taps Request ride. Before the booking is confirmed, the app asks the payment service to check the rider's card. Today the payment service is slow, so the app has to decide what to do. Here is how the four ideas work together on that one screen.

  1. 1

    Set a timeout

    The app stops waiting after a fixed, short time. Without a timeout, the rider's screen would spin forever and the waiting request would tie up resources.

  2. 2

    Retry once for a brief blip

    If the first call times out, the app tries the payment call one more time. Brief problems often clear, so the rider never notices. The app stops after that one retry, because the limit keeps a blip from turning into a storm.

  3. 3

    Open the circuit breaker

    After repeated timeouts, the breaker opens. The app stops calling the payment service and fails fast, so riders no longer wait on a service that is clearly down.

  4. 4

    Show a fallback

    The screen offers a pay later option. The ride is booked now, and payment is collected afterward. The main job, booking a ride, keeps working while the payment step is degraded.

Notice how the work is divided. The timeout and the single retry handle the brief blip. The breaker handles the longer outage. The fallback keeps the rider's main goal alive. A health check would tell the team that payment is down, and a bulkhead would keep payment calls from using up the whole app. The small program below models the breaker idea. It makes the risky call only until it has failed twice, then answers with the fallback without calling again.

Python
class CircuitBreaker:
    def __init__(self, limit):
        self.limit = limit
        self.failures = 0

    def call(self, fn):
        if self.failures >= self.limit:
            return 'fallback: pay later'
        try:
            result = fn()
            self.failures = 0
            return result
        except TimeoutError:
            self.failures += 1
            return 'fallback: pay later'

calls = []

def slow_payment():
    calls.append('call')
    raise TimeoutError

breaker = CircuitBreaker(limit=2)
for attempt in range(4):
    print('attempt', attempt, breaker.call(slow_payment))
print('payment calls made:', len(calls))

Output

attempt 0 fallback: pay later
attempt 1 fallback: pay later
attempt 2 fallback: pay later
attempt 3 fallback: pay later
payment calls made: 2

Tip

A fallback is only as good as the rule behind it. Before you ship pay later, agree with the business team what it means, for example when the rider is charged and what happens if the charge fails. Engineers should not decide that alone. Also note that this small model never tries the service again. Real breakers test the service after a wait, which is covered in the circuit breaker lesson.

Check Your Understanding

Quick check

A shopping site's recommendation service goes down, and every product page hangs for 30 seconds waiting on it. The whole site slows to a crawl. Which pattern would you add first, and what should the page show instead?

Quick check

An app retries failed uploads right away, up to ten times, with no limit on total attempts. During a database outage, the retries pile up, and the database never recovers after it comes back. What change would help most?

Where to Go Next

This overview gives you the map. Each of the following topics goes deeper into one part of it.

  • Health Monitoring

    How to catch failing parts before users notice them.

  • Retry Storm

    How retries can take down a struggling system, and how to prevent that.

  • Back Pressure

    How an overloaded system tells its callers to slow down.

  • Queue-Based Load Leveling

    The full pattern for smoothing bursts of traffic, covered in the messaging lessons.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.