Resiliency: Keep Working Through Failure, Not Just Avoiding It
Learn why parts of a system will fail, and how to design so users see a slowdown or fewer features instead of a full outage.
In short
Resiliency is designing a system to keep working through failure, so users see a slowdown or fewer features instead of a full outage.
11 min readFree
What You Will Learn
Every system people use has parts that break. A server can crash, a network link can drop messages, and another team's service can slow down. Resiliency is the design skill that keeps the whole product usable when this happens. In this lesson you will learn what the word means, how it differs from availability, a basic checklist you can use on almost any design, and a worked example. Read the key words first, then move through the sections in order.
Key words
- Resiliency
- A system's ability to keep working, or to recover quickly, when parts of it fail or become overloaded.
- Fault
- A flaw or event that makes one part of a system behave wrongly, such as a crash or a lost message.
- Failure
- What the user notices when a fault gets through and the product stops doing its job.
- Dependency
- Another service, database, or outside system that your part needs in order to do its work.
- Timeout
- A rule that stops waiting for an answer after a set amount of time.
- Fallback
- A backup answer or backup behavior that you use when the normal call fails.
- Graceful degradation
- Keeping the core of a product working, with fewer features, when some parts fail.
- Chaos engineering
- Breaking parts of a system on purpose, in a test setting, to see how it reacts.
What Does Resiliency Mean?
Resiliency means bouncing back. In system design it describes a product that keeps serving people when something goes wrong, or that returns to normal soon after. Engineers do not expect a perfect system. They expect parts to fail, and they plan what users will still get when that happens.
Notice that the goal is the experience, not a perfect machine room.
Stay useful
When one part breaks, the rest of the product should keep doing its main job.
Bounce back
After a fault, the system should return to normal quickly, with little help from a person.
Shrink the damage
A small failure should stay small. It should not spread into a total outage.
Why Are Failures Normal?
A distributed system is one made of many machines and services that talk to each other over a network. In such a system, things go wrong all the time. Servers stop, network links lose messages, and calls to other services run out of time. Good designers treat these events as routine, not as rare surprises. One paper on distributed computing notes that many issues can appear when many nodes, meaning separate machines or programs, interact. Unreliable networks can lose messages.
Network drops
A message between two services can get lost, so the caller never hears back.
Slow dependencies
A service you call may still be running but answer very slowly. Your app can freeze while it waits.
Crashed instances
An instance is one running copy of a program. When one copy crashes, the others should take over its work.
Traffic spikes
A sudden rush of users can overload a system that worked fine the day before.
You cannot prevent every one of these events. So the design goal changes. You plan how the system will notice a problem, how it will keep the problem from spreading, and how it will recover. The Azure failure mode analysis guidance makes the same point: failures still happen, no matter how many layers of protection you add.
Tip
For every part of your design, ask two questions: What happens if this stops? What happens if this gets slow? Good answers to those two questions are worth more than a promise that the part will never fail.
A simple way to organize this thinking is a four-step loop. You expect the failure, detect it, contain it, and recover from it. Each step has its own design choices, and later lessons on specific patterns go deeper.
What Does Expect, Detect, Contain, Recover Mean?
Expect
Assume each dependency can fail, and write down the ways it might fail.
Detect
Notice the failure quickly, using timing checks, error counts, or signals that show a part is unhealthy.
Contain
Stop the broken part from pulling down the rest, for example by refusing to wait forever for it.
Recover
Return to normal, or close to normal, once the broken part comes back or traffic moves to a healthy copy.
Detection matters more than it first appears. Azure's reliability guidance says failure detection is essential, because a failure you never notice cannot be planned around.
How Is Resiliency Different from Availability?
Availability and resiliency sound alike, and students often mix them up. Availability asks how much of the time a system is up and able to answer requests. People usually measure it as a share of time over a period. Resiliency asks a different question: when faults hit, how well does the system keep working, and how fast does it bounce back?
A system can post strong uptime numbers and still be fragile. Imagine a store that is up almost all the time, but whenever its payment service slows down, every page freezes. The uptime record looks good, yet customers have a terrible time. Resiliency is about the experience during trouble, not only the total time the lights stayed on.
Resiliency depends on several tools working together. The main ones are listed below.
Graceful degradation
Keep the core job going with fewer features when a part fails, instead of failing the whole page.
Redundancy
Keep spare copies of parts, or spare capacity, so that losing one piece does not stop the service.
Timeouts
Stop waiting for a slow answer after a set time. Without a limit, a waiting call can hold on to memory, threads, and connections for a long time.
Recovery
Bring the system back to working order after a fault, such as moving traffic to a healthy copy of a service.
Quick check
A video site has a Similar Videos box that is powered by a recommendations service. The service crashes. The video still plays, but the box below it is empty. Which idea best describes what the team built?
What Is a Basic Resiliency Checklist?
Use this three-step checklist on any design. It is basic on purpose. Later lessons add depth through specific patterns.
- 1
Identify critical paths and dependencies
A critical path is the chain of steps a user must finish for the product's main job, such as checkout or playing a video. List every other service or database each step needs.
- 2
Decide what happens when each one fails
For each dependency, choose a response: a fallback, a degraded view, or a fast failure.
- 3
Test those failures on purpose
Turn the dependency off in a test setting and watch what the product does. Fix the gaps you find, then test again.
Map the Critical Paths
Start with the main jobs your users care about. For a shopping app, that might be browsing, adding to a cart, and paying. Write down each step, then mark every call that step makes to another part. A slow call in the middle of checkout can stop the whole checkout, so mark those calls first. Azure's failure mode analysis guidance assumes teams have already prioritized flows by how critical they are.
Choose a Failure Response
Each dependency needs a plan for failure. Most plans fall into one of three kinds.
Fallback
Use a backup answer when the normal call fails, such as a saved value or a simple default estimate.
Degrade
Keep the page working, but hide or simplify the part that failed.
Fail fast
Return a clear error right away, instead of waiting, when the work cannot be done safely.
Test Failures on Purpose
A plan you have never run is only a guess. Shut down a dependency in a test environment, then check the results. Does the page still load? Does the user see a clear message? Do the other features still respond? When a test reveals a new problem, update your list of failure points.
Tip
Start small. Test one dependency at a time before you try to break several at once. This habit is the core of chaos engineering, and it works best when the team agrees on what success looks like first.
Quick check
A grocery app's cart page shows a full error whenever its coupon service is switched off in a test. Coupons are only a small discount. The team wants the next step. What should they do?
Worked Example: A Home Screen With a Failed Recommendation Service
Streaming services are a useful case to study. That article does not explain how Netflix handles failures inside its system. So the steps below are an illustration of the checklist applied to a streaming home screen, not a report of how Netflix built its product.
- 1
Name the critical path
Opening the home screen and starting a title is the main job. Picking which rows to show is important, but it is not the main job.
- 2
List the dependencies
A personalized recommendations service supplies rows picked for this viewer. A simple list of popular titles is a separate, lighter source of rows.
- 3
Choose the failure response
If the recommendations service fails, use a fallback: show a generic popular-titles row in its place. The personal rows are simply left out, which is a degraded view.
- 4
Add a timeout
Give the recommendations call a short time limit. If the limit passes, the screen stops waiting and uses the fallback row.
- 5
Test on purpose
In a test setting, switch off the recommendations service. Check that the home screen still loads and that the user can still press play.
Tip
The goal is a slightly less personal home screen, not an error page. A user who sees popular titles can keep watching. A user who sees an error page cannot.
A Fallback in Code
This short Python sample shows the same idea in code. A tax service call fails with a timeout error, which is Python's name for a call that ran out of time. The function then returns a simple estimate, so the checkout can keep going. The sample uses only the standard library and prints one line.
def tax_estimate(total):
return round(total * 0.08, 2)
def get_tax(total, tax_service):
try:
return tax_service(total)
except TimeoutError:
return tax_estimate(total)
def broken_tax_service(total):
raise TimeoutError('no answer in time')
print(get_tax(50.0, broken_tax_service))
Output
4.0
Check Your Understanding
Read this final scenario and decide what you would do first. It uses every idea in the lesson.
Quick check
A shopping app's checkout calls a slow third-party tax service. When that service hangs, the whole app freezes for every user, including people who are only browsing. Which resiliency idea would you apply first?
Key Takeaways
Expect failure
Plan for network drops, slow services, crashed copies, and traffic spikes, because you cannot prevent them all.
Protect the core job
Use timeouts, fallbacks, and graceful degradation so the main job of the product survives a broken part.
Test on purpose
Break dependencies in a safe setting, compare the results with what you expected, and fix what you find.
Sources
- Fault Tolerance and Resilience Patterns in Distributed Systems (Medium article)
- Graceful degradation in practice: how FeatureOps builds real resilience (Unleash blog)
- Distributed Computing From First Principles (arXiv paper, section 3.4)
- Reliability & Fault Tolerance in System Design: Why Systems Must Expect Failure (Medium article)
- Resilience and Error Handling (DesignGurus system design guide)
- Architecture strategies for performing failure mode analysis (Microsoft Azure Well-Architected Framework)
- Recommendations for performing failure mode analysis (Microsoft Azure Well-Architected Framework, design resiliency guide)
- Architecture strategies for designing a reliability testing strategy (Microsoft Azure Well-Architected Framework)
- Design review checklist for Reliability (Microsoft Azure Well-Architected Framework)
- Azure Well-Architected Framework: The 5 Pillars (ProsperOps blog)
- Netflix just killed a feature that seemed like a great idea (Tom's Guide article, describes the home screen rows)
Last updated on 18 November 2026
Related lessons
- System designClaim Check: Send a Small Ticket Through the Queue, Keep the Big Payload Outside
- System designMessaging: Sending Small Messages Through a Queue Instead of Direct Calls
- System designSequential Convoy: Keep Related Messages in Order Without Stopping Everything
- System designAnti-Corruption Layer: A Translator That Keeps Old Models Out of Your Design
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.