# Resiliency: Keep Working Through Failure, Not Just Avoiding It

> Learn why parts of a system will fail, and how to design so users see a slowdown or fewer features instead of a full outage.

**In short:** Resiliency is designing a system to keep working through failure, so users see a slowdown or fewer features instead of a full outage.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/resiliency

## What You Will Learn

Every system people use has parts that break. A server can crash, a network link can drop messages, and another team's service can slow down. Resiliency is the design skill that keeps the whole product usable when this happens. In this lesson you will learn what the word means, how it differs from availability, a basic checklist you can use on almost any design, and a worked example. Read the key words first, then move through the sections in order.

- **Resiliency:** A system's ability to keep working, or to recover quickly, when parts of it fail or become overloaded.
- **Fault:** A flaw or event that makes one part of a system behave wrongly, such as a crash or a lost message.
- **Failure:** What the user notices when a fault gets through and the product stops doing its job.
- **Dependency:** Another service, database, or outside system that your part needs in order to do its work.
- **Timeout:** A rule that stops waiting for an answer after a set amount of time.
- **Fallback:** A backup answer or backup behavior that you use when the normal call fails.
- **Graceful degradation:** Keeping the core of a product working, with fewer features, when some parts fail.
- **Chaos engineering:** Breaking parts of a system on purpose, in a test setting, to see how it reacts.

## What Does Resiliency Mean?

Resiliency means bouncing back. In system design it describes a product that keeps serving people when something goes wrong, or that returns to normal soon after. Engineers do not expect a perfect system. They expect parts to fail, and they plan what users will still get when that happens.

Notice that the goal is the experience, not a perfect machine room.

- **Stay useful:** When one part breaks, the rest of the product should keep doing its main job.
- **Bounce back:** After a fault, the system should return to normal quickly, with little help from a person.
- **Shrink the damage:** A small failure should stay small. It should not spread into a total outage.

## Why Are Failures Normal?

A distributed system is one made of many machines and services that talk to each other over a network. In such a system, things go wrong all the time. Servers stop, network links lose messages, and calls to other services run out of time. Good designers treat these events as routine, not as rare surprises. One paper on distributed computing notes that many issues can appear when many nodes, meaning separate machines or programs, interact. Unreliable networks can lose messages.

- **Network drops:** A message between two services can get lost, so the caller never hears back.
- **Slow dependencies:** A service you call may still be running but answer very slowly. Your app can freeze while it waits.
- **Crashed instances:** An instance is one running copy of a program. When one copy crashes, the others should take over its work.
- **Traffic spikes:** A sudden rush of users can overload a system that worked fine the day before.

You cannot prevent every one of these events. So the design goal changes. You plan how the system will notice a problem, how it will keep the problem from spreading, and how it will recover. The Azure failure mode analysis guidance makes the same point: failures still happen, no matter how many layers of protection you add.

> **Tip:** For every part of your design, ask two questions: What happens if this stops? What happens if this gets slow? Good answers to those two questions are worth more than a promise that the part will never fail.

A simple way to organize this thinking is a four-step loop. You expect the failure, detect it, contain it, and recover from it. Each step has its own design choices, and later lessons on specific patterns go deeper.

### What Does Expect, Detect, Contain, Recover Mean?

- **Expect:** Assume each dependency can fail, and write down the ways it might fail.
- **Detect:** Notice the failure quickly, using timing checks, error counts, or signals that show a part is unhealthy.
- **Contain:** Stop the broken part from pulling down the rest, for example by refusing to wait forever for it.
- **Recover:** Return to normal, or close to normal, once the broken part comes back or traffic moves to a healthy copy.

Detection matters more than it first appears. Azure's reliability guidance says failure detection is essential, because a failure you never notice cannot be planned around.

## How Is Resiliency Different from Availability?

Availability and resiliency sound alike, and students often mix them up. Availability asks how much of the time a system is up and able to answer requests. People usually measure it as a share of time over a period. Resiliency asks a different question: when faults hit, how well does the system keep working, and how fast does it bounce back?

A system can post strong uptime numbers and still be fragile. Imagine a store that is up almost all the time, but whenever its payment service slows down, every page freezes. The uptime record looks good, yet customers have a terrible time. Resiliency is about the experience during trouble, not only the total time the lights stayed on.

Resiliency depends on several tools working together. The main ones are listed below.

- **Graceful degradation:** Keep the core job going with fewer features when a part fails, instead of failing the whole page.
- **Redundancy:** Keep spare copies of parts, or spare capacity, so that losing one piece does not stop the service.
- **Timeouts:** Stop waiting for a slow answer after a set time. Without a limit, a waiting call can hold on to memory, threads, and connections for a long time.
- **Recovery:** Bring the system back to working order after a fault, such as moving traffic to a healthy copy of a service.

**Quick check:** A video site has a Similar Videos box that is powered by a recommendations service. The service crashes. The video still plays, but the box below it is empty. Which idea best describes what the team built?

A. Strong consistency, so every page waits until all data copies match
B. Graceful degradation, so the main job keeps working with fewer features
C. Sharding, so each video is stored on its own separate server
D. Chaos engineering, so users never see that anything crashed

**Answer:** B. The core job, playing the video, kept working while a non-essential feature was reduced. That is graceful degradation. The other options describe data storage or testing ideas, not how a product behaves when one part fails.

## What Is a Basic Resiliency Checklist?

Use this three-step checklist on any design. It is basic on purpose. Later lessons add depth through specific patterns.

1. **Identify critical paths and dependencies:** A critical path is the chain of steps a user must finish for the product's main job, such as checkout or playing a video. List every other service or database each step needs.
2. **Decide what happens when each one fails:** For each dependency, choose a response: a fallback, a degraded view, or a fast failure.
3. **Test those failures on purpose:** Turn the dependency off in a test setting and watch what the product does. Fix the gaps you find, then test again.

### Map the Critical Paths

Start with the main jobs your users care about. For a shopping app, that might be browsing, adding to a cart, and paying. Write down each step, then mark every call that step makes to another part. A slow call in the middle of checkout can stop the whole checkout, so mark those calls first. Azure's failure mode analysis guidance assumes teams have already prioritized flows by how critical they are.

### Choose a Failure Response

Each dependency needs a plan for failure. Most plans fall into one of three kinds.

- **Fallback:** Use a backup answer when the normal call fails, such as a saved value or a simple default estimate.
- **Degrade:** Keep the page working, but hide or simplify the part that failed.
- **Fail fast:** Return a clear error right away, instead of waiting, when the work cannot be done safely.

### Test Failures on Purpose

A plan you have never run is only a guess. Shut down a dependency in a test environment, then check the results. Does the page still load? Does the user see a clear message? Do the other features still respond? When a test reveals a new problem, update your list of failure points.

> **Tip:** Start small. Test one dependency at a time before you try to break several at once. This habit is the core of chaos engineering, and it works best when the team agrees on what success looks like first.

**Quick check:** A grocery app's cart page shows a full error whenever its coupon service is switched off in a test. Coupons are only a small discount. The team wants the next step. What should they do?

A. Add more servers to the cart page, then run the same test again
B. Choose a degrade response for coupons, such as hiding them, then test again
C. Remove the coupon feature so the problem can never happen
D. Wait for a real outage so the team can see the real failure

**Answer:** B. The team skipped the step where they decide what happens when a dependency fails. Hiding coupons is a degrade response that keeps the cart working. Adding servers does not change how the cart handles a missing answer, and waiting for a real outage is a risky way to learn.

## Worked Example: A Home Screen With a Failed Recommendation Service

Streaming services are a useful case to study. That article does not explain how Netflix handles failures inside its system. So the steps below are an illustration of the checklist applied to a streaming home screen, not a report of how Netflix built its product.

1. **Name the critical path:** Opening the home screen and starting a title is the main job. Picking which rows to show is important, but it is not the main job.
2. **List the dependencies:** A personalized recommendations service supplies rows picked for this viewer. A simple list of popular titles is a separate, lighter source of rows.
3. **Choose the failure response:** If the recommendations service fails, use a fallback: show a generic popular-titles row in its place. The personal rows are simply left out, which is a degraded view.
4. **Add a timeout:** Give the recommendations call a short time limit. If the limit passes, the screen stops waiting and uses the fallback row.
5. **Test on purpose:** In a test setting, switch off the recommendations service. Check that the home screen still loads and that the user can still press play.

> **Tip:** The goal is a slightly less personal home screen, not an error page. A user who sees popular titles can keep watching. A user who sees an error page cannot.

## A Fallback in Code

This short Python sample shows the same idea in code. A tax service call fails with a timeout error, which is Python's name for a call that ran out of time. The function then returns a simple estimate, so the checkout can keep going. The sample uses only the standard library and prints one line.

```python
def tax_estimate(total):
    return round(total * 0.08, 2)

def get_tax(total, tax_service):
    try:
        return tax_service(total)
    except TimeoutError:
        return tax_estimate(total)

def broken_tax_service(total):
    raise TimeoutError('no answer in time')

print(get_tax(50.0, broken_tax_service))

```

Output:

```text
4.0

```

## Check Your Understanding

Read this final scenario and decide what you would do first. It uses every idea in the lesson.

**Quick check:** A shopping app's checkout calls a slow third-party tax service. When that service hangs, the whole app freezes for every user, including people who are only browsing. Which resiliency idea would you apply first?

A. Add more servers to the checkout page so it can wait longer
B. Set a timeout on the tax call and use a fallback tax estimate
C. Hide the checkout button until the tax service is fixed
D. Store every past tax result in the database forever

**Answer:** B. A timeout stops checkout from waiting forever, so the connections and threads it holds are freed for other users. The fallback estimate lets the order keep moving, and the browsing users are no longer stuck behind a frozen call. The rest of the app keeps working, and the team can later correct the estimate.

## Key Takeaways

- **Expect failure:** Plan for network drops, slow services, crashed copies, and traffic spikes, because you cannot prevent them all.
- **Protect the core job:** Use timeouts, fallbacks, and graceful degradation so the main job of the product survives a broken part.
- **Test on purpose:** Break dependencies in a safe setting, compare the results with what you expected, and fix what you find.

## Sources

- [Fault Tolerance and Resilience Patterns in Distributed Systems (Medium article)](https://medium.com/@yramcharanteja/fault-tolerance-and-resilience-patterns-in-distributed-systems-4bc3e4fb054a)
- [Graceful degradation in practice: how FeatureOps builds real resilience (Unleash blog)](https://www.getunleash.io/blog/graceful-degradation-featureops-resilience)
- [Distributed Computing From First Principles (arXiv paper, section 3.4)](https://arxiv.org/pdf/2506.12959)
- [Reliability & Fault Tolerance in System Design: Why Systems Must Expect Failure (Medium article)](https://medium.com/@iamakashrout/reliability-fault-tolerance-in-system-design-why-systems-must-expect-failure-e508090487af)
- [Resilience and Error Handling (DesignGurus system design guide)](https://www.designgurus.io/learn-system-design/resilience-and-error-handling)
- [Architecture strategies for performing failure mode analysis (Microsoft Azure Well-Architected Framework)](https://learn.microsoft.com/en-us/azure/well-architected/reliability/failure-mode-analysis)
- [Recommendations for performing failure mode analysis (Microsoft Azure Well-Architected Framework, design resiliency guide)](https://learn.microsoft.com/hr-hr/azure/well-architected/resiliency/design-resiliency)
- [Architecture strategies for designing a reliability testing strategy (Microsoft Azure Well-Architected Framework)](https://learn.microsoft.com/en-us/azure/well-architected/reliability/reliability-test)
- [Design review checklist for Reliability (Microsoft Azure Well-Architected Framework)](https://learn.microsoft.com/en-us/azure/well-architected/reliability/checklist)
- [Azure Well-Architected Framework: The 5 Pillars (ProsperOps blog)](https://www.prosperops.com/blog/azure-well-architected-framework/)
- [Netflix just killed a feature that seemed like a great idea (Tom's Guide article, describes the home screen rows)](https://tomsguide.com/news/netflix-just-killed-a-feature-that-seemed-like-a-great-idea)
