# High Availability: Keep Serving Users When Parts Fail

> Learn how uptime is measured in nines, how redundancy and failover work across zones, and how to remove single points of failure, with the trade-offs.

**In short:** High availability means keeping a system serving users when parts fail, using redundancy, load balancing, and failover across zones to remove single points of failure.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/high-availability

## What You Will Learn

High availability means a system keeps serving users even when some of its parts break. Instead of hoping that no server ever fails, the designer assumes parts will fail and plans for it. The goal is that users notice little or nothing when that happens.

In this lesson you will learn how availability is measured, why each extra level of uptime costs more, the main building blocks of a highly available design, and how to find and remove single points of failure. You will also see the trade-offs each fix brings. A worked example follows a streaming app's Play screen through a server failure and a zone failure.

- **High availability:** A design where a service keeps working for users even when single parts fail.
- **Uptime:** The share of time a service is up and working correctly.
- **Nines:** A short way to name an uptime percentage. 99.9% is called three nines.
- **Redundancy:** Keeping extra copies of a part so that one copy can keep the work going.
- **Load balancer:** A machine that spreads incoming requests across several servers.
- **Failover:** Moving work to a backup when the main part stops.
- **Single point of failure:** One part whose failure stops the whole service. Often shortened to SPOF.
- **Failure domain:** A group of parts that can fail together, such as one server, one zone, or one region.
- **Availability zone:** A separate group of data centers inside one cloud region, built so that a problem in one zone does not spread to another.
- **Replica:** A copy of data or of a running service kept on another machine.

## How Is Availability Measured?

Availability is usually written as a percentage of time that a service works in a year. A system that is up 99% of the time is down for the other 1%. Engineers often use the word nines for these levels. 99% is two nines, 99.9% is three nines, and 99.99% is four nines. Each extra nine makes the target much stricter.

The idea of uptime and availability is simple, but the numbers are easy to misread. Before you pick a target, you should know how many minutes or hours of downtime each target allows.

### The Downtime Each Nine Allows

A year has 365 days, which is 525,600 minutes. Downtime allowed is simply the share that is not uptime. Take 0.1% of that year for three nines and you get about 525.6 minutes, or 8.76 hours. The cards below show the common levels.

- **Two nines (99%):** About 3.65 days of downtime a year, or about 7.3 hours a month.
- **Three nines (99.9%):** About 8.76 hours a year, or about 43.8 minutes a month.
- **Four nines (99.99%):** About 52.6 minutes a year, or about 4.38 minutes a month.
- **Five nines (99.999%):** About 5.26 minutes a year, or about 26.3 seconds a month.

```python
minutes_per_year = 365 * 24 * 60
for nines in [2, 3, 4, 5]:
    allowed = minutes_per_year / 10 ** nines
    print(f"{nines} nines: {allowed:.2f} minutes a year")
```

Output:

```text
2 nines: 5256.00 minutes a year
3 nines: 525.60 minutes a year
4 nines: 52.56 minutes a year
5 nines: 5.26 minutes a year

```

> **Tip:** Convert a target into minutes before you talk about it. Saying 99.9% sounds close to 99.99%, but one allows about 8.76 hours of downtime a year and the other about 52.6 minutes.

### Why Does Each Extra Nine Cost More?

Each extra nine cuts the allowed downtime by about a factor of ten. The effort to reach that level grows much faster than the gain in uptime suggests. Another says that every added nine costs far more than the one before.

Downtime also has a price. Three nines allow about 175,000 dollars of exposure at the limit, while four nines reduce that to about 17,500 dollars. A team should compare the cost of each target with the cost of the outages it prevents.

Dependencies make the problem harder. If a service depends on several parts, its availability is lower than the availability of any one part. A service that depends on three parts that are each 99.9% available can reach only about 99.7% overall.

```python
parts = [0.999, 0.999, 0.999]
overall = 1.0
for p in parts:
    overall *= p
print(f"{overall:.1%}")
```

Output:

```text
99.7%

```

**Quick check:** A payments team moves its target from 99.9% to 99.99%. Roughly how much less downtime is allowed each year?

A. About ten times less
B. About two times less
C. About one hundred times less
D. Exactly the same amount

**Answer:** A. Each added nine cuts the allowed downtime by about ten. Going from 8.76 hours to about 52.6 minutes is close to a tenfold drop. The other options describe a change that is too small or too large.

## What Are the Core Building Blocks?

Most highly available designs combine a few ideas. They add redundancy so that a backup exists, they spread traffic so that no single copy is overloaded, and they switch work to a healthy copy when one fails. Each idea needs the others to work well.

### What Is Redundancy?

Redundancy means having several identical parts so that one failure does not stop the service. Multiple instances of a web server, duplicate databases, and spare network links are all examples. A design without redundancy has a single point of failure at the place where the missing copy would have been.

- **Redundant servers:** Several copies of the same application run at once, so one crash leaves the others serving users.
- **Redundant data:** A database keeps replicas on other machines, so the data survives when one machine dies.
- **Redundant paths:** Two network links or two power feeds mean that cutting one does not cut the service.

### What Is Load Balancing?

A load balancer sits in front of the copies and decides which one handles each request. It sends traffic only to copies that are working. Load balancing also keeps any one copy from being overloaded. Many systems also use the balancer to take a server out of service for an upgrade without users noticing.

> **Tip:** A load balancer is itself a part that can fail. Give it a standby too, or the balancer becomes the single point of failure.

### Failover: Active-Active or Active-Passive

Failover is the switch from a failed main part to a healthy backup. It should happen automatically, without a person having to act. Designers usually choose one of two patterns for how the backup behaves.

- **Active-active:** Every copy serves users at the same time. A load balancer spreads the work, and when one copy fails the rest carry more of the load. Capacity must be planned so the survivors can handle the extra work.
- **Active-passive:** A standby copy stays idle until the main copy fails. It must be ready to take over fully, and the switch takes some time. It uses less load-sharing logic, but the standby often costs money while doing no work.

Failover needs a way to notice a failure. A common method is a heartbeat, which is a regular signal that a healthy copy sends. When the signal stops for long enough, the backup takes over. The timing matters, because a slow detection means a longer outage.

### Failure Domains: Servers, Zones, and Regions

Copies only help if they do not fail together. A failure domain is the group of parts that a single problem can reach. Two servers in the same rack share a power feed, so they are in the same domain. Designers place copies in different domains on purpose.

- **Server:** The smallest domain. A crash, a bad disk, or a bug can take down one machine while others keep running.
- **Availability zone:** A group of separate data centers in one cloud region.
- **Region:** A larger area, with many zones, that is far from other regions. A region-wide problem is rare but can be large, so copies may also be spread across regions.

> **Tip:** Ask 'what is the biggest thing that could fail, and do my copies survive it?' Spread copies across the largest domain you can afford to protect against.

## Worked Example: A Streaming Play Screen

Imagine a streaming app, similar in style to a large video service. This is an illustrative design, not a description of any real company's system. When a user presses Play, a playback service checks the title and the user's account, then returns the stream details. The data lives in a database. The goal is that Play keeps working when one server or one whole zone fails.

1. **Run several copies of the playback service:** Instead of one server, run at least three copies. Place them in three separate availability zones, A, B, and C.
2. **Put a load balancer in front:** A load balancer receives every Play request and forwards it to a healthy copy. It checks each copy regularly and stops sending traffic to any copy that fails those checks.
3. **Replicate the database across zones:** Keep a primary database copy in zone A and replicas in zones B and C. A cloud database can keep replicas in several zones, and a write (a saved change) may wait until the copies in the other zones have it. That wait is the cost of protecting the data.
4. **Handle one server failure:** Suppose one playback copy crashes. The load balancer stops sending it requests. The other two copies take the extra work. Users pressing Play see no error, although the remaining copies now carry more load.
5. **Handle one whole zone failure:** Suppose zone A loses power. The load balancer keeps operating from the healthy zones and sends new requests to copies in B and C. The database fails over to a replica in zone B. Some requests in progress may fail and be retried, but new Play requests succeed.
6. **Restore the lost capacity:** Once the zone is back, or once new servers start elsewhere, the system returns to three healthy copies. The team then checks the logs to learn what the failure cost and what to improve.

> **Tip:** Test the plan before a real outage does. Turn off one copy in a test environment, then simulate a whole zone failure, and watch whether Play still works.

## Finding and Removing Single Points of Failure

A single point of failure is any part whose failure stops the service, because no backup exists for it. High availability is mostly the work of finding these parts and giving each one a backup. The search is systematic. You follow the path a request takes and ask what happens when each part stops.

1. **List every part on the request path:** Write down each step from the user's device to the stored data. Include the name lookup (the step that turns a web address into a server address), the load balancer, the application servers, any cache (a fast copy of data kept close to the app), the database, the network links, the power supply, and the data center itself.
2. **Ask what happens if each part stops:** For each item, decide whether the service keeps working for users. If one failure stops everything, that item is a single point of failure.
3. **Add a backup for that part:** Give the part a copy in another machine, rack, zone, or region, depending on the failure you want to survive.
4. **Make the switch automatic:** The backup should take over without waiting for a person. Automatic detection and failover keep the outage short.
5. **Check the spread:** Two copies in the same zone still share the zone's power and network. Confirm that the copies sit in different failure domains.
6. **Test the result:** Simulate the failure and measure how long recovery takes. A backup that has never been tested may not work when needed.

### Trade-offs of Removing a Single Point of Failure

Removing a single point of failure is not free. Each backup adds work, and the work comes with three costs that a designer must weigh.

- **Cost:** Every extra copy needs more servers, storage, and data transfer. More copies also mean more money spent while they sit idle, as with a standby.
- **Complexity:** More moving parts means more configuration, more monitoring, and more ways for the failover logic itself to go wrong. A simple design that is rarely tested can fail worse than expected.
- **Data consistency between replicas:** Replicas must agree on the data. If a write must reach every zone before it completes, each write is slower. If replicas are updated later, a backup may briefly hold older data after a failover.

Good designs accept these costs on purpose. They protect the data that matters most with strict copying, and they accept looser copying where a short delay is harmless.

**Quick check:** An online store runs one web server and one database in a single data center. Every time that data center has a power cut, the whole store goes down. Which single change would most improve availability, and what new trade-off would it bring?

A. Add a second web server in the same data center, which costs little but leaves both servers exposed to the same power cut
B. Replace the database with a faster disk in the same data center, which speeds up queries but does not help during a power cut
C. Run a standby web server and a replicated database in a second data center, which survives the power cut but adds cost and must keep data copies in step
D. Buy a larger server with two power supplies in the same rack, which helps against one failed supply but not a whole-site outage

**Answer:** C. The power cut hits the whole site, so the backup must sit in a different failure domain. A second data center or zone does that. The trade-off is more cost and the need to keep the replicated data consistent, as the lesson described.

## Key Takeaways

- **Measure the target:** Turn each uptime percentage into allowed minutes of downtime before you choose one. Each extra nine is a much stricter goal.
- **Build with copies and switches:** Use redundancy, a load balancer, and automatic failover. Pick active-active or active-passive based on cost and how fast the switch must be.
- **Spread copies across failure domains:** Copies in one server, rack, or zone can fail together. Place them far enough apart to survive the failures you expect.
- **Remove single points of failure with care:** Each backup adds cost and complexity, and replicas must stay consistent. Test the failover so the backup works when it is needed.

## Sources

- [Wikipedia: High availability](https://en.wikipedia.org/wiki/High_availability)
- [Wikipedia: Single point of failure](https://en.wikipedia.org/wiki/Single_point_of_failure)
- [AWS: What is High Availability?](https://aws.amazon.com/what-is/high-availability/)
- [Parallels: Load Balancing vs Failover: An Overview](https://www.parallels.com/blogs/ras/load-balancing-vs-failover/)
- [Nobl9: A Best Practices Guide to High Availability Design](https://www.nobl9.com/service-availability/high-availability-design)
- [Control Plane: What Is High Availability? How It Works and How Many Nines You Need](https://controlplane.com/blog/post/what-is-high-availability-how-it-works-and-how-many-nines-you-actually-need)
- [OpenObserve: How Many Nines Do You Actually Need?](https://openobserve.ai/blog/how-many-nines-do-you-actually-need/)
- [Project Helena: What is Uptime? SLAs, Nines and Downtime Tables](https://projecthelena.com/blog/what-is-uptime/)
- [ZoneITs: What the Uptime Nines Actually Cost You](https://zoneits.com/insights/uptime-nines-explained/)
- [Microsoft Azure: Azure availability zones](https://azure.microsoft.com/en-us/explore/global-infrastructure/availability-zones)
- [Microsoft Learn: Reliability in Azure Load Balancer](https://learn.microsoft.com/en-us/azure/reliability/reliability-load-balancer)
- [Microsoft Learn: Reliability in Azure SQL Database](https://learn.microsoft.com/en-us/azure/reliability/reliability-sql-database)
- [AWS Support: Fault tolerance checks](https://docs.aws.amazon.com/awssupport/latest/user/fault-tolerance-checks.html)
- [Gremlin: How to make your services zone redundant](https://www.gremlin.com/blog/how-to-run-a-zone-redundancy-test-using-gremlin)
- [TechChannel: What AWS Availability Zones Really Mean for Resilience](https://techchannel.com/aws/understanding-availability-zones/)
