esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

High Availability: Keep Serving Users When Parts Fail

Learn how uptime is measured in nines, how redundancy and failover work across zones, and how to remove single points of failure, with the trade-offs.

In short

High availability means keeping a system serving users when parts fail, using redundancy, load balancing, and failover across zones to remove single points of failure.

12 min readFree

What You Will Learn

High availability means a system keeps serving users even when some of its parts break. Instead of hoping that no server ever fails, the designer assumes parts will fail and plans for it. The goal is that users notice little or nothing when that happens.

In this lesson you will learn how availability is measured, why each extra level of uptime costs more, the main building blocks of a highly available design, and how to find and remove single points of failure. You will also see the trade-offs each fix brings. A worked example follows a streaming app's Play screen through a server failure and a zone failure.

Key words

High availability
A design where a service keeps working for users even when single parts fail.
Uptime
The share of time a service is up and working correctly.
Nines
A short way to name an uptime percentage. 99.9% is called three nines.
Redundancy
Keeping extra copies of a part so that one copy can keep the work going.
Load balancer
A machine that spreads incoming requests across several servers.
Failover
Moving work to a backup when the main part stops.
Single point of failure
One part whose failure stops the whole service. Often shortened to SPOF.
Failure domain
A group of parts that can fail together, such as one server, one zone, or one region.
Availability zone
A separate group of data centers inside one cloud region, built so that a problem in one zone does not spread to another.
Replica
A copy of data or of a running service kept on another machine.

How Is Availability Measured?

Availability is usually written as a percentage of time that a service works in a year. A system that is up 99% of the time is down for the other 1%. Engineers often use the word nines for these levels. 99% is two nines, 99.9% is three nines, and 99.99% is four nines. Each extra nine makes the target much stricter.

The idea of uptime and availability is simple, but the numbers are easy to misread. Before you pick a target, you should know how many minutes or hours of downtime each target allows.

The Downtime Each Nine Allows

A year has 365 days, which is 525,600 minutes. Downtime allowed is simply the share that is not uptime. Take 0.1% of that year for three nines and you get about 525.6 minutes, or 8.76 hours. The cards below show the common levels.

  • Two nines (99%)

    About 3.65 days of downtime a year, or about 7.3 hours a month.

  • Three nines (99.9%)

    About 8.76 hours a year, or about 43.8 minutes a month.

  • Four nines (99.99%)

    About 52.6 minutes a year, or about 4.38 minutes a month.

  • Five nines (99.999%)

    About 5.26 minutes a year, or about 26.3 seconds a month.

Python
minutes_per_year = 365 * 24 * 60
for nines in [2, 3, 4, 5]:
    allowed = minutes_per_year / 10 ** nines
    print(f"{nines} nines: {allowed:.2f} minutes a year")

Output

2 nines: 5256.00 minutes a year
3 nines: 525.60 minutes a year
4 nines: 52.56 minutes a year
5 nines: 5.26 minutes a year

Tip

Convert a target into minutes before you talk about it. Saying 99.9% sounds close to 99.99%, but one allows about 8.76 hours of downtime a year and the other about 52.6 minutes.

Why Does Each Extra Nine Cost More?

Each extra nine cuts the allowed downtime by about a factor of ten. The effort to reach that level grows much faster than the gain in uptime suggests. Another says that every added nine costs far more than the one before.

Downtime also has a price. Three nines allow about 175,000 dollars of exposure at the limit, while four nines reduce that to about 17,500 dollars. A team should compare the cost of each target with the cost of the outages it prevents.

Dependencies make the problem harder. If a service depends on several parts, its availability is lower than the availability of any one part. A service that depends on three parts that are each 99.9% available can reach only about 99.7% overall.

Python
parts = [0.999, 0.999, 0.999]
overall = 1.0
for p in parts:
    overall *= p
print(f"{overall:.1%}")

Output

99.7%

Quick check

A payments team moves its target from 99.9% to 99.99%. Roughly how much less downtime is allowed each year?

What Are the Core Building Blocks?

Most highly available designs combine a few ideas. They add redundancy so that a backup exists, they spread traffic so that no single copy is overloaded, and they switch work to a healthy copy when one fails. Each idea needs the others to work well.

What Is Redundancy?

Redundancy means having several identical parts so that one failure does not stop the service. Multiple instances of a web server, duplicate databases, and spare network links are all examples. A design without redundancy has a single point of failure at the place where the missing copy would have been.

  • Redundant servers

    Several copies of the same application run at once, so one crash leaves the others serving users.

  • Redundant data

    A database keeps replicas on other machines, so the data survives when one machine dies.

  • Redundant paths

    Two network links or two power feeds mean that cutting one does not cut the service.

What Is Load Balancing?

A load balancer sits in front of the copies and decides which one handles each request. It sends traffic only to copies that are working. Load balancing also keeps any one copy from being overloaded. Many systems also use the balancer to take a server out of service for an upgrade without users noticing.

Tip

A load balancer is itself a part that can fail. Give it a standby too, or the balancer becomes the single point of failure.

Failover: Active-Active or Active-Passive

Failover is the switch from a failed main part to a healthy backup. It should happen automatically, without a person having to act. Designers usually choose one of two patterns for how the backup behaves.

  • Active-active

    Every copy serves users at the same time. A load balancer spreads the work, and when one copy fails the rest carry more of the load. Capacity must be planned so the survivors can handle the extra work.

  • Active-passive

    A standby copy stays idle until the main copy fails. It must be ready to take over fully, and the switch takes some time. It uses less load-sharing logic, but the standby often costs money while doing no work.

Failover needs a way to notice a failure. A common method is a heartbeat, which is a regular signal that a healthy copy sends. When the signal stops for long enough, the backup takes over. The timing matters, because a slow detection means a longer outage.

Failure Domains: Servers, Zones, and Regions

Copies only help if they do not fail together. A failure domain is the group of parts that a single problem can reach. Two servers in the same rack share a power feed, so they are in the same domain. Designers place copies in different domains on purpose.

  • Server

    The smallest domain. A crash, a bad disk, or a bug can take down one machine while others keep running.

  • Availability zone

    A group of separate data centers in one cloud region.

  • Region

    A larger area, with many zones, that is far from other regions. A region-wide problem is rare but can be large, so copies may also be spread across regions.

Tip

Ask 'what is the biggest thing that could fail, and do my copies survive it?' Spread copies across the largest domain you can afford to protect against.

Worked Example: A Streaming Play Screen

Imagine a streaming app, similar in style to a large video service. This is an illustrative design, not a description of any real company's system. When a user presses Play, a playback service checks the title and the user's account, then returns the stream details. The data lives in a database. The goal is that Play keeps working when one server or one whole zone fails.

  1. 1

    Run several copies of the playback service

    Instead of one server, run at least three copies. Place them in three separate availability zones, A, B, and C.

  2. 2

    Put a load balancer in front

    A load balancer receives every Play request and forwards it to a healthy copy. It checks each copy regularly and stops sending traffic to any copy that fails those checks.

  3. 3

    Replicate the database across zones

    Keep a primary database copy in zone A and replicas in zones B and C. A cloud database can keep replicas in several zones, and a write (a saved change) may wait until the copies in the other zones have it. That wait is the cost of protecting the data.

  4. 4

    Handle one server failure

    Suppose one playback copy crashes. The load balancer stops sending it requests. The other two copies take the extra work. Users pressing Play see no error, although the remaining copies now carry more load.

  5. 5

    Handle one whole zone failure

    Suppose zone A loses power. The load balancer keeps operating from the healthy zones and sends new requests to copies in B and C. The database fails over to a replica in zone B. Some requests in progress may fail and be retried, but new Play requests succeed.

  6. 6

    Restore the lost capacity

    Once the zone is back, or once new servers start elsewhere, the system returns to three healthy copies. The team then checks the logs to learn what the failure cost and what to improve.

Tip

Test the plan before a real outage does. Turn off one copy in a test environment, then simulate a whole zone failure, and watch whether Play still works.

Finding and Removing Single Points of Failure

A single point of failure is any part whose failure stops the service, because no backup exists for it. High availability is mostly the work of finding these parts and giving each one a backup. The search is systematic. You follow the path a request takes and ask what happens when each part stops.

  1. 1

    List every part on the request path

    Write down each step from the user's device to the stored data. Include the name lookup (the step that turns a web address into a server address), the load balancer, the application servers, any cache (a fast copy of data kept close to the app), the database, the network links, the power supply, and the data center itself.

  2. 2

    Ask what happens if each part stops

    For each item, decide whether the service keeps working for users. If one failure stops everything, that item is a single point of failure.

  3. 3

    Add a backup for that part

    Give the part a copy in another machine, rack, zone, or region, depending on the failure you want to survive.

  4. 4

    Make the switch automatic

    The backup should take over without waiting for a person. Automatic detection and failover keep the outage short.

  5. 5

    Check the spread

    Two copies in the same zone still share the zone's power and network. Confirm that the copies sit in different failure domains.

  6. 6

    Test the result

    Simulate the failure and measure how long recovery takes. A backup that has never been tested may not work when needed.

Trade-offs of Removing a Single Point of Failure

Removing a single point of failure is not free. Each backup adds work, and the work comes with three costs that a designer must weigh.

  • Cost

    Every extra copy needs more servers, storage, and data transfer. More copies also mean more money spent while they sit idle, as with a standby.

  • Complexity

    More moving parts means more configuration, more monitoring, and more ways for the failover logic itself to go wrong. A simple design that is rarely tested can fail worse than expected.

  • Data consistency between replicas

    Replicas must agree on the data. If a write must reach every zone before it completes, each write is slower. If replicas are updated later, a backup may briefly hold older data after a failover.

Good designs accept these costs on purpose. They protect the data that matters most with strict copying, and they accept looser copying where a short delay is harmless.

Quick check

An online store runs one web server and one database in a single data center. Every time that data center has a power cut, the whole store goes down. Which single change would most improve availability, and what new trade-off would it bring?

Key Takeaways

  • Measure the target

    Turn each uptime percentage into allowed minutes of downtime before you choose one. Each extra nine is a much stricter goal.

  • Build with copies and switches

    Use redundancy, a load balancer, and automatic failover. Pick active-active or active-passive based on cost and how fast the switch must be.

  • Spread copies across failure domains

    Copies in one server, rack, or zone can fail together. Place them far enough apart to survive the failures you expect.

  • Remove single points of failure with care

    Each backup adds cost and complexity, and replicas must stay consistent. Test the failover so the backup works when it is needed.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.