# Availability Patterns: Fail-Over and Replication to Keep Systems Running

> Learn how systems stay reachable when one server fails, using fail-over and replication, and how to choose a pattern by the downtime a product can accept.

**In short:** Availability patterns, fail-over and replication, keep a system reachable when one server fails, chosen by how much downtime a product can accept.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/availability-patterns

## What you will learn

Some systems stop working when one machine breaks. Users see errors, and orders or messages can be lost. This lesson explains the standard ways to keep a system running and reachable when a part fails. In this course these ways are called availability patterns. There are two main families. Fail-over switches the work to a backup part. Replication keeps copies of data or services on more than one machine, so one failure does not take everything down.

By the end you should be able to explain active-passive and active-active fail-over, describe master-slave and master-master replication at a high level, and choose a pattern based on how much downtime a product can accept. Here are the key words used in this lesson.

- **Availability:** How much of the time a system is up and reachable when people need it.
- **Uptime:** The time a system is running and answering requests correctly.
- **Downtime:** The time a system is down or cannot be reached.
- **Single point of failure:** One part whose failure stops the whole system.
- **Redundancy:** Keeping an extra part ready, so the system survives when the main part fails.
- **Fail-over:** Moving the work from a broken part to a healthy backup part.
- **Heartbeat:** A short, regular 'I am alive' signal sent from one machine to its partner.
- **Replication:** Keeping copies of the same data or service on several machines.

## What does availability mean?

Availability measures how much of the time a system can be used. A system that answers requests is available. A system that is down, or that users cannot reach, is not. Teams talk about uptime, which is the running time, and downtime, which is the time the system is not usable. A simple way to see it: a fast website that is offline is not useful to anyone.

The biggest risk for availability is a single point of failure. Imagine a food ordering site where the whole order system runs on one server. If that server loses power, nobody can order, however well the code is written. Availability patterns exist to remove these single points, so that one broken part does not stop the service.

Redundancy is the main tool for this. It means keeping an extra part that can do the job when the main part fails. The extra part might be a standby server, a second copy of a database, or a second machine in another building. The idea is simple: if one part dies, another keeps going. The price is that you now own, power, and maintain more than one part.

## What is fail-over?

Fail-over is the act of moving work from a failed component to a healthy one. Before it can switch, the system has to notice that something is wrong. A common method is the heartbeat. The active machine sends a short signal at a regular interval, and its partner watches for these signals. When they stop for long enough, the partner decides the active machine has failed and takes over.

Fail-over comes in two main forms. The difference is whether the backup sits idle or works alongside the main part.

### What is active-passive fail-over?

In an active-passive setup, one server is active and handles all the traffic. A second server, called the passive or standby server, stays ready but does no regular work. When heartbeats from the active server stop, the standby takes over. In one common approach, the standby takes over the active server's network address, so users keep reaching the same address without changing anything.

The switch is not instant. The system must first detect the failure, then promote the standby and send traffic to it. Users may see a short pause or a few failed requests during this switch-over. Also, if the standby's copy of the data is behind the active server's copy, recent changes may be lost when it takes over.

### What is active-active fail-over?

In an active-active setup, both servers handle requests at the same time. A traffic splitter, often called a load balancer, shares the requests between them. If one server fails, the other keeps serving and takes over the failed server's work. Because neither machine sits idle, the system can often handle more load. The cost is a more complex design, especially when both servers can change the same data.

### What are the trade-offs of fail-over?

- **Idle standby cost:** An active-passive standby is often a full machine that does little until a failure. You pay for it all the same.
- **Possible downtime:** The switch-over takes time, so users may feel a short outage while the system finds the failure and moves the work.
- **Possible data loss:** If the backup is behind the main part, changes it never received can disappear when it takes over.
- **More parts, more complexity:** Every extra machine needs setup, monitoring, and testing, and the switch itself must be tested too.

## Replication: keeping copies

Replication keeps copies of data, or copies of a service, on several machines. If one copy is lost, another still has the information. Fail-over handles the switching side of availability, while replication handles the data side. The two often work together: copies are kept in several places, and when the main copy fails, one of the others takes its place. Two common forms are master-slave and master-master.

### Master-slave replication

In master-slave replication, one machine is the master. It accepts all writes, which are the changes to the data. Other machines, called slaves or replicas, copy the master's changes. They usually serve reads, which are requests that only look at the data. If the master fails, one replica can be promoted to become the new master. Some teams use the words primary and replica for the same roles.

### Master-master replication

In master-master replication, which is also called multi-master, more than one machine accepts writes. Each one passes its changes to the others. If one master fails, the others keep taking writes, so the system stays up without a promotion step. The cost is conflict. Two masters may change the same record at nearly the same time, so the system needs rules that decide which change wins.

### How replication connects to consistency

Consistency means that readers see the same, correct data. Replication makes this harder, because copies can briefly disagree. A master can send its changes to replicas after it confirms a write, which is fast but can lose the newest changes if the master crashes. It can also wait for replicas to confirm first, which protects the data but makes each write slower. This is the link between availability and consistency: the faster and more available a design is, the more it may accept short disagreement between copies.

A single master makes it easier to keep one agreed order of changes, but it is less flexible. Multi-master systems are usually loosely consistent, which means copies may differ for a while. You do not need to solve this yet. Later lessons on consistency go deeper.

## Redundancy costs time and money

Every availability pattern adds parts, and every part adds cost, setup work, and new ways to fail. A standby that sits idle still needs power and care. A switch-over is one more thing to test. So the real question is not which pattern is best. The real question is how much downtime this product can tolerate, and what it costs to get closer to zero.

### Choosing a pattern by downtime tolerance

- **Tolerates short outages:** An internal tool used during office hours may accept a few minutes offline. A simple active-passive setup is often enough.
- **Cannot pause for long:** A checkout that earns money every minute may need active-active parts, so that one failure does not stop all traffic.
- **Cannot lose recent data:** Synchronous replication waits for copies before confirming a write. This prevents loss, but every write becomes slower.

> **Tip:** Before you pick a pattern, ask two questions. How long can users wait when a part fails? How much recent data can the business afford to lose? The answers point you to the simplest pattern that meets the need.

## Worked example: a food delivery order database

This example is an illustration of the pattern, not a report on any one company's real setup. A food delivery app stores its orders in a database. The main database is the primary, and a standby replica keeps a copy of it. Here is what happens when the primary crashes in the middle of the dinner rush.

1. **Normal work:** The primary accepts every new order and every status change. The replica receives each change and keeps its copy up to date.
2. **Failure:** The primary crashes. The app can no longer write new orders to it.
3. **Detection:** The heartbeat between the two machines stops. After missing signals for a set time, the system decides the primary is down.
4. **Promotion:** The standby replica is promoted to primary, so its copy becomes the one the app writes to.
5. **Redirect:** The app connects to the new primary. Customers may see a brief pause or a retry message.
6. **Recovery:** Customers place orders again. If the replica had not yet received the last few changes, the team checks those orders for gaps.

### Watching a heartbeat in code

The short program below imitates the detection step. Each value in the list is one tick of time, where True means the active server answered and False means it did not. After three misses in a row, the program swaps the roles. Real systems use more careful checks, but the core idea is the same.

```python
missed_limit = 3
heartbeats = [True, True, False, False, False]
active = 'server-A'
standby = 'server-B'
missed = 0
for tick, alive in enumerate(heartbeats, start=1):
    if alive:
        missed = 0
    else:
        missed += 1
    if missed == missed_limit:
        active, standby = standby, active
        print(f'tick {tick}: failover, {active} is now active')
print('final active:', active)

```

Output:

```text
tick 5: failover, server-B is now active
final active: server-B

```

## Check yourself

**Quick check:** A payment service runs on one server. A second server keeps a copy of its settings and does no work until the first server stops answering heartbeats. Which pattern is this, and what is one downside if the second server's data is slightly behind when it takes over?

A. Active-active fail-over. The downside is that both servers share the load and must stay in sync.
B. Active-passive fail-over. The downside is that recent changes not yet copied may be missing after the takeover.
C. Master-master replication. The downside is that reads become slower than writes.
D. Load balancing alone. The downside is that the idle server adds no cost.

**Answer:** B. The second server waits idle until a failure, which is the active-passive form. If it has not received the newest changes, those changes can be lost when it takes over.

**Quick check:** A news site runs three identical web servers that all answer readers at the same time. A traffic splitter sends each request to a server that is still up. One server fails. Which pattern is in use, and what is its main cost?

A. Active-passive fail-over. The main cost is that two idle servers draw power with no work.
B. Active-active fail-over. The main cost is extra design complexity and careful handling of shared state.
C. Master-slave replication. The main cost is that readers see only one copy of the news.
D. A single point of failure. The main cost is that the splitter sends traffic to one server.

**Answer:** B. All three servers are live, so the surviving servers absorb the work. Its cost is complexity rather than idle machines.

## Key takeaways

- **Remove single points:** Availability patterns add redundancy so that one failed part does not stop the whole service.
- **Pick by downtime:** Fail-over and replication each have costs. Choose the simplest pattern that meets the downtime and data-loss limits of the product.
- **Expect trade-offs:** Faster recovery and less data loss usually mean more complexity, more cost, or slower writes.

## Sources

- [Aerospike: Active-Active vs. Active-Passive High-Availability Guide](https://aerospike.com/blog/active-active-vs-active-passive/)
- [Aerospike: Understanding Failover Mechanisms for High Availability and Reliability](https://aerospike.com/blog/understanding-failover-mechanisms/)
- [Medium (Cyberdrudge): System Design: Consistency, Availability and CAP Theorem](https://medium.com/@cyberdrudge/system-design-consistency-availability-and-cap-theorem-effb9326f8b5)
- [Staying Online: A Guide to Availability Patterns (sibilsarjamsoren.in)](https://sibilsarjamsoren.in/blog/availability-patterns)
- [Wikipedia: Multi-master replication](https://en.wikipedia.org/wiki/Multi-master_replication)
- [DigitalOcean: How To Choose a Redundancy Plan To Ensure High Availability](https://www.digitalocean.com/community/tutorials/how-to-choose-a-redundancy-plan-to-ensure-high-availability)
- [DZone: Pros and Cons of MySQL Replication Types](https://dzone.com/articles/pros-and-cons-of-mysql-replication-types)
- [Medium (Navid Barsalari): Master-Slave Database Architecture: A Complete, High-Level, Technical Guide](https://medium.com/@navidbarsalari/master-slave-database-architecture-a-complete-high-level-technical-guide-722caf6ff32e)
- [Medium (Tech Suneel): Database Replication: Master-Slave Architecture](https://medium.com/@techsuneel99/database-replication-master-slave-architecture-1a1246397a22)
- [System Design Classroom: High Availability Isn't About Luck](https://newsletter.systemdesignclassroom.com/p/high-availability-isnt-about-luck)
- [Nobl9: A Best Practices Guide to High Availability Design](https://www.nobl9.com/service-availability/high-availability-design)
- [GitDoc: System Design Primer: From Principles to Production](https://gitdoc.ai/resources/system-design-primer)
- [Grokking the System Design Primer: Step-by-Step Guide](https://grokkingthesystemdesign.com/guides/system-design-primer/)
- [Low Level Design Mastery: Availability Patterns](https://www.lowleveldesignmastery.com/hld-concepts/reliability/01-availability-patterns/)
- [Realty API: Design Resilient High Availability Architecture](https://www.realtyapi.io/blog/high-availability-architecture)
- [DEV Community: System Design: Availability Patterns](https://dev.to/decoders_lord/system-design-availability-patterns-104i)
