# Fail-Over: How a Backup Takes Over When the Main Server Stops

> Learn how a standby machine takes over work when the main one fails, the two main setups, how failures are found, and the costs and risks.

**In short:** Fail-over is when a standby machine takes over work after the main one fails, using one of two main setups and with costs and risks to weigh.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/fail-over

## What you will learn

Fail-over is a plan for the moment a key part of a system stops working. A backup part, called a standby, takes over the work so users can keep going. Designers use fail-over to raise availability, which means how often a service is up and usable when people try to reach it. In this lesson you will learn the two main setups, how a break is found, the difference between hot and cold standbys, and the risks that come with the idea. A worked example follows a store's checkout database through a crash.

- **Fail-over:** Moving work from a main component to a backup component after the main one stops.
- **Primary:** The component that handles live work during normal running. Some write it as the active one.
- **Standby:** A backup component that is ready to take over the work of the primary.
- **Heartbeat:** A short message sent on a regular timer to show that a machine is still alive.
- **Health check:** A test that asks a service whether it is working correctly, often by sending it a small request.
- **Hot standby:** A backup that stays in step with the primary and can take over at once.
- **Cold standby:** A backup that is off or idle and must be started before it can do any work.
- **Promote:** To change a standby so that it becomes the new primary.
- **Replication lag:** The delay between a change on the primary and the moment its copy arrives on the standby.

## What are the two main setups?

Fail-over comes in two main shapes. The difference is whether the backup machine does any work while the main one is healthy. Read the cards below first, then the sections that explain each one.

- **Active-passive:** One machine does the live work and a second machine waits. The waiting machine takes over only when the first one fails.
- **Active-active:** Two or more machines all handle live requests at the same time. When one fails, the others carry its share of the load.
- **Which one to pick:** Simple designs usually start with active-passive. Active-active suits teams that can handle the extra design work it brings.

## What is active-passive fail-over?

In the active-passive setup, the standby watches the primary. The two machines are linked by heartbeat messages. As long as the heartbeats keep arriving, the standby stays quiet and does nothing to users. Some failover setups use a primary and a standby linked by heartbeat messages.

The standby does more than wait for a signal. It also needs a way to receive the traffic. Many setups move a shared network address to the standby, so clients keep using the same address.

Plan for the spare machine to be idle most of the time. The active node must carry the full workload, while the backup resources sit unused during normal operation.

Between the two, the spare machine is the price of the design. You pay for it every day, but you use it only on the bad day.

## What is active-active fail-over?

In the active-active setup, every machine serves users at the same time. Several nodes run together and split the incoming work between them. A load balancer, which is a device that spreads requests across machines, often does the splitting. That topic has its own lessons, so here it is enough to know that the split exists.

The main benefit is what happens during a failure. If one machine stops, the others keep running and take over the tasks that machine was doing. Users may notice a little extra load, but there is no separate switch-over step to wait for.

Active-active also has a cost in design work. Every machine must agree on the same data, so the team has to solve problems about shared state. Here is how one comparison table describes it: Active-active adds complexity because of synchronization and conflict resolution, while active-passive is simpler because only one node is active.

> **Tip:** A simple rule for beginners: if you can accept a short pause during a failure, active-passive is easier to build and run. If the pause is not acceptable, you may need active-active and the extra work it brings.

## How is a failure found and handled?

A fail-over system cannot switch on its own until it knows something is wrong. The usual signals are heartbeats and health checks. Regular heartbeat messages travel between the machines, and they reveal when a server goes offline. Health checks are often more detailed than a heartbeat, because they test whether the service can actually answer a request.

Once heartbeats stop, the system needs a rule for when to act. <cite index="11-3">If a heartbeat is missed for a set period, the system assumes the node has failed and starts the fail-over, sending work to a standby.</cite>

1. **Send signals on a timer:** The primary sends a heartbeat at a fixed rate. The standby or a monitor listens for it.
2. **Count the misses:** Each expected heartbeat that does not arrive adds one to a miss count. A good arrival resets the count to zero.
3. **Decide to fail over:** When the count reaches the limit, the monitor runs a health check before it acts, so a single slow message does not trigger a switch.
4. **Promote the standby:** The standby becomes the new primary.
5. **Move the traffic:** The address or route that users rely on now points to the new primary, and requests resume.
6. **Rebuild the backup:** After a fail-over, only one machine is running. The team must set up a new standby before the next failure.

```python
# Heartbeats arrive once per second from the primary.
# The monitor fails over after 3 missed heartbeats in a row.
LIMIT = 3
arrivals = {1: True, 2: True, 3: True, 4: False, 5: False, 6: False}
missed = 0
for second in range(1, 7):
    if arrivals[second]:
        missed = 0
    else:
        missed += 1
    if missed >= LIMIT:
        print(f"second {second}: failover starts")
        break
    print(f"second {second}: missed={missed}")
```

Output:

```text
second 1: missed=0
second 2: missed=0
second 3: missed=0
second 4: missed=1
second 5: missed=2
second 6: failover starts

```

> **Tip:** Notice that a fail-over tool must do the detecting.

### What are hot, warm, and cold standbys?

Standbys are often grouped by how ready they are. The group a team picks decides how long users wait during a switch and how much the backup costs to keep.

- **Hot standby:** The backup stays running and in step with the primary. If the primary fails, the hot standby takes over at once.
- **Warm standby:** The backup is running but does not take the work yet. <cite index="11-7">It carries some ongoing cost to stay ready, yet the disruption during a switch is much smaller than with a cold standby.</cite>
- **Cold standby:** The backup is off or idle. After a failure, someone must start it by hand. <cite index="11-6">This setup suits non-critical systems where a long recovery is acceptable.</cite>

## Worked example: a store's checkout database

Imagine an online store. Its checkout page writes each order to a PostgreSQL database. The primary database sends its changes to a standby copy. Here is how a crash is handled, step by step.

1. **Normal day:** Checkout writes orders to the primary. The standby receives the changes, but a copy of the last few writes may still be on the way.
2. **The primary crashes:** The heartbeat messages stop. The fail-over software counts the missed heartbeats and sees that the limit has been reached.
3. **A health check confirms the failure:** A second test asks the database to answer a simple request. It does not answer, so the team's tool decides to fail over.
4. **The standby is promoted:** The tool runs the PostgreSQL promote command on the standby. The standby leaves its waiting mode and starts accepting writes as the new primary.
5. **Checkout resumes:** The store's address now points to the new primary. Checkout pauses for a short time and then works again.
6. **Some orders may be missing:** If the copy was lagging, orders written in the last second before the crash may never have reached the standby. The store must check those orders and rebuild a standby.

> **Tip:** Test the switch on purpose before you need it. Testing a fail-over before it is needed shows whether it will really work.

## Trade-offs and risks

Fail-over improves availability, but it is not free. Each choice below has a cost that a team must weigh.

- **Extra hardware and cost:** A hot standby is a full second system that runs all the time. Teams often pay for capacity they use only on the rare day of a failure.
- **Downtime during the switch:** Even a well-tuned setup usually needs a moment to detect the failure and move the work. Some requests are likely to be dropped during the transition.
- **Possible data loss:** If the primary fails before its newest writes reach the standby, those writes can disappear.

There is also a complexity cost. A team that skips this work may find the switch fails at the worst moment.

## Check your understanding

**Quick check:** A shopping app has one active server and one idle standby. The active server crashes at 2 a.m., and the standby takes over after 30 seconds. Which fail-over setup is this?

A. Active-active, because both servers exist and share the load before the crash
B. Active-passive, because one server works while the other waits to take over
C. Cold standby with no heartbeats, because the spare must be built from scratch
D. Load balancing, because requests are spread across two servers

**Answer:** B. Only one server does the work at a time, and the other waits. That is active-passive. The 30 seconds of silence and the chance of lost recent data are risks of that design, not signs of a different setup.

**Quick check:** A store's standby copies its data every few minutes. An order was saved on the primary at 1:59:50 a.m., and the crash happened right after. The standby took over at 2:00. What is the most likely result?

A. The order is safe on both machines, so nothing is lost
B. The order may be missing on the new primary because its copy had not arrived yet
C. The standby refuses all orders until the primary is repaired
D. The heartbeats saved the order before the switch

**Answer:** B. A copy that runs every few minutes can lag behind the primary. Writes made just before a crash may not have reached the standby, so they can be lost during the switch.

## Key takeaways

- **The idea:** Fail-over moves work from a failed primary to a standby, so users keep getting service with little or no interruption.
- **The setups:** Active-passive keeps a waiting standby. Active-active runs all machines at once and lets the rest absorb a failed machine's work.
- **The risks:** Watch the extra cost, the short switch-over pause, and the chance that recent data was not yet copied to the standby.

## Sources

- [Collabnix: Active-Active vs Active-Passive: Multi-Region Failover Design](https://collabnix.com/active-active-vs-active-passive-multi-region-failover-design-for-teams-without-a-dedicated-sre/)
- [Aerospike: Understanding Failover Mechanisms for High Availability and Reliability](https://aerospike.com/blog/understanding-failover-mechanisms/)
- [Aerospike: Active-Active vs. Active-Passive: High-Availability Guide](https://aerospike.com/blog/active-active-vs-active-passive/)
- [StarWind: Active-Active vs Active-Passive: Key Differences, Benefits & Use Cases](https://www.starwindsoftware.com/blog/active-active-vs-active-passive-differences/)
- [JSCAPE: Active-Active Vs. Active-Passive High-Availability Clustering](https://www.jscape.com/blog/active-active-vs-active-passive-high-availability-cluster)
- [GeeksforGeeks: Active Passive and Active Active Architecture for High Availability](https://www.geeksforgeeks.org/system-design/active-passive-active-active-architecture-for-high-availability-system/)
- [PostgreSQL documentation (current): 26.3 Failover](https://www.postgresql.org/docs/current/warm-standby-failover.html)
- [PostgreSQL documentation (current): Chapter 26 High Availability, Load Balancing, and Replication](https://www.postgresql.org/docs/current/high-availability.html)
- [PostgreSQL documentation (version 13): Failover](https://www.postgresql.org/docs/13/warm-standby-failover.html)
