# Leader Election: Choosing One Instance to Be in Charge

> When identical copies of a service must not all act at once, they elect one leader, and a new one takes over if it fails.

**In short:** Leader election is how identical copies of a service pick one instance to be in charge, with a new leader taking over if the current one fails.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/leader-election

## What You Will Learn

Some jobs must run in exactly one place at a time, even when a service runs as five or fifty identical copies. Leader election is the pattern that picks the one copy allowed to do that job. The chosen copy is the leader. The others wait as followers or standbys. If the leader stops working, the group chooses a new leader without a person stepping in. This lesson explains why the pattern is needed, how it works, what can go wrong, and how Kubernetes uses it.

- **Instance:** One running copy of a program, such as one container or one server process.
- **Leader election:** A method that lets a group of instances agree on which single instance holds a special role.
- **Leader:** The one instance that holds the role and does the shared job.
- **Lease:** A time-limited lock. Whoever holds it may act until the time runs out, unless it is renewed.
- **Heartbeat:** A small signal that shows a process is still alive. Renewing a lease often works as a heartbeat.
- **TTL (time to live):** How long a lease lasts before it expires, usually a few seconds or tens of seconds.
- **Split brain:** A failure where two instances both believe they are the leader at the same time.

## Why Is One Leader Needed?

Imagine a service that runs as three identical copies behind a load balancer. A load balancer is a front door that spreads incoming requests across the copies. Each copy can answer web requests, and that part is easy to share. Some jobs are different. A job that cleans old files, sends a bill, or updates a shared counter should run once per cycle. If all three copies start that job at the same moment, the work is repeated or the data is damaged. Some tasks, such as coordinating the other instances, should be done by only one worker at a time.

- **Conflicting changes:** Two copies may write different values to the same record. The final data then depends on timing, which makes bugs hard to find.
- **Duplicate work:** The same job runs several times. Customers may get repeated emails or be charged twice.
- **Damaged shared data:** When two copies change one resource together, the result can be half-updated and wrong.

Distributed systems, which are groups of machines that work together, often need one node (one machine or process in the group) to coordinate tasks. That keeps data consistent and avoids conflicts. Leader election gives the group that single coordinator, while keeping the other copies ready to step in.

## How Do Instances Elect a Leader?

The group needs a way to pick one winner and to notice when the winner disappears. Most designs use a shared place outside the instances, where the current leader is recorded. Every instance tries to write its own name there. Only one attempt can succeed, because the shared place accepts one claim at a time. The winner must keep proving it is alive. If that proof stops, the others can take over the role.

1. **Start the group:** Run several identical instances. Each one knows where the shared record lives.
2. **Compete for the lease:** Each instance tries to take the lease. The first to hold it becomes the leader, and the rest become followers.
3. **Renew the lease:** The leader renews the lease on a fixed schedule. Every renewal tells the group that the leader is still working.
4. **Notice the silence:** If the leader crashes or loses its connection, it cannot renew. The lease then runs out on its time limit.
5. **Elect a replacement:** A follower takes the expired lease and becomes the new leader. The others keep waiting.

There are two main families of methods. The first uses leases and heartbeats with a shared store. The second uses votes among the servers themselves. Both aim for the same result: at most one leader at a time.

### What Are Leases and Heartbeats?

A lease is a time-bounded lock. The leader must renew it before it expires. If the leader cannot renew, other nodes can take the lease and become leader. The lease is like a timed permission slip. The leader may act only while the slip is valid. If the leader fails to renew the lease within the TTL, the lease expires automatically.

The renewal is also a heartbeat. It is a small message that says the leader is still alive. A leader that is slow, stuck, or cut off from the shared store cannot renew, so its authority ends on its own. That is the main benefit: no one has to decide that the leader is dead. The clock does it.

> **Tip:** Choose the TTL with care. A very short lease causes frequent handovers when the network is briefly slow. A very long lease means the group waits a long time after a real crash.

### What Are Consensus Votes?

Consensus means that a group of servers agrees on one answer by voting. Raft is a well-known set of voting rules for this. Raft works in rounds called terms. Each term has at most one leader. A follower that stops hearing from the leader starts an election. If a follower gets no heartbeat within its election timeout, it becomes a candidate. A candidate wins only when a majority of the group supports it.

That majority rule is the key safety idea. A leader must keep acknowledgment from a majority of nodes. A small group cut off from the rest cannot crown its own leader, because it does not have a majority. Raft, Paxos (another voting method), and ZooKeeper (a coordination service) all rely on this kind of majority approach.

## Worked Example: Kubernetes Controller Manager

Kubernetes runs its control plane, the part that watches the cluster and fixes differences, on several machines for safety. One of its parts, kube-controller-manager, runs as several replicas, which are identical copies. Here is how the replicas share the job.

1. **Replicas race for the lock:** Each replica tries to get hold of a Lease object. A namespace is a named group of objects inside the cluster. All instances of the Controller Manager compete to acquire or update that specific Lease object.
2. **The winner is recorded:** The replica that succeeds writes its identity into the lease. That replica becomes the active leader.
3. **The leader keeps renewing:** The active leader renews the lease again and again. In plain words, the lease lasts 15 seconds, the leader must renew within a 10 second window, and the waiting replicas check again every 2 seconds.
4. **The others stand by:** The other replicas read the lease, see who holds it, and make no changes. All running instances watch or periodically read the Lease object to find out which instance is acting as the leader.
5. **A takeover after a crash:** If the leader dies, its renewals stop. A standby acquires the lease after it expires. The old lease is not released when the process exits, so the takeover waits for the time limit.

The lease itself is stored by the Kubernetes API server, which keeps its data in etcd (a small, reliable shared database for cluster data). A Lease acts as a lightweight distributed lock, stored by the Kubernetes API server. Because the store accepts only one successful update at a time, two replicas cannot both win the same lease.

## Trade-offs and Pitfalls

Leader election solves one problem and creates others. Before you use it, understand the costs. The three cards below cover the most common ones.

- **Single point of failure and bottleneck:** The elected leader may become a performance constraint. Also, if the election fails, the whole job stops until a leader appears.
- **Split brain:** Split brain occurs when a network partition, a break that splits the group into parts, causes two subgroups to each elect their own leader. Both leaders then change shared data, which can corrupt it.
- **Clocks and pauses:** Clock skew, where machines disagree about the time, can cause two nodes to hold the lease at the same moment. A leader that pauses, for example during a long garbage collection cycle (a memory cleanup that freezes a program), may keep acting after its lease has ended.

Designers often add a fencing token to handle the last problem. A fencing token is a number that grows each time a new leader is chosen. The storage checks the number and rejects writes from an old leader. Lease expiration and fencing tokens are design options that reduce the risk of split-brain, but they cannot guarantee exactly one leader.

### Why Keep the Leader's Role Small?

The more work only the leader can do, the more damage a bad leader can cause. Keep the leader focused on deciding and handing out work. Let the heavy lifting run on the other instances, or in a queue. Also make the leader's actions safe to repeat. A repeated action is safe when running it twice gives the same result as running it once. Then a new leader can redo half-finished work without causing harm.

> **Tip:** Give each task a unique ID and record which tasks are done. When a new leader starts, it checks the record first, so customers do not get a second email.

### Prefer Existing Tools

Writing your own election code is risky. Lease timing, clock drift, and pauses are easy to get subtly wrong. For most cloud-native scenarios, you do not need to build Raft or Paxos yourself. Use a tested coordination service such as etcd, ZooKeeper, or the lease feature built into Kubernetes. ZooKeeper is a classic choice.

> **Tip:** Test the failure case on purpose. Stop the leader, pause it, or cut its network, and check that exactly one new leader appears and that the old one stops acting.

## See the Lease Logic in Code

The program below models a lease with a fake clock, so the output is the same on every run. A real system stores the lease in a shared service. The rule is the same: one holder at a time, and the holder must renew before the time runs out. Each call passes the current time, so the example never reads a real clock.

```python
class Lease:
    def __init__(self, ttl):
        self.ttl = ttl
        self.holder = None
        self.expires_at = 0

    def try_acquire(self, name, now):
        if self.holder is None or now >= self.expires_at:
            self.holder = name
            self.expires_at = now + self.ttl
            return True
        return self.holder == name

    def renew(self, name, now):
        if self.holder == name and now < self.expires_at:
            self.expires_at = now + self.ttl
            return True
        return False

lease = Lease(ttl=3)
print(lease.try_acquire('pod-a', 0))
print(lease.try_acquire('pod-b', 1))
print(lease.renew('pod-a', 2))
print(lease.try_acquire('pod-b', 4))
print(lease.try_acquire('pod-b', 5))

```

Output:

```text
True
False
True
False
True

```

Read the output line by line. At time 0, pod-a takes the lease. At time 1, pod-b tries and fails. At time 2, pod-a renews, which pushes the expiry to time 5. At time 4, pod-b still fails because the lease is valid. At time 5, the lease has run out, so pod-b takes it.

## Check Yourself

Try these scenarios. Each one needs the idea from this lesson to answer it.

**Quick check:** Your app runs five identical instances. Each one sends the nightly summary email, so customers receive five copies. Which change fixes this best?

A. Turn off the email job by hand on four instances each night
B. Use leader election so that one instance holds a lease and only the leader sends the email
C. Add a sixth instance so the emails are shared across more copies
D. Give each instance more memory so the emails finish sooner

**Answer:** B. Leader election makes one instance responsible for the shared job. The others wait, so the emails go out once. A manual switch breaks as soon as an instance is replaced, and more copies make the duplicate problem worse.

Now take the first scenario one step further. The leader starts sending the emails, and then it crashes after forty customers have their emails.

**Quick check:** The leader crashes halfway through the nightly email job. Which design is safest for the new leader?

A. Send the whole email to every customer again, because the old leader's work is lost
B. Do nothing, because the lease expired and the job is finished
C. Resume the job, and record each customer who already got the email so none is sent twice
D. Ask each customer by hand whether they received the email

**Answer:** C. The new leader needs to know what is already done. Recording progress lets it continue safely, and the job can be repeated without sending duplicates. Resending everything causes the duplicate problem again.

**Quick check:** An old leader loses its network link but keeps running. After its lease expires, a new leader starts. What should stop the old leader from still acting?

A. Nothing, so both leaders keep working until someone restarts a machine
B. The old leader should stop acting once it can no longer renew its lease, but a paused leader may still act, so extra protection is needed.
C. The new leader deletes the old leader's process from the cluster
D. Followers send a message to the old leader's address to vote it out

**Answer:** B. A lease gives the leader permission only for a limited time. Without renewal the leader should stop, but a leader that wrongly believes it still holds the lock can keep acting. Clock problems and long pauses still need extra protection such as fencing tokens.

## Key Takeaways

Leader election lets identical instances agree on one coordinator. Use it when a shared job must run in one place at a time. The chosen leader holds a lease or a consensus role, renews it on a schedule, and gives way when it fails. Keep the leader's job small, plan for split brain, and use tools that have been tested by others. In the next steps of your study, compare this pattern with approaches where many instances share work instead of waiting for one leader.

## Sources

- [AWS Builder Center: Leader Election in Distributed Systems](https://aws.amazon.com/builders-library/leader-election-in-distributed-systems/)
- [Kubernetes documentation: Coordinated Leader Election](https://kubernetes.io/docs/concepts/cluster-administration/coordinated-leader-election)
- [Kubernetes website source: coordinated-leader-election.md](https://github.com/kubernetes/website/blob/main/content/en/docs/concepts/cluster-administration/coordinated-leader-election.md)
- [Matthew Swong: Leader Election in Distributed Systems: Leases and Fencing](https://www.matthewswong.com/en/blog/leader-election-distributed-systems/)
- [Netdata guide: Kubernetes Controller-Manager Leader Election Failures](https://www.netdata.cloud/guides/kubernetes/kubernetes-controller-manager-leader-election/)
- [Dev.to: Kubernetes Controller Manager Explained](https://dev.to/aws-builders/kubernetes-controller-manager-explained-k8s-with-pravesh-episode-6-h70)
- [Codelit: Leader Election: Algorithms, Protocols & Split-Brain Prevention](https://codelit.io/blog/leader-election-algorithms)
- [Layrs: Leader Election (Resiliency), System Design Interview Guide](https://layrs.me/course/hld/12-reliability-patterns/leader-election-resiliency)
- [Mortal Apps: Leader Election](https://mortalapps.com/system-design/distributed-coordination/leader-election/)
- [Neel Mishra: Leader Election](https://neelmishra.github.io/blog/hld/distributed-systems/leader-election.html)
- [Vetora Labs: Leader Election, Consensus and Coordination](https://www.vetoralabs.com/system-design/concepts/consensus/leader-election)
- [GitHub mspnp cloud-design-patterns: leader-election example](https://github.com/mspnp/cloud-design-patterns/tree/main/leader-election)
- [DevGenius: Leader Election, Cloud Design Pattern](https://blog.devgenius.io/leader-election-cloud-design-pattern-515192a3e3f5?gi=27272d0fd1ba)
