esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Leader Election: Choosing One Instance to Be in Charge

When identical copies of a service must not all act at once, they elect one leader, and a new one takes over if it fails.

In short

Leader election is how identical copies of a service pick one instance to be in charge, with a new leader taking over if the current one fails.

12 min readFree

What You Will Learn

Some jobs must run in exactly one place at a time, even when a service runs as five or fifty identical copies. Leader election is the pattern that picks the one copy allowed to do that job. The chosen copy is the leader. The others wait as followers or standbys. If the leader stops working, the group chooses a new leader without a person stepping in. This lesson explains why the pattern is needed, how it works, what can go wrong, and how Kubernetes uses it.

Key words

Instance
One running copy of a program, such as one container or one server process.
Leader election
A method that lets a group of instances agree on which single instance holds a special role.
Leader
The one instance that holds the role and does the shared job.
Lease
A time-limited lock. Whoever holds it may act until the time runs out, unless it is renewed.
Heartbeat
A small signal that shows a process is still alive. Renewing a lease often works as a heartbeat.
TTL (time to live)
How long a lease lasts before it expires, usually a few seconds or tens of seconds.
Split brain
A failure where two instances both believe they are the leader at the same time.

Why Is One Leader Needed?

Imagine a service that runs as three identical copies behind a load balancer. A load balancer is a front door that spreads incoming requests across the copies. Each copy can answer web requests, and that part is easy to share. Some jobs are different. A job that cleans old files, sends a bill, or updates a shared counter should run once per cycle. If all three copies start that job at the same moment, the work is repeated or the data is damaged. Some tasks, such as coordinating the other instances, should be done by only one worker at a time.

  • Conflicting changes

    Two copies may write different values to the same record. The final data then depends on timing, which makes bugs hard to find.

  • Duplicate work

    The same job runs several times. Customers may get repeated emails or be charged twice.

  • Damaged shared data

    When two copies change one resource together, the result can be half-updated and wrong.

Distributed systems, which are groups of machines that work together, often need one node (one machine or process in the group) to coordinate tasks. That keeps data consistent and avoids conflicts. Leader election gives the group that single coordinator, while keeping the other copies ready to step in.

How Do Instances Elect a Leader?

The group needs a way to pick one winner and to notice when the winner disappears. Most designs use a shared place outside the instances, where the current leader is recorded. Every instance tries to write its own name there. Only one attempt can succeed, because the shared place accepts one claim at a time. The winner must keep proving it is alive. If that proof stops, the others can take over the role.

  1. 1

    Start the group

    Run several identical instances. Each one knows where the shared record lives.

  2. 2

    Compete for the lease

    Each instance tries to take the lease. The first to hold it becomes the leader, and the rest become followers.

  3. 3

    Renew the lease

    The leader renews the lease on a fixed schedule. Every renewal tells the group that the leader is still working.

  4. 4

    Notice the silence

    If the leader crashes or loses its connection, it cannot renew. The lease then runs out on its time limit.

  5. 5

    Elect a replacement

    A follower takes the expired lease and becomes the new leader. The others keep waiting.

There are two main families of methods. The first uses leases and heartbeats with a shared store. The second uses votes among the servers themselves. Both aim for the same result: at most one leader at a time.

What Are Leases and Heartbeats?

A lease is a time-bounded lock. The leader must renew it before it expires. If the leader cannot renew, other nodes can take the lease and become leader. The lease is like a timed permission slip. The leader may act only while the slip is valid. If the leader fails to renew the lease within the TTL, the lease expires automatically.

The renewal is also a heartbeat. It is a small message that says the leader is still alive. A leader that is slow, stuck, or cut off from the shared store cannot renew, so its authority ends on its own. That is the main benefit: no one has to decide that the leader is dead. The clock does it.

Tip

Choose the TTL with care. A very short lease causes frequent handovers when the network is briefly slow. A very long lease means the group waits a long time after a real crash.

What Are Consensus Votes?

Consensus means that a group of servers agrees on one answer by voting. Raft is a well-known set of voting rules for this. Raft works in rounds called terms. Each term has at most one leader. A follower that stops hearing from the leader starts an election. If a follower gets no heartbeat within its election timeout, it becomes a candidate. A candidate wins only when a majority of the group supports it.

That majority rule is the key safety idea. A leader must keep acknowledgment from a majority of nodes. A small group cut off from the rest cannot crown its own leader, because it does not have a majority. Raft, Paxos (another voting method), and ZooKeeper (a coordination service) all rely on this kind of majority approach.

Worked Example: Kubernetes Controller Manager

Kubernetes runs its control plane, the part that watches the cluster and fixes differences, on several machines for safety. One of its parts, kube-controller-manager, runs as several replicas, which are identical copies. Here is how the replicas share the job.

  1. 1

    Replicas race for the lock

    Each replica tries to get hold of a Lease object. A namespace is a named group of objects inside the cluster. All instances of the Controller Manager compete to acquire or update that specific Lease object.

  2. 2

    The winner is recorded

    The replica that succeeds writes its identity into the lease. That replica becomes the active leader.

  3. 3

    The leader keeps renewing

    The active leader renews the lease again and again. In plain words, the lease lasts 15 seconds, the leader must renew within a 10 second window, and the waiting replicas check again every 2 seconds.

  4. 4

    The others stand by

    The other replicas read the lease, see who holds it, and make no changes. All running instances watch or periodically read the Lease object to find out which instance is acting as the leader.

  5. 5

    A takeover after a crash

    If the leader dies, its renewals stop. A standby acquires the lease after it expires. The old lease is not released when the process exits, so the takeover waits for the time limit.

The lease itself is stored by the Kubernetes API server, which keeps its data in etcd (a small, reliable shared database for cluster data). A Lease acts as a lightweight distributed lock, stored by the Kubernetes API server. Because the store accepts only one successful update at a time, two replicas cannot both win the same lease.

Trade-offs and Pitfalls

Leader election solves one problem and creates others. Before you use it, understand the costs. The three cards below cover the most common ones.

  • Single point of failure and bottleneck

    The elected leader may become a performance constraint. Also, if the election fails, the whole job stops until a leader appears.

  • Split brain

    Split brain occurs when a network partition, a break that splits the group into parts, causes two subgroups to each elect their own leader. Both leaders then change shared data, which can corrupt it.

  • Clocks and pauses

    Clock skew, where machines disagree about the time, can cause two nodes to hold the lease at the same moment. A leader that pauses, for example during a long garbage collection cycle (a memory cleanup that freezes a program), may keep acting after its lease has ended.

Designers often add a fencing token to handle the last problem. A fencing token is a number that grows each time a new leader is chosen. The storage checks the number and rejects writes from an old leader. Lease expiration and fencing tokens are design options that reduce the risk of split-brain, but they cannot guarantee exactly one leader.

Why Keep the Leader's Role Small?

The more work only the leader can do, the more damage a bad leader can cause. Keep the leader focused on deciding and handing out work. Let the heavy lifting run on the other instances, or in a queue. Also make the leader's actions safe to repeat. A repeated action is safe when running it twice gives the same result as running it once. Then a new leader can redo half-finished work without causing harm.

Tip

Give each task a unique ID and record which tasks are done. When a new leader starts, it checks the record first, so customers do not get a second email.

Prefer Existing Tools

Writing your own election code is risky. Lease timing, clock drift, and pauses are easy to get subtly wrong. For most cloud-native scenarios, you do not need to build Raft or Paxos yourself. Use a tested coordination service such as etcd, ZooKeeper, or the lease feature built into Kubernetes. ZooKeeper is a classic choice.

Tip

Test the failure case on purpose. Stop the leader, pause it, or cut its network, and check that exactly one new leader appears and that the old one stops acting.

See the Lease Logic in Code

The program below models a lease with a fake clock, so the output is the same on every run. A real system stores the lease in a shared service. The rule is the same: one holder at a time, and the holder must renew before the time runs out. Each call passes the current time, so the example never reads a real clock.

Python
class Lease:
    def __init__(self, ttl):
        self.ttl = ttl
        self.holder = None
        self.expires_at = 0

    def try_acquire(self, name, now):
        if self.holder is None or now >= self.expires_at:
            self.holder = name
            self.expires_at = now + self.ttl
            return True
        return self.holder == name

    def renew(self, name, now):
        if self.holder == name and now < self.expires_at:
            self.expires_at = now + self.ttl
            return True
        return False

lease = Lease(ttl=3)
print(lease.try_acquire('pod-a', 0))
print(lease.try_acquire('pod-b', 1))
print(lease.renew('pod-a', 2))
print(lease.try_acquire('pod-b', 4))
print(lease.try_acquire('pod-b', 5))

Output

True
False
True
False
True

Read the output line by line. At time 0, pod-a takes the lease. At time 1, pod-b tries and fails. At time 2, pod-a renews, which pushes the expiry to time 5. At time 4, pod-b still fails because the lease is valid. At time 5, the lease has run out, so pod-b takes it.

Check Yourself

Try these scenarios. Each one needs the idea from this lesson to answer it.

Quick check

Your app runs five identical instances. Each one sends the nightly summary email, so customers receive five copies. Which change fixes this best?

Now take the first scenario one step further. The leader starts sending the emails, and then it crashes after forty customers have their emails.

Quick check

The leader crashes halfway through the nightly email job. Which design is safest for the new leader?

Quick check

An old leader loses its network link but keeps running. After its lease expires, a new leader starts. What should stop the old leader from still acting?

Key Takeaways

Leader election lets identical instances agree on one coordinator. Use it when a shared job must run in one place at a time. The chosen leader holds a lease or a consensus role, renews it on a schedule, and gives way when it fails. Keep the leader's job small, plan for split brain, and use tools that have been tested by others. In the next steps of your study, compare this pattern with approaches where many instances share work instead of waiting for one leader.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.