Leader election: preventing split-brain in distributed clusters

In distributed architectures like Kubernetes (etcd), Apache Kafka (KRaft), Consul, and CockroachDB, clusters rely on a single Leader node to coordinate writes, maintain state consistency, and sequence replication logs.

What happens when that leader node crashes, freezes during a garbage collection pause, or gets disconnected by a severed network switch?

Systems use Leader Election algorithms (such as Raft or Paxos) to automatically elect a new leader without human intervention.

                  ┌───────────────────────────────┐
                  │       LEADER (Node 1)         │
                  └───────────────┬───────────────┘
                                  │ Heartbeats (every 50ms)
                    ┌─────────────┴─────────────┐
                    ▼                           ▼
            ┌───────────────┐           ┌───────────────┐
            │ Follower (N2) │           │ Follower (N3) │
            └───────────────┘           └───────────────┘

---

1. Heartbeats & Failure Detection

Under normal operation:

1. The Leader continuously broadcasts periodic heartbeat pings (every 50–100ms) to all Follower nodes.
2. As long as Followers receive heartbeats, they reset their internal Election Timers and remain passive.
3. If the Leader crashes or becomes partitioned, Followers stop receiving heartbeats.
4. When a Follower’s election timer expires (typically randomized between 150ms and 300ms to prevent vote splitting), it promotes itself to a Candidate, increments the cluster term counter, and requests votes from peers.

---

2. The Split-Brain Catastrophe

The most dangerous failure mode in distributed consensus is Split-Brain.

Imagine a 5-node cluster spanning two data centers that suffers a network partition:

    [Partition A: 3 Nodes]       X-- SPLIT --X       [Partition B: 2 Nodes]
   ┌─────────┐   ┌─────────┐                        ┌─────────┐   ┌─────────┐
   │ Node 1  │   │ Node 2  │                        │ Node 4  │   │ Node 5  │
   │(Leader) │   │(Follower│                        │(Leader?)│   │(Follower│
   └─────────┘   └─────────┘                        └─────────┘   └─────────┘
         │             │                                  │             │
         └──────┬──────┘                                  └──────┬──────┘
            ┌───┴───┐                                        ┌───┴───┐
            │Node 3 │                                        │ (No   │
            └───────┘                                        │quorum)│
                                                             └───────┘

If both Partition A and Partition B elect their own separate Leaders, both partitions accept writes independently.

The database states diverge, creating contradictory records (e.g. money spent twice) that are mathematically impossible to reconcile automatically.

---

3. How Majority Quorum Prevents Split-Brain

Consensus protocols prevent split-brain by enforcing a strict Quorum Rule:

Quorum = floor(N / 2) + 1

A candidate can only become Leader, and a Leader can only commit writes, if it secures votes from a strict majority (>50%) of the entire cluster configuration:

• In a 5-node cluster, quorum requires at least 3 votes.
• Partition A (3 nodes): Has 3 out of 5 nodes (>= 3). It achieves quorum, elects a leader, and processes client traffic normally.
• Partition B (2 nodes): Has only 2 out of 5 nodes (< 3). It cannot reach quorum, refuses to elect a leader, and rejects all incoming writes.

When the network heals, Partition B synchronizes its state log from Partition A’s authoritative leader.

---

Why Distributed Clusters Always Use Odd Node Counts (3, 5, 7)

Engineers often wonder why production clusters mandate odd numbers of nodes:

Total Nodes (N)	Quorum Needed (floor(N/2) + 1)	Max Failed Nodes Tolerated
3	2	1 node
4	3	1 node
5	3	2 nodes
6	4	2 nodes
7	4	3 nodes

Adding a 4th node to a 3-node cluster does not increase fault tolerance—both can only survive 1 failure, but a 4-node cluster adds extra network chatter and cost.

Hence, production consensus systems always deploy 3, 5, or 7 nodes.