Surviving the Split-Brain (Distributed Consensus)

1. Problem Statement & Scope

A student confidently explains that they have built a Highly Available (HA) database by deploying two master nodes. If Master A goes down, a load balancer automatically routes all traffic to Master B. They tell you they have achieved 100% uptime.

You pull the literal plug on the network switch connecting Data Center A to Data Center B. Both masters are still powered on, but they cannot talk to each other. Because they cannot see each other, both assume the other is dead. Both promote themselves to the primary writer. For three hours, Master A accepts writes from European clients, while Master B accepts writes from Asian clients.

When you plug the switch back in, the datasets are hopelessly divergent. The student has just discovered the Split-Brain Problem. To fix it, we must abandon simple active-passive failovers and introduce a rigorous Distributed Consensus algorithm like Raft.

Functional Requirements

• Leader Election: The cluster must autonomously elect a single leader to serialize all write operations.
• Log Replication: The leader must safely replicate state changes to follower nodes in the exact same order.

Non-Functional Requirements

• Partition Tolerance: The system must continue to operate correctly as long as a strict majority (quorum) of nodes remain connected.
• Strict Consistency: A read immediately following a successful write must return the updated value, regardless of which node serves the read.

2. High-Level Design (HLD)

To survive a network partition, a consensus cluster must always have an odd number of nodes (typically 3, 5, or 7). This guarantees that in the event of a network split, one side will always hold a mathematical majority (the quorum), while the other side is mathematically frozen.

3. The Viva: Pushing the Boundaries

4. Low-Level Design (LLD): The Raft State Machine

At its core, a consensus node is a finite state machine. Below is a concise Python conceptualization of the transition logic that governs how a Follower becomes a Leader.

import threading
import time
from enum import Enum
from typing import List, Dict

class NodeState(Enum):
    FOLLOWER = 1
    CANDIDATE = 2
    LEADER = 3

class RaftNode:
    def __init__(self, node_id: int, peers: List[int]):
        self.node_id = node_id
        self.peers = peers
        
        # Persistent state on all servers
        self.current_term = 0
        self.voted_for = None
        self.log = []
        
        # Volatile state
        self.state = NodeState.FOLLOWER
        self.commit_index = 0
        self.last_heartbeat = time.time()
        
    def on_election_timeout(self):
        """Triggered if no heartbeat is received within the randomized timeout window."""
        if self.state == NodeState.LEADER:
            return
            
        print(f"Node {self.node_id}: Election timeout. Transitioning to Candidate.")
        self.state = NodeState.CANDIDATE
        self.current_term += 1
        self.voted_for = self.node_id
        
        # Start the election process
        self.request_votes()

    def request_votes(self):
        votes_received = 1  # Votes for itself
        quorum = (len(self.peers) + 1) // 2 + 1
        
        for peer in self.peers:
            # In LLD, this is an asynchronous RPC call
            vote_granted = self.send_request_vote_rpc(peer, self.current_term)
            if vote_granted:
                votes_received += 1
                
        if votes_received >= quorum and self.state == NodeState.CANDIDATE:
            print(f"Node {self.node_id}: Quorum achieved. I am the Leader for Term {self.current_term}.")
            self.state = NodeState.LEADER
            self.start_sending_heartbeats()

    def receive_append_entries_rpc(self, leader_term: int, leader_id: int, entries: List):
        """Handles incoming heartbeats and log replications from a Leader."""
        if leader_term >= self.current_term:
            # A valid leader exists. Step down if we were a candidate or a stale leader.
            if self.state != NodeState.FOLLOWER:
                print(f"Node {self.node_id}: Found higher term {leader_term}. Stepping down.")
            
            self.state = NodeState.FOLLOWER
            self.current_term = leader_term
            self.last_heartbeat = time.time()
            
            # (Log replication logic goes here)
            return True
        return False
        
    def send_request_vote_rpc(self, peer_id: int, term: int) -> bool:
        # Mock network call to a peer
        pass
        
    def start_sending_heartbeats(self):
        # Spawns a background thread to continuously assert leadership
        pass

5. Fault Tolerance & Edge Cases

• Log Inconsistency: When a leader crashes, it might leave followers with partial, uncommitted logs. When the new leader takes over, it does not attempt to merge these logs. Raft enforces the rule that the Leader’s log is absolute truth. The new leader will aggressively overwrite conflicting entries in the followers’ logs to force them into exact alignment.
• Byzantine Failures: What if Node 2 is hacked and intentionally alters its log data before responding to the leader? Raft assumes a Crash-Fault-Tolerant (CFT) model, meaning nodes might die or delay, but they do not lie. To protect against malicious actors or corrupted memory returning forged data, the architecture must abandon Raft and implement much slower Practical Byzantine Fault Tolerance (PBFT), which requires rigorous cryptographic signing of every single state transition.