Raft Consensus Over Membership List Versus Gossip / SWIM-Based Membership
TradeoffDistributed SystemsSeniorStaffUpdated
Tracking which of ~1500 worker nodes are alive in a batch-processing cluster.
Short answer: Gossip / SWIM-based membership. Raft membership churn at 1500 nodes saturates the leader; you do not need linearizable liveness, just convergence within seconds.
The Two Options
A
Raft consensus over membership list
B
Gossip / SWIM-based membership
Drill This: Your Pick First
The Better Pick
Option B
Gossip / SWIM-based membership
Raft membership churn at 1500 nodes saturates the leader; you do not need linearizable liveness, just convergence within seconds. SWIM gives sub-second failure detection at constant per-node cost.
When the Other One Wins
Raft consensus over membership list
For a small, security-critical control plane (10s of nodes) where 'who is in the cluster' must be strongly consistent and auditable, Raft membership wins.