I wrote Raft from scratch and spent a month losing elections

Two weeks into implementing Raft for flywheel, my three-node test cluster had a leader election rate that could only be described as vibes. Every few minutes the cluster would dissolve into a civil war of candidates, resolve itself, and do it again.

The bug was not in the election logic. The bug was in the clock.

The setup

Each node drives its election timer from a monotonic clock. When a follower hasn't heard from a leader for election_timeout milliseconds, it becomes a candidate:

// simplified from flywheel's election.rs
let elapsed = now_monotonic() - self.last_heartbeat_at;
if elapsed >= self.election_timeout {
    self.start_election();
}

Timeouts were randomized between 150 and 300 ms; heartbeats fire every 40 ms. On paper that is a comfortable margin.

What was actually happening

perf and a few strategically placed eprintln!s told the real story:

1. Release builds start three nodes within ~2 ms of each other.
2. I sampled the random election timeout once, at boot.
3. A supervisor restart loop meant node B always rejoined with the fastest possible timeout value available at that moment.

Every campaign for leadership started at the same offset. Split votes, forever, in a loop.

Raft does not require synchronized clocks — but it does require uncorrelated randomization. The paper is explicit about this, and I had read it the way people read error messages: skimming for the part I already agreed with.

...election timeouts are chosen randomly from a fixed interval... each server randomizes its election timeout every time it receives an RPC.

"If votes are split, each candidate randomizes its timeout and tries again." The important half of that sentence is each time.

The fix

Re-sample the timeout on every reset, not at boot:

fn reset_election_timer(&mut self) {
    let spread = self.max_timeout_ms - self.min_timeout_ms;
    self.election_timeout = self.min_timeout_ms + rng().gen_range(0..spread);
}

The election storm dropped from ~40 per hour to zero over the following week of chaos testing, including the kill -9 loops that originally exposed it.

The lesson is not "randomize harder". It is that when a distributed algorithm tells you to randomize, it means every time the relevant event happens — not once, during construction, when it felt like enough.