Every gateway node in our fleet needs the same piece of truth at boot: the shard-to-node mapping, the config revision, the list of revoked keys. Before Raft, that truth lived in a primary database with async replication, and “the truth” occasionally disagreed with itself. We needed a replicated, linearizable source of truth small enough to be memory-resident – and we needed the write path to survive node loss without an operator waking up at 3am.

Raft is the right tool: leader election, log replication, and a state machine, with the strongest consistency guarantee that is practical in a network that partitions. Its entire design goal was to be understandable enough to get right, where Paxos was provably correct but not buildable by normal humans.

The three invariants that make it safe

Everything in Raft reduces to three properties, and the bugs we have debugged in Raft implementations were always a violation of one of them:

1. Election safety. At most one leader per term. Randomized election timeouts – drawn from a range well above the heartbeat interval – prevent the split-vote flapping you get with fixed timeouts.
2. Log matching. A follower only appends a batch of entries if the entry before the batch exists with the same term. This single check is what lets a leader safely overwrite conflicting entries without ever rewriting a majority-committed one.
3. Commit by majority. An entry is committed only when it is replicated to a quorum (n/2 + 1). A leader never reports an entry committed until the quorum actually has it.

The AppendEntries handler

The heart of the implementation is the log-matching check. This is the one function whose correctness we sat on for a week:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
type Raft struct {
	mu          sync.Mutex
	term        uint64
	votedFor    uint64
	log         []Entry // index 0 is a dummy entry with term 0
	commitIndex uint64
	lastApplied uint64
	state       State
}

// AppendEntries replicates a batch of entries from a leader.
func (r *Raft) AppendEntries(args *AppendEntriesArgs, reply *AppendEntriesReply) error {
	r.mu.Lock()
	defer r.mu.Unlock()

	if args.Term < r.term {
		reply.Term, reply.Success = r.term, false
		return nil
	}
	if args.Term > r.term {
		r.term, r.votedFor, r.state = args.Term, 0, Follower
	}
	r.resetElectionTimer()

	// Log matching: the entry before the batch must exist with the same term.
	if prev, ok := r.entryAt(args.PrevLogIndex); !ok || prev.Term != args.PrevLogTerm {
		reply.Term, reply.Success = r.term, false
		return nil
	}
	r.truncateAndAppend(args.PrevLogIndex, args.Entries)

	if args.LeaderCommit > r.commitIndex {
		lastIndex := r.log[len(r.log)-1].Index
		if args.LeaderCommit < lastIndex {
			r.commitIndex = args.LeaderCommit
		} else {
			r.commitIndex = lastIndex
		}
	}
	reply.Term, reply.Success = r.term, true
	return nil
}

The pieces that look like ceremony are not. The stale-term reject at the top is what makes the leader step down when it is actually partitioned away. The resetElectionTimer on every valid AppendEntries is what keeps a healthy follower from ever starting an election. And the truncateAndAppend call is where the log-matching property converts into “conflicting suffix gets overwritten, committed prefix never does.”

What production taught us

fsync latency is the whole game. A leader cannot acknowledge a write until a quorum has durably fsynced it. On a 3-node cluster in one AZ with NVMe, a batched commit runs at ~40k writes/s with p50 ~0.8ms and p99 ~3ms. The moment a follower’s disk stalls for 50ms – a noisy neighbor, a snapshot writing at the wrong time – every commit waits on that disk and heartbeat deadlines blow past. Batch fsync of entries (etcd does this) keeps commit latency stable when disks are only mostly predictable.

Election timeouts are a latency-vs-churn tradeoff. With a 50ms heartbeat and a 100ms timeout, any GC pause over 50ms triggers an election storm on every replica. We run randomized timeouts in the 300-1500ms range; that makes a partition detectable in well under a second but tolerates the occasional 200ms scheduler hiccup without churning leadership.

Serving reads from the leader is not linearizable. A partitioned leader can keep answering reads while it can no longer commit. Two options: ReadIndex (ask a quorum for the current commit index before serving) or lease reads (trust a leader whose election timeout has not elapsed, given bounded clock skew). We shipped lease reads and then had a real incident when a buggy NTP daemon on one host violated the clock-skew assumption; if you cannot guarantee clock discipline, use ReadIndex.

Logs grow without bound. Without compaction, a 3-year-old Raft log eats the disk. Snapshot the state machine periodically and send InstallSnapshot to followers that fall behind the retained log prefix. Do this before the log becomes a disk emergency, because the snapshot itself is a burst of IO that can stall commits if it shares a disk with the log.

Build or buy

We shipped our v1 hand-rolled for the education and the control. Then the fsync edge cases, the snapshot storm, and the membership-change bugs (joint consensus is where the paper’s footnotes live) convinced us that consensus is a library problem, not an application problem. In Go, github.com/hashicorp/raft or etcd’s raft (go.etcd.io/etcd/v3/raft in the 3.4 era) are battle-tested; the value we added was in the surrounding machinery – the storage layer, snapshot policy, and operational alerts – not in the Raft core.

Production lessons

• Prototype the AppendEntries log-matching logic with a property test harness: random partition injection, random leader kills, then assert log matching and one-leader-per-term hold for hundreds of thousands of operations.
• Watch for election churn in metrics: a rising current_term with no membership change means your timeouts are too tight for your GC pause distribution.
• Never fsync and snapshot on the same disk. The snapshot storm is the second-most-common cause of commit latency spikes after follower disk stalls.