HFT-52 · Week 20 · Failover and hot standby

Leases, fencing and never two leaders

A primary leads only while it holds a lease from the arbiter; a hot standby takes over only after that lease has expired. Crash, partition or pause the primary, switch off one safety mechanism at a time, and see which failure each one prevents.

Primary A, standby B, one arbiter · orders arrive at 200 per ms
Who leads, in real time (arbiter clock) Primary AStandby Btwo leadersfault
Step through the failover
    The journal around the fault (µs)
    A sequencesB has itclient sees the acklost

    The safety rule: a node acts as leader only inside its lease, and the arbiter grants a new lease only after the old one has expired, so leaderships never overlap. Each mechanism closes one gap. Lease check: A runs may_lead, timing its lease from when it sent the renewal and stopping at sent_at + D − margin. Drift margin: A's clock may run slow; the margin (drift × D) keeps A's deadline before the arbiter's expiry. Fencing: B takes epoch 2 on promotion and refuses frames from epoch 1, which stops a paused or partitioned A that is wrong about the time. Output gating: A releases acks and market data for a command only after B acknowledges it, so nothing a client saw can be lost and a deposed A can never show anything. Model: real time on the arbiter's clock; A's clock runs slow by the drift setting; one-way network latency 20 µs; B's acquire waits at the arbiter and is granted the instant A's lease expires; frames leave A 8 µs after sequencing (kernel send path), reach B 20 µs later, and B's ack returns after 22 µs more; a crash loses frames not yet on the wire, a partition also loses frames in flight; during a pause the kernel still sends what was queued. Orders keep arriving at 200 per ms, so lost-order counts scale with time.