HFT-52 · Weeks 10 and 46 · Market data gap recovery

Gaps, snapshots and the reorder window

A client builds its book from sequenced incrementals over UDP. When a sequence number goes missing it stops trusting the book, buffers, waits for a snapshot, drops what the snapshot already covers and replays the rest. Break the network and watch it stay correct.

Publisher → network → reorder window → book builder
Packets by time: sent above, received below (delayed packets drop a row)
Client state machine
TransitionWhenFired
Reorder window
Recovery buffer (kept while Recovering)
Last snapshot resync
Client book vs publisher book

The model: the publisher emits one absolute L2 update per sequence number (set level to qty, 0 deletes it) and a snapshot of the whole book after every 12th, labelled with the last sequence number it includes. Snapshots travel on a separate recovery channel, arrive about 4 sequence numbers behind the live stream and are never lost. Each incremental is lost on a line with the probability you set; with duplicates on, every packet also goes out on a B line with independent loss (A/B arbitration). With reorder on, 12% of packets are delayed by 1–3 packets. The book builder follows the week 10 drill exactly: Live applies seq == next, ignores seq < next and turns Recovering on seq > next; on a snapshot that is not older than its book it replaces the book, drops kept messages with seq ≤ last_seq, replays the contiguous rest and goes Live, or keeps waiting if a hole remains. A window above 1 puts the week 46 reorder buffer in front: it holds packets with seq < next + window and declares the oldest holes lost only when a packet beyond the window arrives. The client is compared with the publisher's book at the same sequence number; "live and wrong" counts every moment the client was Live with a different book.