Market Replay from Captures

There are three places to record a feed, and they give different answers:

• Host capture. tcpdump on the trading server, using libpcap over AF_PACKET. It's cheap and needs nothing extra. It's also the weakest option. The kernel timestamps packets in software somewhere in the receive path, the capture ring can overflow under bursts (tcpdump reports "packets dropped by kernel" when it exits), and the capture competes with your application for CPU and cache. Use it on loopback in development. Don't build production evidence on it.
• Switch mirror (SPAN). The switch copies a port's traffic to a capture port. It's convenient, but the mirror is best effort: an oversubscribed mirror port drops packets silently, and the switch may reorder traffic across mirrored ports.
• Passive TAP plus a capture appliance. An optical tap splits the light, and an FPGA-based capture card timestamps each frame as it arrives, typically with nanosecond resolution and a PTP-disciplined clock. This is what firms use for compliance records and for measuring wire-to-wire latency.

The timestamp is the part people get wrong. A software timestamp includes interrupt coalescing, softirq scheduling and anything else the kernel did first, so tens of microseconds of jitter under load is normal. A hardware timestamp at the tap is taken before any of that. When you later compute "feed packet in → order packet out" from a capture, both timestamps come from the same clock at the same point in the network. That's why a capture is the most trustworthy tick-to-trade measurement you can get.

A classic pcap file is a 24-byte file header followed by records. Each record is a 16-byte header plus incl_len captured bytes:

file header (24 B)   magic u32 | ver_major u16 | ver_minor u16 | thiszone i32 | sigfigs u32 | snaplen u32 | linktype u32
record header (16 B) ts_sec u32 | ts_frac u32 | incl_len u32 | orig_len u32
record data          incl_len bytes (the start of the frame)

The magic tells you two things. 0xa1b2c3d4 means the fractional timestamp is in microseconds, and 0xa1b23c4d means nanoseconds. The header fields are in the writing host's byte order: if the magic reads back as 0xd4c3b2a1, every header field in the file is byte-swapped. The packet bytes inside a record are untouched, and network protocols are big-endian ("network byte order"). So a typical file written on x86 has little-endian pcap headers wrapped around big-endian protocol fields. Mixing the two up is the most common pcap bug.

incl_len < orig_len means the capture was cut at snaplen. That's fine if you only need headers, and fatal if you need the whole feed message, so a replay tool should count truncated records and refuse to build books from them. The newer pcapng format starts with a Section Header Block (0x0A0D0D0A) and isn't a classic pcap file; detect it and say so instead of reporting a corrupt file.

Inside each record (linktype 1, Ethernet) you peel headers off the frame in order:

// Ethernet II: dst MAC 6 | src MAC 6 | EtherType 2. All multi-byte
// protocol fields are big-endian.
fn udp_payload(frame: &[u8]) -> Option<(u16, &[u8])> {
    let mut off = 12;
    let mut ethertype = u16::from_be_bytes([frame[off], frame[off + 1]]);
    if ethertype == 0x8100 {                    // 802.1Q VLAN tag: 4 more bytes
        off += 4;
        ethertype = u16::from_be_bytes([frame[off], frame[off + 1]]);
    }
    if ethertype != 0x0800 { return None; }     // not IPv4
    let ip = &frame[off + 2..];
    let ihl = ((ip[0] & 0x0f) as usize) * 4;   // usually 20, never assume it
    if ip[9] != 17 { return None; }             // protocol 17 = UDP
    let udp = &ip[ihl..];
    let dst_port = u16::from_be_bytes([udp[2], udp[3]]);
    let udp_len = u16::from_be_bytes([udp[4], udp[5]]) as usize; // header + payload
    Some((dst_port, &udp[8..udp_len]))
}

(Production code checks every slice length before indexing. A capture is untrusted input, and a truncated record will turn each [] above into a panic.)

The UDP payload is the feed's own framing. Nasdaq's MoldUDP64, which carries ITCH, is a good model because many binary feeds use the same shape: a 20-byte header (10-byte session id, 8-byte sequence number of the first message, 2-byte message count), then each message as a 2-byte length followed by the message bytes. One packet carries messages seq .. seq + count − 1. A count of zero is a heartbeat that tells you the next expected sequence number, and 0xFFFF marks end of session. The repo's feed packet for this week (feed.rs, described in the capstone) has the same shape.

Now write the record parser. It borrows the payload from the input buffer instead of copying it: a 10 GB capture is replayed by walking a memory map, not by allocating 100 million Vecs.

// libpcap savefile format (see pcap-savefile(5)).
//
// File header, 24 bytes:
//    0  magic          u32  0xa1b2c3d4 = µs timestamps, 0xa1b23c4d = ns timestamps
//    4  version_major  u16  (2)
//    6  version_minor  u16  (4)
//    8  thiszone       i32  (0 in practice)
//   12  sigfigs        u32  (0 in practice)
//   16  snaplen        u32  max bytes stored per packet
//   20  linktype       u32  1 = Ethernet
// The magic is written in the capturing host's byte order: read it as
// little-endian; 0xd4c3b2a1 / 0x4d3cb2a1 mean the whole file is big-endian.
//
// Record header, 16 bytes, in the file's byte order:
//    0  ts_sec    u32
//    4  ts_frac   u32  microseconds or nanoseconds, per the magic
//    8  incl_len  u32  bytes stored after this header
//   12  orig_len  u32  bytes that were on the wire

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Endian { Little, Big }

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct FileHeader {
    pub endian: Endian,
    pub nanos: bool,
    pub snaplen: u32,
    pub linktype: u32,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct RecordHeader {
    pub ts_ns: u64, // nanoseconds since the Unix epoch
    pub incl_len: u32,
    pub orig_len: u32,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PcapError {
    TooShort,  // fewer bytes than the header itself needs
    BadMagic,  // not a classic pcap file (pcapng starts with 0x0A0D0D0A)
    Truncated, // incl_len bytes should follow, but the buffer ends first
    Corrupt,   // incl_len > snaplen, or incl_len > orig_len
}

pub fn parse_file_header(buf: &[u8]) -> Result<FileHeader, PcapError> {
    // TODO: length check, magic → (endian, nanos), then snaplen and
    // linktype in the file's byte order.
    let _ = buf;
    todo!()
}

/// Parse the record that starts at buf[0]. Returns the header, the captured
/// bytes (a sub-slice of `buf`, no copy) and how far to advance (16 + incl_len).
/// Check Corrupt before Truncated: a garbage length usually also looks short.
pub fn parse_record<'a>(
    file: &FileHeader,
    buf: &'a [u8],
) -> Result<(RecordHeader, &'a [u8], usize), PcapError> {
    let _ = (file, buf);
    todo!()
}

Exchanges publish every feed twice, on an A line and a B line: separate multicast groups, often over physically separate networks. A feed handler listens to both and keeps the first copy of each sequence number. This is A/B arbitration. It turns "a switch dropped a packet on A" into a non-event. It does nothing about a packet the publisher never sent or lost before the split, because then both lines miss the same sequence number.

When a sequence number is missing on both lines you have a gap, and three questions:

1. How long do you wait? Packets on two paths can arrive out of order. Wait too briefly and you declare false gaps. Wait too long and your book is stale while you wait. Real handlers use a small time budget (often microseconds to low milliseconds, tuned per feed), a count limit, or both. The drill uses a count window because it's deterministic.
2. How do you recover? Either ask for the missing range (MoldUDP64 re-request servers; CME MDP 3.0 offers TCP replay for small ranges) or discard the book and rebuild it from a snapshot (CME's market recovery feeds loop snapshots continuously). Small gaps go to retransmission. Large gaps, or gaps after a restart, go to snapshot.
3. What do consumers see meanwhile? The book must be marked stale. Week 48 makes you prove this under random loss.

Turn reordering on with a window of 1 and count the gaps that were really just late packets. Then widen the reorder window until they stop costing snapshot waits, and watch what the window costs in held packets. Turn on the B line to see arbitration hide single-line loss.

In a replay tool you have one extra advantage: if the capture contains both lines, arbitration recovers almost everything, and whatever is left is a real gap that any handler behind the same capture point must also have seen. Report it, with the sequence range and capture timestamp. That's often the bug you were looking for.

Build the reorder buffer. It's the core of every feed handler: it accepts packets in any order, delivers them in sequence order exactly once, and gives up on a hole only when it has no room left to wait.

use std::collections::BTreeMap;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Out {
    /// Deliver this packet to the book builder.
    Packet { seq: u64, payload: u32 },
    /// Sequence numbers from..=to are declared lost: recover them by
    /// retransmission or a snapshot before trusting the book.
    Gap { from: u64, to: u64 },
}

pub struct ReorderBuffer {
    next: u64,                   // next sequence number we may deliver
    window: u64,                 // buffer packets with seq < next + window (window >= 1)
    pending: BTreeMap<u64, u32>, // early packets waiting for the hole to fill
    duplicates: u64,
}

impl ReorderBuffer {
    pub fn new(first_seq: u64, window: u64) -> Self {
        Self { next: first_seq, window, pending: BTreeMap::new(), duplicates: 0 }
    }

    /// Rules, in order:
    ///  1. seq < next, or seq already buffered → duplicate: count it, return [].
    ///  2. seq >= next + window → stop waiting for the oldest holes: walk
    ///     s = next .. seq - window + 1, delivering buffered packets and
    ///     reporting each run of missing s as ONE Gap, in sequence order;
    ///     set next = seq - window + 1, then deliver any buffered run from
    ///     there. Then continue with rule 3 for this packet.
    ///  3. seq == next → deliver it, then every consecutive buffered packet.
    ///  4. otherwise → buffer it.
    pub fn push(&mut self, seq: u64, payload: u32) -> Vec<Out> {
        let _ = (seq, payload);
        todo!()
    }

    pub fn next_expected(&self) -> u64 { self.next }
    pub fn buffered(&self) -> usize { self.pending.len() }
    pub fn duplicates(&self) -> u64 { self.duplicates }
}

Week 9 made the engine a deterministic function of its input log. Replay extends that promise to the consumer side: the same capture must produce the same sequence of book states on every run and every machine. The usual ways to break it:

• Wall-clock reads. Anything that calls Instant::now() or SystemTime::now() during replay (staleness timers, conflation intervals, timeouts) makes output depend on how fast the machine is. Drive every clock in the replay from the capture timestamps, using the injected clock from weeks 17 and 21.
• Hash-map iteration order. std::collections::HashMap uses a randomly seeded RandomState, so iteration order differs between processes. If you ever emit levels, symbols or orders by iterating a HashMap, two replays disagree. Use BTreeMap, a Vec sorted by key, or a fixed hasher where iteration order matters.
• Thread interleaving. If you shard replay across threads (by instrument, as in week 36), each shard is deterministic but the interleaving of their outputs isn't. Merge on (capture timestamp, sequence number), never on arrival order at a channel.
• Floating point. Fixed-point prices in ticks are exact. A float average computed in a different order can differ in the last bit, and a state hash will catch it.

Verification is what makes replay trustworthy. The publisher already emits periodic snapshots (week 10). Record their state hash together with the sequence number they correspond to, and at that sequence number in replay compare your rebuilt book's hash against it. One mismatch tells you exactly which interval to bisect.

Replay runs in two modes. As fast as possible is for regression tests and backtests (week 44): the capture is just input and timestamps are just data. Paced (real time, or ×10) is for watching an incident unfold in a UI. Pacing belongs in a thin layer on top. The core replay must never sleep.

A typical incident: "At 14:31:07.1234 our strategy saw a best offer of 100.25 and sent an IOC buy at 100.25. Nothing filled. The exchange says that offer had been gone for 3 ms." Your workflow:

1. Cut the window. editcap -A "<start>" -B "<stop>" day.pcap window.pcap keeps a few seconds around the event. Use mergecap if the A and B lines were captured on different ports. Keep enough history before the event to start from a snapshot.
2. Replay to the moment. Start from the last snapshot before the window, replay incrementals up to the order's timestamp, and dump the book. Compare it with what the strategy logged. If they differ, the bug is in the strategy's feed handler (a missed gap, a wrong decode, a stale book used as live). If they match, the price moved between decision and arrival: compute the gap between the strategy's decision and its order reaching the wire, from the same capture.
3. Bisect with hashes. If the replayed book itself looks wrong, compare state hashes against the recorded snapshots and bisect to the first divergent sequence number. Look at that message's bytes.
4. Freeze it. Check the trimmed capture into the test fixtures with the assertion that would have caught it. From now on, this incident is a unit test.

Captures also settle latency arguments. With tap timestamps on the feed packet that triggered a decision and on the order packet that followed, order_ts − trigger_ts is the strategy's wire-to-wire tick-to-trade. No software clock on the host can be trusted for that.

Read the capstone below, then plan the pipeline as a chain of borrowed slices: &[u8] file → record → frame → UDP payload → feed packet → message. Nothing in that chain needs to allocate. The only state that grows is the book.