Market Data: Books on the Wire

• L1 (top of book): best bid and ask with their sizes, and the last trade. Enough for a quote display and for many execution algorithms.
• L2 (market by price, MBP): the aggregate quantity at each price, usually for the best N levels per side. Most trading strategies use this: depth shows liquidity, and level-by-level changes show pressure.
• L3 (market by order, MBO): every individual order — added, partially executed, cancelled, replaced — identified by an order reference. It is the only view that shows queue position: where a given order sits within its level, and therefore how likely it is to fill (week 41). Market makers pay for it for that reason.

Real feeds span the range: Nasdaq TotalView-ITCH is MBO; CME's MDP 3.0 publishes market-by-price books to a fixed depth and, for many products, an MBO feed alongside. An exchange that has L3 can always derive L2 and L1; the reverse is impossible.

Turning an order-by-order stream into price-level updates is a small state machine: keep each live order's side, price and remaining quantity; keep each level's aggregate; for every L3 event, adjust the aggregates it touches and emit one update per level whose total actually changed, carrying the new total.

Most events touch one level. Replace is the interesting one: the old order leaves its level and the new one joins at a possibly different price. If the prices differ, two levels change and two updates go out, old level first. If they are the same, the level's total changes by the difference — or not at all, in which case nothing goes out even though the old order lost its queue position. That case is a neat summary of what L2 hides.

The harness ends with 5,000 random events and a consumer that sees only your L2 updates: after every event, its view must equal an independent aggregation of the live orders, and every update must change something.

use std::collections::{BTreeMap, HashMap};

#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Debug)]
pub enum Side {
    Bid,
    Ask,
}

/// Order-by-order (L3) events, ITCH-style.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum L3 {
    Add { id: u64, side: Side, price: i64, qty: u64 },
    /// Part (or all) of a resting order traded.
    Execute { id: u64, qty: u64 },
    /// Part (or all) of a resting order cancelled; `qty` is the amount removed.
    Cancel { id: u64, qty: u64 },
    /// The whole remaining order removed.
    Delete { id: u64 },
    /// `old_id` leaves the book; `new_id` joins at the back of `price` with
    /// `qty`, on the same side.
    Replace { old_id: u64, new_id: u64, price: i64, qty: u64 },
}

/// "Level `price` on `side` now holds `qty` in total" — absolute, not a
/// delta; qty 0 means the level is gone.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub struct L2Update {
    pub side: Side,
    pub price: i64,
    pub qty: u64,
}

pub struct L2Builder {
    orders: HashMap<u64, (Side, i64, u64)>,
    levels: BTreeMap<(Side, i64), u64>,
}

impl L2Builder {
    pub fn new() -> Self {
        L2Builder { orders: HashMap::new(), levels: BTreeMap::new() }
    }

    /// Apply one L3 event and return the L2 updates it causes:
    ///   - one update per level whose aggregate changed, carrying the new total;
    ///   - Replace across two levels: the old level first, then the new one;
    ///     Replace within one level: one update, or none if the total is unchanged;
    ///   - Execute/Cancel for more than the order holds: clamp to what is left;
    ///   - unknown ids and duplicate Add ids: no updates, no change.
    pub fn apply(&mut self, ev: &L3) -> Vec<L2Update> {
        todo!()
    }

    /// Current aggregate at a level (0 if empty).
    pub fn level_qty(&self, side: Side, price: i64) -> u64 {
        todo!()
    }
}

Every feed has two kinds of message:

• Incrementals describe changes as they happen. They are small and frequent, and a consumer must apply every one of them, in order.
• Snapshots describe the whole book (or its top N levels) as of a stated sequence number. They are large and periodic, and they exist so that a consumer that joins late — or loses messages — can start from a known state.

Within incrementals, there is a choice that shapes everything downstream. A delta says "level 100 changed by −5". An absolute update says "level 100 now holds 37". Deltas are compact and natural to generate, but applying one twice, or missing one, silently corrupts the consumer's book forever. Absolute updates are idempotent (applying one twice is harmless) and conflatable (if three updates to level 100 are queued, only the last matters). For L2 feeds, absolute level quantities are the safer default; L3 feeds are necessarily event-based, which is one reason they need stricter recovery.

One more property matters to consumers: event boundaries. One aggressive order can change five levels and print five trades. A consumer that reads the book between the third and fourth update sees a state that never existed — the order half-executed. Feeds mark the end of each matching event so consumers act only on consistent states; CME's MDP 3.0, for example, flags the last message of each event in its MatchEventIndicator field (tag 5799). Your publisher should emit all the updates caused by one input command together and mark where they end.

Real market data travels over UDP multicast: one packet from the exchange reaches every subscriber, with no acknowledgements and no retransmission. Packets get lost (a full socket buffer, a congested switch port), so feeds are designed around detecting and repairing loss:

• Sequence numbers. Every message (or packet) carries one. A consumer expecting n that receives n + 3 knows two are missing.
• A/B line arbitration. Exchanges publish every packet twice, on two multicast groups over separate network paths. Consumers take the first copy of each sequence number and drop the second. Most single-path losses disappear here.
• Recovery for losses on both lines: request the missing range from a retransmission server (Nasdaq's MoldUDP64 re-request service), or rebuild from a snapshot. CME's MDP 3.0 publishes snapshots continuously on separate recovery channels; each snapshot carries the sequence number of the last incremental it includes (LastMsgSeqNumProcessed, tag 369).

The snapshot procedure is what you implement today. On a gap: stop applying incrementals, keep buffering them, wait for a snapshot. When one arrives that is not older than what you already have, replace the book with it, then apply the buffered incrementals that come after its sequence number, in order, skipping duplicates. If they run contiguously to the end, you are live again; if there is still a hole, keep waiting for a newer snapshot. Throughout, the book you expose is either correct or explicitly marked as recovering — never silently wrong.

Raise the packet loss and step through a gap: the client goes Recovering and buffers, then, when a snapshot arrives, drops every buffered message at or below its sequence number and replays the rest. Keep an eye on "Live and wrong": it never leaves zero.

use std::collections::BTreeMap;

#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Side {
    Bid,
    Ask,
}

/// Incremental L2 update: set level (side, price) to qty (0 deletes it).
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub struct Incr {
    pub seq: u64,
    pub side: Side,
    pub price: i64,
    pub qty: u64,
}

/// The full book as of incremental `last_seq`.
#[derive(Clone, PartialEq, Eq, Debug)]
pub struct Snapshot {
    pub last_seq: u64,
    pub levels: Vec<(Side, i64, u64)>,
}

#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Status {
    Live,
    Recovering,
}

pub struct BookBuilder {
    bids: BTreeMap<i64, u64>,
    asks: BTreeMap<i64, u64>,
    next_seq: u64,
    status: Status,
    pending: Vec<Incr>,
    gaps: u32,
}

impl BookBuilder {
    /// Has the (empty) book as of seq 0; expects seq 1 next.
    pub fn new() -> Self {
        BookBuilder { bids: BTreeMap::new(), asks: BTreeMap::new(), next_seq: 1, status: Status::Live, pending: Vec::new(), gaps: 0 }
    }

    /// Live:
    ///   seq == next  → apply it, next += 1
    ///   seq <  next  → duplicate (e.g. the other feed line): ignore
    ///   seq >  next  → gap: gaps += 1, status = Recovering, keep the message
    /// Recovering: keep every message; do not touch the book.
    pub fn on_incremental(&mut self, m: Incr) {
        todo!()
    }

    /// Live: ignore snapshots.
    /// Recovering:
    ///   - a snapshot older than the book you already have
    ///     (last_seq + 1 < next) cannot help: ignore it;
    ///   - otherwise replace the book with it, set next = last_seq + 1, then
    ///     apply kept messages in seq order: skip seq < next (stale or
    ///     duplicate), apply seq == next, stop at a hole (seq > next);
    ///   - no hole → Live (keep nothing); hole → stay Recovering and keep only
    ///     the messages beyond it.
    pub fn on_snapshot(&mut self, s: &Snapshot) {
        todo!()
    }

    pub fn status(&self) -> Status {
        self.status
    }
    pub fn next_seq(&self) -> u64 {
        self.next_seq
    }
    pub fn gaps_detected(&self) -> u32 {
        self.gaps
    }

    pub fn level(&self, side: Side, price: i64) -> u64 {
        todo!()
    }
    pub fn best_bid(&self) -> Option<(i64, u64)> {
        todo!()
    }
    pub fn best_ask(&self) -> Option<(i64, u64)> {
        todo!()
    }
}

A matching engine can change a book hundreds of thousands of times a second; a browser tab cannot repaint that often, and a risk dashboard does not need to. Conflation sends such consumers the latest state of each changed level per interval instead of every change: keep a set of dirty levels, and every N milliseconds publish one absolute update per dirty level (or a fresh top-of-book snapshot) and clear the set.

Conflation only works on state-based data. Absolute L2 updates conflate naturally; L3 order events do not, because dropping an add or a cancel loses an order's existence. That is why exchanges run separate feeds for separate audiences: full-rate, lossless incrementals for co-located participants, and conflated or throttled feeds for everyone else. Week 16 builds the fan-out that serves both from one engine without letting a slow consumer slow anyone else.

The publisher sits behind the engine and owns the feed's contract:

• It consumes the engine's events in order, updates its own per-level aggregates, and emits L2 updates only for levels that changed — the drill's state machine, now fed by your engine.
• It assigns feed sequence numbers, independent of the engine's input sequence numbers: one input command can produce zero or several messages.
• It groups all messages caused by one input command into one batch with an end-of-event marker, and packs batches into packets.
• It takes snapshots at batch boundaries only, each labelled with the last feed sequence number it includes, so a snapshot never contains half an event.