System Design Interviews

Use the same seven steps every time, so your working memory goes to the problem and not the structure:

1. Requirements, as numbers. Functional scope (which order types, how many instruments), the latency target as a percentile and where it's measured (wire-to-wire? gateway-in to ack-out?), peak versus average throughput, fan-out, and correctness requirements: ordering, determinism, recovery point and recovery time, audit.
2. Back-of-the-envelope. Bytes per message, messages per second at peak, bandwidth, memory for the state, cores needed with headroom. Two minutes of arithmetic here prevents twenty minutes of wrong design.
3. Latency budget. Split the end-to-end target across stages before choosing components. If the network hop alone eats 5 µs of a 10 µs budget, that decides the design.
4. Components and threading. Which thread owns which state, what crosses each boundary (queue? shared memory? TCP?), where sequencing happens.
5. Failure modes. For each component: what if it's slow, dead, or wrong? Detection, mitigation, and what clients see.
6. Trade-offs. What you chose, the alternative, and the price: latency versus durability, simplicity versus throughput, fairness versus speed.
7. Evolution and measurement. How you'd know it works (which histograms, which invariants) and what changes at 10× load.

The prompt: "An exchange feed peaks at 3 million messages per second, 40 bytes on average, on a 10 Gb/s link. Normalise it into books and fan it out to 50 strategy processes on the same host. p99 from NIC to strategy must be under 10 µs."

A good answer, compressed:

• Ingress. Subscribe to both the A and B multicast lines, each on its own NIC queue. Use kernel bypass (ef_vi, DPDK or AF_XDP) or a tuned kernel path with busy polling. A pinned feed-handler thread per line, or one thread arbitrating both, sits on the NIC's NUMA node.
• Arbitration and recovery. First copy of each sequence number wins. Gaps go to retransmission, or to a snapshot when they're large, and the book is marked stale while recovering (weeks 46 and 48).
• Book building. Decode zero-copy (week 11), update a cache-friendly book (week 28), one writer thread per book.
• Distribution. A single-writer broadcast ring in shared memory (week 16) for the full update stream, plus a seqlock'd top-of-book per instrument (week 34) for consumers that only want the latest state. No locks, no syscalls per message, and every consumer reads at its own pace.
• Slow consumers. The writer never waits. A consumer that falls more than a ring's worth behind detects the overrun from sequence numbers and resynchronises from the seqlock'd state.
• Budget. NIC to user space ~1–3 µs with bypass (substantially more through the kernel), decode and book update a few hundred ns, publish into the ring ~100 ns, and the cache-line transfer to a consumer on the same socket ~100 ns. That leaves headroom for the tail, which is the point.
• Capacity. 3 M × 40 B × 8 ≈ 1 Gb/s of payload: a 10 Gb/s link is about 10% full on average, but microbursts at line rate are what size your rings.

The drill makes the rubric explicit. You write critique(), the reviewer that catches the classic weak-answer patterns, and then write your own outline for the prompt above and make it pass.

#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum Failure {
    FeedGap,        // packet loss / sequence gap on the inbound feed
    SlowConsumer,   // one consumer can't keep up
    PublisherCrash, // the normalise-and-fan-out process dies
    LinkSaturation, // a burst exceeds what the link or NIC can carry
}

#[derive(Debug, Clone)]
pub struct Stage {
    pub name: &'static str,
    pub budget_ns: u64, // this stage's share of the p99 budget; 0 = not stated
    pub replicas: u32,  // independent instances that can carry the load
}

#[derive(Debug, Clone)]
pub struct Outline {
    pub p99_target_ns: u64,     // end-to-end p99 target; 0 = not stated
    pub peak_msgs_per_sec: u64, // peak inbound rate; 0 = not stated
    pub msg_bytes: u64,         // average message size
    pub link_bits_per_sec: u64, // inbound link capacity
    pub consumers: u32,
    pub stages: Vec<Stage>,     // the critical path, in order
    pub handled: Vec<Failure>,  // failure modes with a stated detection + mitigation
}

#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub enum Issue {
    NoLatencyTarget,                              // p99_target_ns == 0
    NoThroughputTarget,                           // peak_msgs_per_sec == 0
    UnbudgetedStage(&'static str),                // a stage with budget_ns == 0
    OverBudget { total_ns: u64, target_ns: u64 }, // target stated and Σ budgets > target
    SinglePointOfFailure(&'static str),           // a stage with replicas < 2
    LinkOverHalfFull,                             // peak stated and peak × bytes × 8 > link / 2
    Unhandled(Failure),                           // each Failure variant missing from `handled`
}

/// Every issue in `o`, each once, in any order.
pub fn critique(o: &Outline) -> Vec<Issue> {
    let _ = o;
    todo!()
}

/// Your design for: an exchange feed peaking at 3,000,000 msgs/s of 40-byte
/// messages on a 10 Gb/s link, normalised and fanned out to 50 strategy
/// processes on the same host, p99 NIC→strategy <= 10 µs (10_000 ns).
/// Name your stages and budget them; critique(&my_outline()) must be empty.
pub fn my_outline() -> Outline {
    todo!()
}

A handful of formulas cover most of what you'll be asked:

• Ethernet framing. A UDP/IPv4 packet carries 14 + 20 + 8 bytes of headers plus a 4-byte FCS, padded to a 64-byte minimum frame, plus 8 bytes of preamble and a 12-byte inter-frame gap on the wire. A minimum-size frame therefore occupies 84 bytes = 672 bits, and 10 GbE tops out at 14.88 million packets per second. Small messages are dominated by overhead, which is why feeds pack many messages per packet.
• Serialisation delay. A full 1,500-byte IP packet takes ~1.2 µs to serialise at 10 Gb/s and ~12 µs at 1 Gb/s. That's paid at every store-and-forward hop, which is one reason trading networks use cut-through switches and faster links even when bandwidth isn't the bottleneck.
• Microbursts. Average rates hide bursts at line rate lasting microseconds to milliseconds. Size rings, socket buffers and NIC queues for the burst, not the average.
• Cores. messages/s × ns/message is CPU-nanoseconds per second. Divide by 10⁹ for cores at 100%, then by your utilisation ceiling, because queueing delay explodes as a core approaches 100% busy.
• Little's law. Items in flight = arrival rate × time in system. At 2 M msgs/s and 50 µs end to end, 100 messages are in flight on average. That number sizes buffers and tells you how many orders a crash can strand.

Practise these until they're automatic. The quick drills include a latency-number estimation round. Aim to be within a factor of two without looking anything up.

/// Bits on the wire for one UDP/IPv4 packet carrying `payload` bytes over
/// Ethernet: Ethernet header 14 + IPv4 header 20 + UDP header 8 + payload +
/// FCS 4, padded up to the 64-byte minimum frame, plus 8 bytes of
/// preamble/SFD and a 12-byte inter-frame gap.
pub fn wire_bits(payload: u64) -> u64 {
    let _ = payload;
    todo!()
}

/// Packets per second the link can carry at line rate (round down).
pub fn line_rate_pps(link_bps: u64, payload: u64) -> u64 {
    let _ = (link_bps, payload);
    todo!()
}

/// Nanoseconds to serialise one such packet onto the link (round up).
pub fn serialization_ns(link_bps: u64, payload: u64) -> u64 {
    let _ = (link_bps, payload);
    todo!()
}

/// Messages per second at line rate when every packet carries a
/// `pkt_header`-byte feed header plus `per_pkt` messages of `msg` bytes:
/// full packets per second × per_pkt.
pub fn line_rate_msgs(link_bps: u64, pkt_header: u64, msg: u64, per_pkt: u64) -> u64 {
    let _ = (link_bps, pkt_header, msg, per_pkt);
    todo!()
}

/// Cores needed to process `msgs_per_sec` at `ns_per_msg` each without any
/// core above `max_util_pct` percent busy (round up; 0 if there's no load).
pub fn cores_needed(msgs_per_sec: u64, ns_per_msg: u64, max_util_pct: u64) -> u64 {
    let _ = (msgs_per_sec, ns_per_msg, max_util_pct);
    todo!()
}

/// Little's law: average messages in flight = arrival rate × latency (round up).
pub fn in_flight(arrivals_per_sec: u64, latency_ns: u64) -> u64 {
    let _ = (arrivals_per_sec, latency_ns);
    todo!()
}

"Design a stock exchange" is the question where your year shows. The structure your answer should follow is the one you built:

• Gateways terminate sessions (FIX or binary), validate, throttle and run pre-trade risk. They scale horizontally and hold no book state.
• A sequencer assigns one total order to every inbound command. This is where fairness is defined: first sequenced, first matched.
• The matching engine is single-threaded per instrument, or per shard of instruments: a deterministic state machine over the sequenced stream. Single-threaded because price-time priority is inherently sequential per book, and because determinism makes replication and replay trivial. Throughput comes from mechanical sympathy and sharding, not locks.
• The journal and replication. The sequenced input is written to a journal and replicated to a hot standby that runs the same deterministic engine. Acknowledge after replication. Fail over on lease expiry with exactly one leader.
• Market data is generated from engine output events: sequenced L2/L3 incrementals, snapshots for recovery, A/B multicast.
• Drop copies and reporting consume the same event stream off the hot path.

Expect the follow-ups: How do you guarantee no double fill on failover? (Deterministic replay of the replicated sequence, plus idempotent client-visible ids.) Why not a database? (Fsync on every order costs tens of µs to ms. Replication to memory on a standby is faster, and the journal is still written for durability.) How do you add instruments? (Shard by symbol, one sequencer per partition, week 36.) What about cross-instrument orders? (They're hard. Say so, and describe the coordination cost.)

The buy-side question is usually "design an order management system" or "a smart order router". The heart of a good answer is the order state machine and the races around it:

• States: pending-new → new (acked) → partially filled → filled, plus pending-cancel, pending-replace, cancelled, rejected and expired. Every transition is driven by an execution report, and you must decide what to do with reports that arrive "impossibly". A fill can arrive after you've sent a cancel, because the cancel crossed it on the wire. That's not an error. The venue says "too late to cancel", and your position must reflect the fill.
• Positions and risk across venues. The router splits a parent order into child orders on several venues, so the open quantity across all of them counts against limits (week 15). A fast fill on venue A must reduce what you're willing to send to venue B.
• Identity. Client order ids for every child, mapping back to the parent. Make cancel/replace chains explicit.
• Recovery and reconciliation. After a disconnect, resynchronise from sequence numbers (FIX resend, week 14), and reconcile against the venue's drop copy at the end of the day. A position that disagrees with the venue is an incident.
• Latency. Routing decisions use the freshest book you have, and the latency to each venue is part of the decision.

Run at least three timed sessions with the AI system-design interviewer: one exchange, one market-data system, one OMS. Treat them like the real thing: 45 minutes, no notes, talk continuously, draw as you go. Afterwards, review the transcript against the seven-step framework. Where did you name a technology instead of a mechanism? Where did you skip a number? Which failure mode did the interviewer have to drag out of you?

The common failure patterns are predictable: designing before clarifying, a budget that doesn't add up, a single component with no failure story, "Kafka" on a microsecond path, and running out of time before failure modes. Fix one per session.