Chaos Engineering

Why this week

By now your exchange has a recovery story for everything: gap recovery (week 10), slow-consumer policies (week 16), crash recovery from snapshot plus journal (week 18), hot-standby failover (week 20) and a kill switch (week 15). Each one was tested once, against the failure you had in mind while writing it. Real failures don't arrive one at a time or in the order you imagined. A standby that lags by 40 ms is fine until the primary dies during a packet-loss burst, while a slow WebSocket client is holding the publisher's oldest buffer.

Chaos engineering is the discipline of finding out before the market does. You form a hypothesis about how the system behaves under a fault, inject the fault deliberately with a bounded blast radius, and check the hypothesis against invariants. For an exchange that mostly means deterministic simulation: every component in one process, a simulated network and clock, faults from a seeded plan, and a failing seed that replays the same bug every time.

Day 1 — Principles, translated for trading

Day 2 — A fault catalogue and a replayable plan

/// Deterministic PRNG: a chaos run must be replayable from its seed.
pub struct XorShift64(u64);

impl XorShift64 {
    pub fn new(seed: u64) -> Self {
        Self(if seed == 0 { 0x9E37_79B9_7F4A_7C15 } else { seed })
    }
    pub fn next_u64(&mut self) -> u64 {
        let mut x = self.0;
        x ^= x << 13;
        x ^= x >> 7;
        x ^= x << 17;
        self.0 = x;
        x
    }
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Fault {
    /// Drop every packet with from <= seq <= to.
    DropRange { from: u64, to: u64 },
    /// Drop each packet with probability per_mille / 1000.
    DropRandom { per_mille: u32 },
    /// Deliver this packet twice in a row.
    Duplicate { seq: u64 },
    /// Deliver this packet right after the next delivered packet.
    SwapNext { seq: u64 },
}

pub struct Plan {
    pub seed: u64,
    pub faults: Vec<Fault>,
}

/// Apply `plan` to `input` (sequence numbers in send order) and return what
/// the receiver sees. Use one XorShift64::new(plan.seed) per call. For each
/// input packet p, in order:
///   1. If any DropRange covers p → drop it (no random draw).
///   2. For each DropRandom in plan order: draw next_u64() % 1000; if the
///      draw < per_mille → drop p (and stop drawing for p).
///   3. If a SwapNext names p → if a packet is already held, emit the held
///      one now; then hold p.
///   4. Otherwise emit p; then, if a packet is held, emit the held one.
/// At the end of input, emit any held packet.
/// "Emit x" = push x, and push it a second time if a Duplicate names x.
pub fn inject(plan: &Plan, input: &[u64]) -> Vec<u64> {
    let _ = (plan, input);
    todo!()
}

Day 3 — The feed handler under loss

use std::collections::BTreeMap;

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Msg {
    /// Set the quantity at `price` to `qty` (0 removes the level).
    Update { seq: u64, price: i64, qty: u64 },
    /// The full book after every update <= seq.
    Snapshot { seq: u64, levels: Vec<(i64, u64)> },
    /// The publisher's next sequence number (sent when the line is quiet).
    Heartbeat { next_seq: u64 },
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Action {
    RequestSnapshot,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Status {
    Live,
    Recovering,
}

pub struct FeedHandler {
    book: BTreeMap<i64, u64>,
    next: u64,                           // next update seq to apply
    status: Status,
    buffered: BTreeMap<u64, (i64, u64)>, // updates received while recovering
}

impl FeedHandler {
    /// Starts Live with an empty book, expecting update 1.
    pub fn new() -> Self {
        Self { book: BTreeMap::new(), next: 1, status: Status::Live, buffered: BTreeMap::new() }
    }

    /// The book, or None while recovering: never serve a book you can't vouch for.
    pub fn book(&self) -> Option<&BTreeMap<i64, u64>> {
        match self.status {
            Status::Live => Some(&self.book),
            Status::Recovering => None,
        }
    }

    pub fn status(&self) -> Status { self.status }
    pub fn next_expected(&self) -> u64 { self.next }

    /// Rules:
    ///  any state: Update with seq < next → already applied, ignore.
    ///  Live:
    ///    Update seq == next → apply, next += 1.
    ///    Update seq > next  → gap: buffer it, go Recovering, return [RequestSnapshot].
    ///    Heartbeat next_seq > next → tail loss: go Recovering, return [RequestSnapshot].
    ///    Snapshot / other Heartbeats → ignore.
    ///  Recovering:
    ///    Update → buffer it (no new request). Heartbeat → ignore.
    ///    Snapshot { seq, levels } → book = levels, next = seq + 1, discard
    ///      buffered <= seq, apply buffered updates while they continue the
    ///      sequence. If nothing is left buffered → Live, return [].
    ///      Otherwise a hole remains → stay Recovering, return [RequestSnapshot].
    pub fn on_msg(&mut self, msg: &Msg) -> Vec<Action> {
        let _ = msg;
        todo!()
    }
}

Day 4 — Deterministic simulation testing

Day 5 — Failover drills and game days

Day 6 — Invariants, written down

Capstone

Build a fault-injection suite in crates/sim, in the chaos module. Every scenario runs in one thread on simulated steps, takes a configuration struct and a FaultPlan, and checks its invariants after every step.

The plan (provided). A FaultPlan is a seed plus Scheduled faults, each active over a window of steps. Per-packet faults act on one Link (md, exec, to-standby or to-primary): Drop, Duplicate, Reorder, Delay and Corrupt, each at a rate in per mille. Partition cuts both replication links, Stall { consumer } stops a reader, and CrashPrimary kills the primary at one step. Display and FromStr use a one-line text form, so seed=7 drop(md,50)@0..800 crash@450 describes a whole run. Fate's docs fix how the faults on one link compose for a packet: a partition loses it; otherwise the random faults are consulted in plan order, and a drop ends the list. shrink_plan and the simulated datagram format in chaos::wire are provided too. You write the Injector (one Fate per packet, logged when it isn't clean), ChaosLink (a UDP-like link: send at a step, recv what is due), the four scenario drivers, the consumers they drive, and the checks.

Four scenarios.

1. Feed faults (FeedSim). The engine's L2 publisher sends each step's updates as checksummed incremental packets, or a heartbeat carrying the next sequence number when there's nothing to send, over a single md line to a FeedClient. The client treats a datagram that fails its checksum as lost, puts updates back in order with the week 46 ReorderBuffer, feeds the week 10 BookBuilder, and asks a snapshot service for help when it falls behind. It is Live only while the builder is synced, it has reached the highest sequence number it knows exists, and the line isn't silent (nothing valid for more than patience steps). Otherwise book() is None. After every step, FeedChecker asserts that a Live client's book is the publisher's book at the client's sequence number, never ahead of the publisher and never going backwards. The run must end Live on the publisher's book after the quiet steps, and live_within can also bound any stretch out of Live. The repo doesn't model A/B lines.
2. Slow consumers (FanoutSim). The publisher writes every update to the week 16 BroadcastRing and stores a periodic snapshot in a SnapshotCell. Several RingConsumers (three by default) each read up to a budget per step, and the plan's Stall faults stop some of them. The writer never waits: a reader stalled for long enough gets lapped, learns how much it missed from Recv::Lagged, and resyncs from the cell. The test asserts that every update is in the ring in the step it was produced, healthy readers never lag, gap or resync, a short stall just catches up, and the lapped reader is never Live with a wrong book and ends on the publisher's book. It then repeats the point on real threads: a reader parked before the first publish can't slow the publisher down. (Week 16's conflation for WebSocket clients isn't part of this scenario.)
3. Leader crash (FailoverSim). The week 12 Pipeline runs over a week 20 Primary, with a Standby running the engine behind it and a ChaosLink in each direction for replication, all on one ManualClock. The plan crashes the primary at a seeded step, partitions the pair, or drops, duplicates, corrupts and delays replication traffic. A command counts as acknowledged only once the standby has acknowledged it. After every step the checks are: never two leaders (the primary while its lease is current, plus the promoted standby), and nothing acknowledged missing from the surviving journal. At promotion: the promoted journal is exactly a prefix of the client's commands, and the promoted engine's state hash equals both a replay of that journal and the reference hash. The run must finish within max_steps or it is Stuck, and failover_steps measures the steps from the crash (or the partition's start) to the promotion. The tests also cover a partition longer than the lease (the primary steps down before the standby takes over, and stays down after the network heals) and a short or one-way partition, where nobody fails over (at worst, order entry pauses while the primary's lease runs out).
4. A faulty venue (ChaosVenue). It wraps any week 23 Venue, so the Runner and its agents run unchanged, and each flush is one step of the plan. Market data goes through an md ChaosLink, and the runner's book recovers with market_snapshot. Execution reports travel on per-account order sessions, and those are TCP: a dropped or corrupted report resets the session, which reconnects at the next flush and resends from the first report the client lacks. A delay stalls everything behind it, a duplicate is a resend the client drops by sequence number, and reordering can't happen inside a stream. After every step, each agent must hold exactly a prefix of its true reports, and the runner's book, when synced, must be the true book. Once the faults stop, every report has arrived exactly once and in order, and positions and cash sum to zero and agree with pre-trade risk and with week 45's position keeping.

Design decisions. Inject at seams the simulator owns (the links between components, the ring reader, the process itself) rather than sprinkling if chaos {} through business logic. The repo has no disk faults: week 18's torn-write tests cover the journal. Express each scenario as data, a config struct plus a FaultPlan, so a failing case is one line of text you can print, save and replay. Check safety invariants after every simulated event. When a seed fails, shrink the fault plan (the idea from week 12's property tests) to the smallest one that still fails before you start debugging. w48_same_seed_same_run_and_a_failing_seed_replays_and_shrinks walks through the whole workflow. It sweeps seeds against a one-lease failover SLO under lossy replication, prints the first failing plan, parses it back and replays it exactly, then runs shrink_plan. On the reference implementation it prints minimal plan: seed=1 delay(to-standby,48,4)@86..221 crash@222. A frame the primary sent just before dying arrives late, and the standby takes it as a sign of life and renews its promise not to take over.

Pitfalls. A simulator that quietly uses Instant::now(), a HashMap iteration or a real thread somewhere isn't deterministic, and a non-replayable failure costs you a day. Faults injected at the wrong layer (dropping bytes from a TCP stream) give false confidence. Asserting only at the end of a run misses transient split-brain. And a suite that only passes because faults are rare is measuring luck: turn the rates up until things break, then fix them.

cargo xtask test --week 48      # tests named w48_*
cargo xtask bench --week 48     # p50 / p99 / p99.9 on your machine
cargo xtask report --upload  # update this page and your lab

Feeds into

No later week lists chaos engineering as a hard prerequisite, but its artifacts travel. The invariant catalogue and a failing-seed story make strong material for the week 49 system design interviews ("how do you know your failover works?"), and the suite's seed count and recovery-time numbers belong in the week 51 README next to the latency figures.