Hot Path / Cold Path

Before an instruction executes, the CPU's front end has to predict where control goes next, fetch the bytes, and decode them into micro-ops. The structures involved are small:

• L1 instruction cache: 32 KiB on most Intel and AMD server cores of the last decade (a few newer designs have 64 KiB), in 64-byte lines. A miss that hits L2 costs on the order of 10–15 cycles; one that has to go to L3 costs several times more.
• Decoded µop cache: a few thousand already-decoded micro-ops (the size varies a lot by generation). Code that runs from it skips the legacy decoders, which are slower and narrower.
• Branch predictors and the branch target buffer: tables, indexed by branch address and history, that predict the direction and target of every branch before it's decoded. They have finite capacity, and cold code with many branches competes for it.
• Instruction TLB: a big binary whose hot code is spread over many 4 KiB pages pays iTLB misses.

Cold code hurts the hot path even when it never runs. A 64-byte line holding six hot instructions and forty bytes of an error handler wastes most of a fetch. A hot function spread across three pages because of inlined formatting code needs three iTLB entries instead of one. And every branch around a cold block is one more entry in the predictor's tables.

Measure the front end before changing anything. perf stat -e L1-icache-load-misses,iTLB-load-misses,branch-misses on your engine benchmark gives you the raw counts. On CPUs that support it, perf stat --topdown splits stall cycles into front-end bound, back-end bound, bad speculation and retiring. That's Ahmad Yasin's top-down method, built into Intel's PMU. A hot loop in a microbenchmark will look 95% retiring. Run the same code interleaved with realistic other work and watch the front-end-bound share grow.

Rust gives you three levers, and they compose:

• #[cold] on a function tells LLVM that calls to it are unlikely. Branches that lead to a cold call get low weight, so the compiler lays out the other side as the straight-line fall-through and moves the cold block to the end of the function. The cold function itself is optimised with less inlining around it, and on ELF targets it may be placed in a separate .text.unlikely section, away from hot code.
• #[inline(never)] keeps the cold function's body from being inlined back into the hot function. Without it, a small cold helper can still be inlined, and its instructions come back.
• std::hint::cold_path() marks one branch arm as cold without creating a function. It's stable only in recent Rust releases. On an older toolchain, calling an empty #[cold] function in that arm does the same job. likely/unlikely intrinsics exist but are still unstable.

#[cold]
#[inline(never)]
fn reject(code: RejectCode) -> Result<Accepted, Reject> {
    Err(Reject { code })            // the slow, rare work goes here
}

#[inline]
pub fn validate(o: &Order, lim: &[Limits]) -> Result<Accepted, Reject> {
    let Some(l) = lim.get(o.instrument as usize) else { return reject(RejectCode::UnknownInstrument) };
    if o.price.rem_euclid(l.tick) != 0 { return reject(RejectCode::BadTick) }
    // ... each failing check jumps out of line; the success path falls straight through
    Ok(Accepted { /* ... */ })
}

Panics are already handled this way: the standard library's panic entry points are cold and out of line, which is why an unwrap() costs only a compare and a branch on the hot path. Watch for eager work in the arguments, though. .expect(&format!("bad order {id}")) formats a String on every call, success or not. Logging macros expand formatting setup at the call site, so keep them out of hot functions entirely (week 19's binary log records exist for this). And keep the error type small: a Result<T, E> with a large E makes every return larger, and an E that owns a String gives every caller drop glue on the error path and an allocation per reject.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RejectCode {
    UnknownInstrument,
    BadTick,
    OutsideBand,
    BadLot,
    TooLarge,
}

#[derive(Clone, Copy, Debug)]
pub struct Order {
    pub instrument: u32,
    pub price: i64,
    pub qty: u32,
}

#[derive(Clone, Copy, Debug)]
pub struct Limits {
    pub tick: i64,
    pub lot: u32,
    pub band_lo: i64,
    pub band_hi: i64,
    pub max_qty: u32,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Accepted {
    pub instrument: u32,
    pub price: i64,
    pub qty: u32,
}

/// The error the hot path returns. As written it carries a formatted
/// message: every reject allocates, and every caller inherits drop glue.
/// Slim it down (keep `code()`), and build the text in `describe` instead.
#[derive(Debug)]
pub struct Reject {
    code: RejectCode,
    message: String,
}

impl Reject {
    pub fn code(&self) -> RejectCode {
        self.code
    }
}

/// Hot path. Checks in this order: unknown instrument (index out of
/// range), tick (price must be a multiple), band (inclusive), lot (qty
/// must be a positive multiple), max qty.
pub fn validate(o: &Order, limits: &[Limits]) -> Result<Accepted, Reject> {
    todo!()
}

/// Cold path: the human-readable reject text, built only when someone
/// actually logs it. Exact formats:
///   UnknownInstrument  "unknown instrument 7"
///   BadTick            "price 10005 is not a multiple of tick 10"
///   OutsideBand        "price 20000 outside band [9000, 11000]"
///   BadLot             "qty 150 is not a positive multiple of lot 100"
///   TooLarge           "qty 50000 exceeds max 10000"
pub fn describe(o: &Order, limits: &[Limits], r: &Reject) -> String {
    todo!()
}

Two of the harness checks can't be expressed as behaviour, so they inspect the type: size_of::<Reject>() must be at most 8 bytes, and std::mem::needs_drop::<Reject>() must be false. After it passes, put validate into Compiler Explorer with and without #[cold] on your reject helper and compare where the reject blocks end up.

Moving rejects out of line protects the hot path. That isn't permission to make rejects slow. The moments when rejects spike are exactly the moments the exchange is under the most stress: a fat-fingered algorithm hammering a price band during a fast market, a client reconnecting with a stale sequence, a misconfigured lot size after a corporate action. If each reject allocates a String, formats it and writes a log line synchronously, a reject storm becomes a latency incident for every client.

A design that holds up:

• The reject is a value: a one-byte code plus the order id the caller already has. It's Copy, there's nothing to drop, and constructing it costs nothing.
• Counting stays on the engine thread, formatting doesn't. Increment a per-reason counter (plain u64s owned by the engine thread, published like week 19's metrics). Send a compact binary reject record to the logging thread. The text is built there, or never.
• The client still gets a precise reason: the reject code maps to the protocol's reject reason field. The gateway (or the encoder) formats any human-readable text off the engine thread.
• Rare order types are cold functions too. FOK feasibility scans, post-only cross handling and anything else most orders never touch go behind #[inline(never)] functions called from one branch, so the plain-limit-order path stays short.

A modern core predicts every conditional branch before decoding it and speculates down the predicted path. When it's right, the branch costs almost nothing. When it's wrong, all the work started on the wrong path is thrown away and the pipeline refills. That costs roughly 15–20 cycles on recent x86 cores, about 5 ns at 4 GHz. It's the same order as an L2 miss, and it can happen on every iteration of a loop.

How predictors learn, from simplest to what you actually run on:

• Bimodal: a table of 2-bit saturating counters indexed by the branch's address. A branch that's almost always taken (or almost never) is predicted perfectly after one miss. A branch that alternates defeats it completely.
• Global history (gshare): index the counters by the branch address XORed with the outcomes of the last n branches (Scott McFarling, 1993). Now patterns are learnable: alternating, every third time, a loop that exits every 8th iteration, if the history is long enough to cover the pattern.
• Modern predictors (TAGE-style designs using several history lengths at once, perceptron-based predictors) learn long and correlated patterns remarkably well. What no predictor can learn is randomness: a branch on data that's genuinely unpredictable, such as "is this order a buy?" on mixed flow or "is this price at or above the threshold?" on shuffled data, is mispredicted about half the time.

The famous demonstration is summing the elements of an array that pass a threshold. It runs several times faster when the array is sorted, because the branch becomes "false, false, … false, true, true, … true", which the predictor gets wrong only once or twice. You'll build the predictors and count the mispredictions exactly, which is deterministic and needs no noisy timer, and then see the timing effect on real hardware.

/// A table of 2-bit saturating counters. 0 and 1 predict not-taken, 2 and 3
/// predict taken. Every counter starts at 1 (weakly not-taken).
pub struct Predictor {
    table: Vec<u8>,
    history: u32,      // last `history_bits` outcomes, newest in bit 0 (taken = 1)
    history_bits: u32, // 0 = bimodal (no history)
}

impl Predictor {
    /// `entries` is a power of two. Index = pc & (entries - 1).
    pub fn bimodal(entries: usize) -> Self {
        todo!()
    }

    /// gshare: index = (pc ^ history) & (entries - 1). history_bits <= 16.
    pub fn gshare(entries: usize, history_bits: u32) -> Self {
        todo!()
    }

    /// Predict this branch, then train the counter on the real outcome
    /// (saturating at 0 and 3) and shift the outcome into the history.
    /// Returns true if the prediction was right.
    pub fn predict_and_update(&mut self, pc: u32, taken: bool) -> bool {
        todo!()
    }
}

/// Feed every outcome through the predictor; return how many were mispredicted.
pub fn count_mispredicts(p: &mut Predictor, pc: u32, outcomes: &[bool]) -> usize {
    todo!()
}

/// Sum of xs[i] where xs[i] >= t, with no data-dependent branch:
/// turn the comparison into an all-ones / all-zeros mask.
pub fn sum_at_least_branchless(xs: &[u32], t: u32) -> u64 {
    todo!()
}

About the timings in the last case: in a debug build (the Playground's default) you should see the branchy loop run noticeably slower on shuffled data than on sorted data, while the branchless loop doesn't care. In a release build, LLVM often turns the branchy version into a conditional move or vectorises it, and the gap disappears. That's the compiler doing what you just did by hand. Check the assembly before claiming either result.

When a branch is genuinely unpredictable, remove it:

• Arithmetic and masks, as in the drill: compute both outcomes and select with a mask.
• Conditional moves: the compiler emits cmov for simple selects. A cmov never mispredicts, but it makes the result depend on both inputs, which can lengthen a dependency chain. When the branch was predictable, the branchy version is often faster.
• Structure instead of branching. The exchange's most common data-dependent branch is the order's side. Instead of if side == Buy { … } else { … } throughout the matching code, index by side: two ladders in an array books[side], with comparison direction encoded once. The branch disappears into an address calculation.

When a branch is predictable but rare, keep the branch and fix the layout. Profile-guided optimisation does that from evidence instead of annotations. Build an instrumented binary (-Cprofile-generate), run a representative workload, rebuild with -Cprofile-use, and the compiler places hot blocks together, orders branches by observed frequency and inlines based on real call counts. BOLT goes a step further and rearranges the linked binary from a perf profile. rustc itself is built with PGO and, on Linux, BOLT. Treat your hand-placed #[cold] annotations as a baseline and compare them against a PGO build. If PGO beats you, keep PGO in the release pipeline (week 22) and keep the annotations as documentation.

Finally, the trading-specific trick. The deepest cause of tail latency on the order path is that it hasn't run recently: between orders, other code evicted it from the caches and the predictor. Firms keep the path warm by running it continuously: the engine or strategy periodically pushes a dummy order through the whole path, flagged so it's discarded at the very end (before the journal, or before the NIC). Carl Cook's CppCon talk describes this technique, along with much of this week.

Write down the plan before touching code:

1. The baseline. Your current engine's p50/p99/p999 submit-to-events latency, the size of the hot function (cargo asm line count, or nm --size-sort on the binary), and perf stat front-end counters.
2. What moves. From Day 1's inventory: reject construction and text, logging, rare order types, debug assertions you don't need in release, anything that formats.
3. A benchmark that can see the change. A tight loop of identical orders keeps everything warm, and hot/cold splitting will look useless in it. Interleave realistic other work between orders, as the real engine does: publish market data, append to the journal, maybe touch a few hundred KiB of unrelated data. Use a realistic order mix that includes a few percent of rejects.