Performance Sprint & Report

Begin with three inputs, not ideas. First, last week's budget table: which stages are over budget, at which percentile. Second, fresh profiles of the current build under realistic load (week 26): on-CPU flame graphs plus the counters that explain them (IPC, cache misses, branch misses). Third, the targets: the curriculum's numbers below, plus any of your own.

Then do the arithmetic before the work. Amdahl's law: if a part takes fraction f of the time and you make it s times faster, the whole speeds up by 1 / ((1 − f) + f/s). The ceiling, if that part became free, is 1 / (1 − f). A stage that takes 35% of the path can never give more than a 1.54× overall speedup, however clever the change.

Be careful about which time f is a fraction of:

• A CPU flame graph is weighted by samples, so it shows where time goes on average. That's the right f for throughput.
• For p99 latency, f is the stage's share of the slow orders' time, which is what your week 39 attribution measures. Rare stalls such as page faults, interrupts and queue waits can dominate the tail while barely showing up in a sampled profile, and off-CPU waits don't show up in an on-CPU profile at all.
• In a pipeline where stages run concurrently on different cores, throughput is set by the slowest stage and latency is the sum. Speeding up a stage that isn't the bottleneck reduces latency but leaves throughput exactly where it was.

Encode the arithmetic you'll use all week. Two questions come up constantly. "How much faster must this part get for the whole to hit the target?" Sometimes the honest answer is no finite amount. And "if I speed up this stage, does throughput move at all?"

#[derive(Clone, Copy, Debug)]
pub struct Part {
    /// Fraction of the ORIGINAL time spent in this part (0.0..=1.0).
    pub fraction: f64,
    /// How much faster this part became (2.0 = twice as fast).
    pub speedup: f64,
}

/// Amdahl's law: overall = 1 / ((1 − Σ fraction) + Σ fraction / speedup).
/// Time not covered by any part is unchanged.
pub fn amdahl(parts: &[Part]) -> f64 {
    todo!()
}

/// The ceiling: overall speedup if a part with `fraction` became
/// infinitely fast. fraction == 1.0 → f64::INFINITY.
pub fn max_speedup(fraction: f64) -> f64 {
    todo!()
}

/// How much faster must the part with `fraction` become for the whole to
/// reach `target` overall? None if no finite speedup gets there —
/// including exactly at the ceiling (compare with a small epsilon).
pub fn required_speedup(fraction: f64, target: f64) -> Option<f64> {
    todo!()
}

/// Ops/s of a pipeline whose stages run concurrently, one per core,
/// each taking `stage_ns` per item: the slowest stage sets the rate.
pub fn pipeline_throughput(stage_ns: &[f64]) -> f64 {
    todo!()
}

/// Latency of one item through the same pipeline (no queueing): the sum.
pub fn pipeline_latency(stage_ns: &[f64]) -> f64 {
    todo!()
}

Look closely at one case in the drill: a stage that is 35% of the path needs a 21× speedup for the whole to reach 1.5×. When the required speedup comes out like that, the honest conclusion is that this stage won't get you there and you need to change the architecture or the target.

Each experiment gets an entry before you write any code:

# example entry: the numbers are illustrative
E-07  2026-10-12  commit a1b2c3d (base) → e4f5g6h
hypothesis   order-id → slot lookup via slab index instead of HashMap
             removes ~40 ns per cancel at p99 (profile: 18% in hashing)
change       book::index uses OrderId as slab key; ids are dense per session
command      cargo xtask bench --week 27   (20 runs per variant, ABAB)
environment  EPYC 7543P, isolcpus=2-7, governor=performance, boost off
result       w27.add_cancel_slab p99: 212 → 168 ns (median of 20 runs each)
             w40.engine_throughput: 5.1M → 5.4M ops/s
verdict      KEEP
notes        p99.9 unchanged — tail is elsewhere (see E-05 attribution)

Change one thing at a time. Revert the losers, and record that you tried them. A sprint log where every entry says KEEP is a log someone edited afterwards.

Reproducibility checklist. Most benchmark disagreements come from the setup, not the code:

• Build. Same release profile every time (lto, codegen-units = 1, panic = "abort" from week 22). If you use -C target-cpu=native, say so: the binary won't run on older CPUs, and the comparison is only valid on the same microarchitecture.
• Machine state. Pinned threads on isolated cores (week 30), a fixed frequency governor, and turbo either disabled or noted. With turbo on, a short benchmark runs faster than a long one and results drift with temperature and with load on other cores. Nothing else running. Laptops plugged in.
• Runs, not samples. Samples within one run are correlated: same heap layout, same code alignment, same thermal state. The unit of evidence is the run. Take 10–30 runs per variant and interleave them A, B, A, B so slow drift hits both equally.
• Fixed seeds and warmup. The same order stream every time (types::Rng with a fixed seed), and discard the warmup.
• Layout luck. Moving unrelated code can shift a hot loop's alignment and swing results by several percent. If a 3% "win" disappears when you add a function somewhere else, it was never a win.

Without a regression gate, a sprint's gains wear away within months. The gate has to answer a hard question on noisy data: did this change make the benchmark worse, or is this just run-to-run variation? Two separate tests have to pass before you call something a regression:

1. Statistical: the candidate's runs really are shifted relative to the baseline's. Benchmark distributions are skewed and often bimodal, so use a test that doesn't assume normality. Mann–Whitney U counts how often a candidate run beats a baseline run. Under "no change" that's about half the pairs, and the normal approximation turns the count into a z-score.
2. Practical: the shift is big enough to matter, say ≥2% on the median. With very quiet runs, a 0.3% change can be statistically certain and still not worth anyone's attention.

Criterion.rs uses the same two ideas. It has a configurable significance level (default 0.05) and a noise threshold (default 1%), and reports a change only when both are crossed.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Verdict {
    Regression,
    Improvement,
    NoChange,
}

/// Median of an UNSORTED slice (mean of the two middle values when the
/// length is even). Empty → 0.0.
pub fn median(xs: &[f64]) -> f64 {
    todo!()
}

/// Mann–Whitney U for `a` versus `b`: the number of pairs (a_i, b_j) with
/// a_i > b_j, counting ties as 0.5. Returns (U, z) where z is the normal
/// approximation (no tie correction):
///     z = (U − n_a·n_b / 2) / sqrt(n_a·n_b·(n_a + n_b + 1) / 12)
/// z > 0 means `a` tends to be LARGER than `b`.
pub fn mann_whitney(a: &[f64], b: &[f64]) -> (f64, f64) {
    todo!()
}

/// Compare per-run results of one benchmark (e.g. 30 runs each).
/// `lower_is_better`: true for latency, false for throughput.
/// Regression  ⇔ the candidate is worse on BOTH gates:
///     statistically: |z| ≥ z_crit in the bad direction, and
///     practically:   the median moved the bad way by ≥ min_effect
///                    (relative to the baseline median; 0.02 = 2%).
/// Improvement ⇔ the same two gates in the good direction.
/// Otherwise NoChange.
pub fn verdict(
    baseline: &[f64],
    candidate: &[f64],
    lower_is_better: bool,
    min_effect: f64,
    z_crit: f64,
) -> Verdict {
    todo!()
}

Look at the "very noisy" case in the drill. The medians differ by about 5%, yet the test says no change, because with that much spread a 5% gap in medians happens by chance all the time. If you compare two single runs, you'll "find" regressions like this every week.

Write the report so that a sceptical colleague could reproduce every number in it. A structure that works:

1. Summary. One table: metric (with percentile and load), before, after, target, met or missed. Then two sentences on what made the difference.
2. Environment. CPU model and microcode, core count and SMT state, kernel and boot parameters (isolation, nohz_full), governor and turbo state, compiler version and profile, commit hashes.
3. Methodology. Open- or closed-loop and at what offered rate, warmup, run length, number of runs, how runs were interleaved, the statistical gate.
4. Results. Distributions, not single numbers: percentile plots (p50 → p99.99) or histograms for before and after, with the max and sample counts.
5. What changed. The KEEP experiments in order, each with its measured contribution. If contributions overlap, say so rather than adding them up.
6. What didn't work. The REVERT experiments and what you learned from them. Reviewers trust this section most.
7. Threats to validity. What could make these numbers wrong: a single machine, synthetic order flow, a benchmark shape that flatters the ladder book, turbo.
8. How to reproduce. Exact commands.

Make every comparison like for like: the same machine, load, percentile and build settings. Never put your p50 next to someone else's p99, or a closed-loop number next to an open-loop one. Don't quote your best run. Give the median run, the spread, and N.