Production Deployment

Light in optical fibre travels at roughly two-thirds of its vacuum speed, about 5 µs per kilometre. Through air, microwave and millimetre-wave links get close to the vacuum speed, about 3.3 µs per km. That gap is why firms built microwave networks between Chicago and the New Jersey data centres: the best fibre routes run around 13 ms round trip, the best wireless routes around 8 ms. Within a building the numbers shrink but the logic doesn't: 100 m of extra cable costs about 500 ns, which is more than a well-tuned matching engine spends on an order.

Hence colocation: you rent rack space in the same data centre as the venue's matching engine and connect to it with a cross-connect. Major venues each sit in one primary data centre (NYSE in Mahwah, Nasdaq in Carteret and CME in Aurora, Illinois, for example), and some equalise the cable length to every colocated customer so that nobody can buy a shorter cross-connect. Inside the rack, latency-sensitive firms use cut-through switches that forward a frame in a few hundred nanoseconds, or layer-1 devices that fan a feed out to many ports in single-digit nanoseconds by not parsing the frame at all.

A Linux container is a process with namespaces and cgroups. Executing instructions costs the same inside one as outside. The latency problems come from the defaults that surround containers:

• Networking. Default bridge networking puts a veth pair, a bridge and usually NAT rules between your socket and the NIC. Each hop is more kernel work per packet and more jitter. --network host removes them, and kernel-bypass NICs (week 33) need direct device access anyway.
• CPU limits. A CPU limit is enforced by CFS bandwidth control. Once a container spends its quota for the period (100 ms by default), its threads are throttled until the next period. A container whose busy-polling thread already uses a full core exhausts the quota early as soon as anything else in it runs (a logger, a GC-like cleanup, a second poller), and then every thread stops for milliseconds. Pin with cpusets (--cpuset-cpus, or exclusive CPUs from Kubernetes' static CPU manager) instead of capping with quotas.
• Neighbours. Containers on a shared host share L3 cache and memory bandwidth. A batch job three cores away can evict your order book. Cache partitioning (Intel RDT/CAT) exists for this, but the usual answer is simply not to share the host.
• Orchestration. A scheduler that reschedules pods, liveness probes that restart a busy engine mid-session, sidecars on your cores and overlay networks are all features in a web stack and incidents on a trading host.

Kubernetes can give a Guaranteed-QoS pod exclusive cores (the static CPU manager policy) and NUMA-aligned resources (the topology manager), and some firms do run latency-tolerant trading components that way. The common split is still: hot path on bare metal, managed by configuration management with pinned threads, and everything else (risk dashboards, reference data, backtests, research) on whatever platform the firm's infrastructure team prefers. Containers on the hot path, when used, are a packaging format: host networking, cpusets, no CPU limits, one per host.

Low-latency host tuning is a checklist, and every item exists because someone measured a stall:

• CPU governor performance. On-demand frequency scaling adds ramp-up time after idle. Some firms also fix the turbo state so latency doesn't depend on thermal headroom.
• Shallow C-states only (C1 or polling). Waking from a deep C-state takes tens to hundreds of µs (read /sys/devices/system/cpu/cpu*/cpuidle/state*/latency on your machine). Set via intel_idle.max_cstate=, processor.max_cstate= or a PM QoS request on /dev/cpu_dma_latency.
• isolcpus (or cpusets). Keeps the scheduler from placing other tasks on the cores your pinned threads spin on.
• nohz_full + rcu_nocbs. Stops the periodic scheduler tick (250–1000 Hz on most kernels) on a core running a single task, and moves RCU callbacks to housekeeping cores.
• NIC interrupts off hot cores, irqbalance stopped. An interrupt on the engine's core costs microseconds and pollutes its caches. Steer IRQs explicitly via /proc/irq/*/smp_affinity.
• Hot threads on the NIC's NUMA node. The NIC hangs off one socket's PCIe lanes (see numa_node under the interface's sysfs device directory). Remote DRAM is typically ~1.5–2× local latency, and DMA crosses the socket interconnect.
• THP never or madvise. always lets the kernel compact and collapse pages behind your back, with stalls of up to milliseconds. Use explicit huge pages for hot buffers.
• Swap off, mlockall. A page fault on the hot path costs microseconds. Faulting from disk costs milliseconds.
• Clocksource tsc. A vDSO clock_gettime on the TSC costs ~20 ns. Falling back to HPET makes every timestamp a slow hardware read, roughly a microsecond.
• SMT sibling idle, or SMT off. Hyperthreads share L1/L2 and execution ports with your hot thread.

None of this is exotic. Red Hat ships most of it as the latency-performance and network-latency tuned profiles. What matters is that you verify it on every host at every boot: a BIOS update that silently re-enables C6 shows up as a p99 regression weeks later.

Encode the checklist as code. A checklist in a wiki rots, but an audit that runs at boot and refuses to start the engine on a misconfigured host doesn't.

#[derive(Debug, Clone)]
pub struct Host {
    pub governor: &'static str,    // cpufreq governor: "performance", "powersave", "schedutil", ...
    pub isolcpus: Vec<u32>,        // CPUs isolated from the scheduler (kernel cmdline)
    pub nohz_full: Vec<u32>,       // CPUs running tickless
    pub thp: &'static str,         // transparent_hugepage/enabled: "always" | "madvise" | "never"
    pub swap_enabled: bool,
    pub clocksource: &'static str, // current_clocksource: "tsc", "hpet", "acpi_pm", ...
    pub max_cstate: u32,           // deepest idle state allowed (0 = poll, 1 = C1, 6 = C6, ...)
    pub irqbalance: bool,          // irqbalance daemon running
    pub cpu_node: Vec<u32>,        // cpu_node[cpu] = NUMA node of that CPU
    pub nic_node: u32,             // NUMA node the trading NIC is attached to
    pub nic_irq_cpus: Vec<u32>,    // CPUs the NIC's queue interrupts are steered to
}

pub struct Plan {
    pub hot_cpus: Vec<u32>,        // CPUs where pinned hot-path threads will spin
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum Finding {
    Governor,          // governor != "performance"
    Thp,               // thp == "always"
    Swap,              // swap enabled
    Clocksource,       // clocksource != "tsc"
    DeepCStates,       // max_cstate > 1
    IrqBalance,        // irqbalance running
    NotIsolated(u32),  // hot CPU missing from isolcpus
    NotTickless(u32),  // hot CPU missing from nohz_full
    IrqOnHotCpu(u32),  // a NIC interrupt is steered to a hot CPU
    RemoteNuma(u32),   // hot CPU on a different NUMA node than the NIC
}

/// Every finding for this host and plan. Order doesn't matter, but each
/// finding appears exactly once (the plan may list a CPU twice).
pub fn audit(host: &Host, plan: &Plan) -> Vec<Finding> {
    let _ = (host, plan);
    todo!()
}

On 1 August 2012, Knight Capital deployed new order-routing code to eight servers. One server didn't receive it. A configuration flag that the new code reused still triggered long-dead logic on that server, which then sent orders it never tracked as filled. In about 45 minutes the firm made more than 4 million executions and lost roughly $460 million. The SEC's order describes a deployment with no second-person review, no automated check that every server ran the same code, and no fast way to stop. Every one of those has a direct engineering counterpart: release verification, never repurposing flags, and a kill switch (week 15).

How exchanges release:

• Outside trading hours for the matching engine. Many venues change engine code only on weekends, after members have certified against a test environment for weeks.
• Primary/standby swaps for components that must stay up: upgrade the standby, let it replay and match the primary's state hash (week 20), fail over at a quiet moment, then upgrade the old primary. This requires the new version to read the old journal format, which is why week 35 cared about schema evolution.
• Session-aware gateway rollouts: clients keep a primary and a backup gateway, so a drained gateway's sessions reconnect to the backup with sequence numbers intact (week 14). Never drain both halves of a pair at once, and make sure the remaining fleet can absorb the sessions.
• Canary first: one host with the smallest blast radius, watched for long enough to see a full cycle of traffic, before anything else moves.
• Verify what shipped: every host reports its build id, and the rollout fails if any host disagrees.

Now plan a gateway rollout under those constraints. Waves drain in parallel, pairs never drain together, the fleet must absorb every drained session, and the canary goes first.

#[derive(Debug, Clone)]
pub struct Host {
    pub id: u32,
    pub sessions: u32, // client sessions currently on this gateway
    pub capacity: u32, // max sessions it can hold
}

pub struct Fleet {
    pub hosts: Vec<Host>,
    /// (a, b): clients use a and b as primary/backup — never drain both at once.
    pub pairs: Vec<(u32, u32)>,
    pub max_per_wave: usize,
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum PlanError {
    NoHosts,
    CannotDrain(u32),
}

/// Return the waves, in order, as lists of host ids.
///  1. Wave 0 is a single canary: the host with the fewest sessions
///     (ties → lowest id).
///  2. Every wave has 1..=max_per_wave hosts; every host is in exactly one wave.
///  3. No wave contains both hosts of a pair.
///  4. While a wave drains, its total sessions must fit in the spare capacity
///     (capacity − sessions) of all hosts NOT in that wave.
///  5. If a host cannot drain even alone, return Err(CannotDrain(id)) for the
///     lowest such id. Empty fleet → Err(NoHosts).
/// Fewer waves is better; any plan satisfying 1–4 is accepted.
pub fn plan_rollout(fleet: &Fleet) -> Result<Vec<Vec<u32>>, PlanError> {
    let _ = fleet;
    todo!()
}

The demo is your exchange in one process: the engine, simulated order flow (sim::LiveSession, the same provided flow that the WebAssembly page and the native terminal run) and the WebSocket market-data feed (week 16). It runs on a small virtual machine anyone can connect to. It isn't a latency showcase. A shared VM has noisy neighbours, steal time and no kernel tuning. It's proof that the system runs, recovers and behaves, and it's something to watch.

Packaging choices that matter:

• Build once, promote the artifact. A multi-stage container build with your week 22 release profile produces the image, and the same image digest goes to every environment. If you build fully static against musl, swap in a faster allocator: musl's default malloc is much slower than glibc's under allocation-heavy load.
• Configuration from the environment or a mounted file, validated at startup by week 22's rules. Bad config should stop the process with a readable error, not fall back to a silent default. The repo's demo has its own small demo::Config (it doesn't use the gateway's exchange.toml): defaults, then HFT_DEMO_* variables, then flags, parsed once at startup. Bad input exits with status 2 and a message naming the setting and where it came from.
• Graceful shutdown on SIGTERM: stop accepting connections, flush the journal, close WebSocket sessions, exit inside a deadline (week 3). Container platforms send SIGTERM and then SIGKILL after a grace period, and your drain has to fit inside it. The repo's demo installs no signal handler (std has none), so SIGTERM ends it at once. That's a clean stop only because the demo keeps no journal. Run it under an init (docker run --init, or the platform's own) so the signal reaches it at all. Its graceful paths are --duration, a token-guarded POST /shutdown and Demo::shutdown.
• Expose only what's safe. The public endpoint is read-only market data. Cap connections, limit per-IP subscription rates (week 17), give every client a bounded buffer with conflation (week 16), and terminate TLS in a reverse proxy so browsers can use wss://. Keep order entry and metrics off the public interface. The repo's demo has no order entry at all, refuses WebSocket clients beyond --max-ws-clients with a 503, and leaves TLS to the platform's proxy. It has no per-IP limits, and it serves /metrics on the same port as the page; fly.toml notes where to move it in a real deployment.

Exchanges run on a schedule: start-of-day checks, a pre-open phase that accepts orders without matching, an opening auction, continuous trading, a closing auction, end-of-day processing, and often a nightly restart that resets sessions and sequence numbers. Operations means every phase transition is scripted, observed and reversible. Your demo should have the same shape in miniature: a start-of-day check (config valid, journal readable, host audit clean), a published session schedule, and an end-of-day snapshot.