Configuration, Builds & Deployment

Seven rules, in rough order of how much pain they prevent:

1. Parse into types once, at startup. After main has a ValidConfig, no code path reads raw strings or re-checks ranges. Week 8 applied this to orders. Configuration is the same idea: make invalid states unrepresentable, so the engine never sees a tick size of zero.
2. Unknown keys are errors. A misspelled key that silently falls back to a default is the worst failure mode there is, because everything looks fine. With serde that's one attribute, #[serde(deny_unknown_fields)].
3. Report every error at once. An operator fixing a config at 7 a.m. should get the full list, not one error per restart.
4. Cross-field rules are validation too. The engine core is not a gateway core; the maximum order size is a whole number of lots; the lease outlives at least three heartbeats. Check a rule only when the fields it reads parsed cleanly, so one typo produces one error, not a cascade.
5. Make defaults visible. Log the effective configuration at startup, with defaults filled in and secrets redacted, plus its hash.
6. Configuration that affects matching is engine state. Tick size, lot size, price bands and the instrument list change what the engine does with an order. Journal them (or their hash, plus the config itself in the snapshot) so that replaying Tuesday's journal on Thursday uses Tuesday's tick size, not whatever the file says now.
7. Change things at runtime through the sequenced stream. Halting an instrument or widening a band is an admin command, sequenced, journaled and replicated like any order. It is never a side thread re-reading a file, which would bypass the journal, the standby and determinism all at once.

With serde, the shape is two stages: deserialize, then validate into a type with private fields.

#[derive(Debug, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RawConfig {
    pub listen_port: u16,
    pub engine_core: usize,
    pub gateway_cores: Vec<usize>,
    pub tick_size: u64,
    pub lot_size: u64,
    #[serde(default = "defaults::price_band_bps")]
    pub price_band_bps: u32,
    // …
}

/// Only constructible through validation.
pub struct ValidConfig { inner: RawConfig }

impl TryFrom<RawConfig> for ValidConfig {
    type Error = Vec<ConfigError>;
    fn try_from(raw: RawConfig) -> Result<Self, Self::Error> {
        let mut errs = Vec::new();
        if raw.gateway_cores.contains(&raw.engine_core) {
            errs.push(ConfigError::Rule(RULE_ENGINE_CORE));
        }
        // … every rule, collecting rather than returning early
        if errs.is_empty() { Ok(ValidConfig { inner: raw }) } else { Err(errs) }
    }
}

There's one limitation to know: serde's derive stops at the first deserialization error (a bad type or a missing field). To report every problem, including several bad values at once, you need the more permissive first stage that today's drill builds by hand: read raw strings per key, then parse and validate each field independently.

The drill is the two-stage validator from Day 1, written without serde so you control every error. The spec is in the comment at the top. The design decisions worth noticing:

• Line numbers for syntax errors, key names for everything else. That is what the operator needs to find the problem.
• Duplicate keys are errors, not "last one wins". Two values for one key in a reviewed file means someone merged badly.
• No cascades. If lot_size = abc, the rule "max_order_qty is a multiple of lot_size" can't be evaluated, and reporting it would send the operator after a second problem that doesn't exist.

// Format: one `key = value` per line. Blank lines and lines starting with
// '#' are ignored; whitespace around keys and values is trimmed.
//
//   key             value                              required  default
//   listen_port     u16 in 1..=65535                   yes
//   engine_core     usize                              yes
//   gateway_cores   comma-separated usizes, non-empty  yes
//   tick_size       u64 > 0                            yes
//   lot_size        u64 > 0                            yes
//   max_order_qty   u64 > 0                            yes
//   price_band_bps  u32 in 1..=10000                   no        500
//   fsync           always | never | interval:<ms>     no        always
//                   (ms is a u64 > 0)
//   heartbeat_ms    u64 > 0                            no        100
//   lease_ms        u64 > 0                            no        1000
//
// Report EVERY problem, not just the first:
//   line with no '='                  → Syntax { line }  (1-based)
//   key not in the table              → UnknownKey(key)
//   key given twice                   → DuplicateKey(key) (first one wins)
//   value that fails to parse/range   → BadValue { key, value } (trimmed)
//   required key never given          → Missing(key)
//   cross-field rule broken           → Rule(RULE_…)
// Check a cross-field rule only if every field it reads parsed cleanly:
// one typo should produce one error, not a cascade.

use std::collections::HashMap;

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Fsync {
    Always,
    Never,
    IntervalMs(u64),
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ExchangeConfig {
    pub listen_port: u16,
    pub engine_core: usize,
    pub gateway_cores: Vec<usize>,
    pub tick_size: u64,
    pub lot_size: u64,
    pub max_order_qty: u64,
    pub price_band_bps: u32,
    pub fsync: Fsync,
    pub heartbeat_ms: u64,
    pub lease_ms: u64,
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ConfigError {
    Syntax { line: usize },
    UnknownKey(String),
    DuplicateKey(String),
    BadValue { key: String, value: String },
    Missing(&'static str),
    Rule(&'static str),
}

pub const RULE_ENGINE_CORE: &str = "engine_core must not be one of gateway_cores";
pub const RULE_LOT: &str = "max_order_qty must be a multiple of lot_size";
pub const RULE_LEASE: &str = "lease_ms must be at least 3 x heartbeat_ms";

pub const KEYS: [&str; 10] = [
    "listen_port", "engine_core", "gateway_cores", "tick_size", "lot_size",
    "max_order_qty", "price_band_bps", "fsync", "heartbeat_ms", "lease_ms",
];

pub fn parse_config(text: &str) -> Result<ExchangeConfig, Vec<ConfigError>> {
    // Suggested shape:
    //   1. Walk the lines into a HashMap<&'static str, String> of raw values
    //      (KEYS gives you the &'static str), pushing line-level errors.
    //   2. Parse each field to Option<T>, pushing BadValue / Missing.
    //   3. Run the rules whose inputs are all Some.
    //   4. Ok(config) only if no errors were pushed.
    let _ = text;
    let _: HashMap<&'static str, String> = HashMap::new();
    todo!()
}

Wire the real version into a --check-config flag that validates and exits with a non-zero status on any error. CI runs it on every config change, and the deploy pipeline runs it against the exact file it is about to ship. The cheapest incident is the one a pull request check prevents.

Cargo's release profile is a set of trade-offs between build time, binary size, debuggability and speed, and the defaults are tuned for none of them in particular. For a latency-critical binary, go through them one by one:

• opt-level = 3 is already the release default. Leave it.
• codegen-units = 1. By default a release build splits each crate into 16 units that LLVM optimizes in parallel. Inlining and other optimizations can't see across those boundaries. One unit gives LLVM the whole crate at once: slower builds, usually somewhat faster code.
• lto = "fat". Your engine calls into book, risk and wire, which are separate crates. Without LTO, only generic and #[inline] functions can be inlined across a crate boundary. Fat LTO optimizes the whole program as one module; "thin" gets much of the benefit with far cheaper builds. Beware the trap here: lto = false does not disable LTO. It means "thin LTO across this crate's own codegen units". To turn it off you write lto = "off".
• panic = "abort". No unwinding landing pads, slightly smaller and sometimes faster code, and a panic kills the process immediately. For the engine that is exactly right: a panic is a bug, the process should die, and the hot standby (week 20) takes over. The cost is that catch_unwind stops working. A gateway that wants to drop one misbehaving session and keep the others needs unwinding, and that is a legitimate reason to build it with a different profile (custom profiles can inherits = "release").
• debug. Debug info doesn't change the generated machine code in any way that matters for speed. It costs binary size and link time. Keep at least "line-tables-only" in any binary you'll profile in production (week 26), or build full debug info and strip it into a separate file you archive.
• overflow-checks is off in release by default. Rather than turning it on globally, use checked_* or saturating_* arithmetic where overflow is actually possible (notional = price × quantity), so the check sits exactly where it matters.

Two more settings live in RUSTFLAGS (put them in the repo's .cargo/config.toml so every build uses them):

• -C target-cpu=<cpu> lets LLVM use newer instructions (AVX2, BMI2, AVX-512 where available) and schedule for the right core. Set it to the oldest CPU in your fleet. Never use native: that means "the CPU of whichever machine ran the build", which makes the binary depend on the CI runner and can crash with an illegal instruction on an older production host.
• -C force-frame-pointers=yes makes stack walking for perf cheap and reliable at the cost of one register. The overhead is usually around a percent, small enough that Fedora and Ubuntu now build their distributions with frame pointers by default.

# Cargo.toml (workspace)
[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
panic = "abort"
debug = "line-tables-only"

[profile.release-gateway]   # same, but sessions can catch panics
inherits = "release"
panic = "unwind"

# .cargo/config.toml
[build]
rustflags = ["-Ctarget-cpu=icelake-server", "-Cforce-frame-pointers=yes"]

Then measure. Fat LTO plus one codegen unit is often worth a few percent to low double digits on code like an engine, and sometimes nothing. Profile-guided optimization can add more; that belongs in week 40's sprint. Run the week 7 and week 12 benchmarks before and after, and keep the change only if the numbers say so.

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Lto {
    Off,
    Thin,
    Fat,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Panic {
    Unwind,
    Abort,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum DebugInfo {
    None,
    LineTablesOnly,
    Full,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Profile {
    // [profile.*] keys
    pub opt_level: u8,
    pub lto: Lto,
    pub codegen_units: u32,
    pub panic: Panic,
    pub debug: DebugInfo,
    pub overflow_checks: bool,
    pub incremental: bool,
    // not profile keys: these become RUSTFLAGS
    pub target_cpu: Option<String>,
    pub frame_pointers: bool,
}

pub struct Requirements {
    /// A panic inside one session's handler must be caught with
    /// catch_unwind so only that session is dropped.
    pub isolate_panics: bool,
    /// We run perf and build flame graphs against production binaries.
    pub profile_in_production: bool,
    /// The OLDEST CPU model in the deploy fleet when every host is known
    /// (e.g. "icelake-server"); None for a mixed or unknown fleet.
    pub fleet_cpu: Option<String>,
}

/// Apply this week's policy for latency-critical binaries (Day 3).
pub fn choose_profile(req: &Requirements) -> Profile {
    let _ = req;
    todo!()
}

/// Render a Cargo profile table, keys in this order, one per line, each
/// line ending in '\n':
///   [profile.<name>]
///   inherits = "release"         ← only when name != "release"
///   opt-level = 3
///   lto = "off" | "thin" | "fat"
///   codegen-units = 1
///   panic = "unwind" | "abort"
///   debug = false | "line-tables-only" | true
///   overflow-checks = false
///   incremental = false
pub fn render_profile(name: &str, p: &Profile) -> String {
    let _ = (name, p);
    todo!()
}

/// Flags for RUSTFLAGS / .cargo/config.toml, in this order:
///   "-Ctarget-cpu=<cpu>"          when target_cpu is Some
///   "-Cforce-frame-pointers=yes"  when frame_pointers
pub fn rustflags(p: &Profile) -> Vec<String> {
    let _ = p;
    todo!()
}

A build is reproducible if the same source, toolchain and flags produce a bit-identical binary. For an exchange this isn't an academic nicety. It's how you know exactly what is running, how you rebuild that exact binary to debug a crash six months later, and how you show an auditor that the binary in production came from the commit that was reviewed.

The ingredients for a Rust workspace:

• Commit Cargo.lock and build with --locked, which fails instead of silently updating dependencies. Cargo verifies every downloaded crate against the checksums recorded in the lockfile.
• Pin the toolchain in rust-toolchain.toml (channel = "1.xx.y", a full version, not stable).
• Keep flags in the repo (.cargo/config.toml), not in someone's shell profile or a CI variable.
• Strip absolute paths with --remap-path-prefix. Build directories otherwise end up in panic messages and debug info, and differ between machines.
• Pin the build image by digest, not by tag, and set SOURCE_DATE_EPOCH for anything that embeds a timestamp.

Then embed what you need to identify a binary: the git commit, the toolchain version, the profile and target CPU. A build.rs can emit them with cargo:rustc-env=GIT_SHA=…, and the code reads them with env!("GIT_SHA"). Print them at startup, include them in the journal header, and export them as an info metric, exch_build_info{git_sha="…",rustc="…"} 1, so every dashboard can show which build produced which numbers.

Verify the whole thing the simple way: build twice in clean environments and compare sha256sum.

A container isn't a virtual machine. It's an ordinary process placed in Linux namespaces (its own view of mounts, PIDs, network) and cgroups (resource accounting and limits). CPU-bound code inside it runs exactly as fast as outside. The costs appear at the edges, and each has a standard fix:

• Networking. Default bridge networking puts a veth pair, a bridge and NAT rules between your socket and the NIC: more code per packet and more jitter. Run the gateway and the market-data publisher with host networking (--network host).
• CPU quotas. --cpus uses CFS bandwidth control: once your threads use their quota for the current period (100 ms by default), they are throttled until the next one. A busy-polling engine thread burns its quota fast, then stalls for milliseconds. Use cpusets (--cpuset-cpus) to give latency-critical threads dedicated cores instead of quotas.
• Memory limits. Approaching the cgroup limit triggers reclaim, and crossing it gets the process killed. Size limits generously, preallocate at startup, and if you lock memory (mlockall), grant the capability and raise memlock.
• Privileges. Pinning threads within your cpuset is fine unprivileged; real-time scheduling (SCHED_FIFO) needs CAP_SYS_NICE; huge pages and IRQ affinity are host configuration. Prefer granting specific capabilities over --privileged.

A multi-stage image keeps the toolchain out of production:

# syntax=docker/dockerfile:1
FROM rust:1.xx-bookworm@sha256:<digest> AS build
WORKDIR /src
COPY . .
RUN cargo build --profile release --locked -p gateway --bin exchange

FROM gcr.io/distroless/cc-debian12@sha256:<digest>
COPY --from=build /src/target/release/exchange /usr/local/bin/exchange
COPY exchange.toml /etc/hft/exchange.toml
ENTRYPOINT ["/usr/local/bin/exchange", "--config", "/etc/hft/exchange.toml"]

docker run --network host --cpuset-cpus 2-7 --ulimit memlock=-1 exchange:<git-sha>

One more trap: a static musl build is tempting for tiny images, but musl's default allocator is markedly slower than glibc's under multithreaded allocation. Stay on glibc (the cc distroless base), or bring your own allocator and measure.

Build the capstone below, then spend twenty minutes as an adversary: write the five worst config files you can imagine (typos, swapped units, a reference price off the tick grid, an account throttle burst smaller than a session's, a bad HFT_* override) and confirm --check-config catches each one with a message an operator can act on.