Observability Without Overhead

Start with the instructions, because they decide everything else:

• A plain increment of a thread-owned u64: about one cycle, and the compiler may keep it in a register.
• A Relaxed atomic load and store: on x86-64 and AArch64 these compile to ordinary loads and stores. The same cost as the plain increment, but legal to read from another thread. (Reading a plain u64 that another thread is writing is a data race, which is undefined behaviour in Rust, even if the assembly would look identical.)
• fetch_add, which is lock xadd on x86: around 20 cycles of latency even when uncontended on recent Intel and AMD cores (see Agner Fog's instruction tables for your microarchitecture), and it acts as a full barrier for the surrounding memory operations.
• fetch_add on a line other cores also write: the line has to move between cores for every increment (the false-sharing mechanics from week 4, except here the sharing is real). Expect tens of nanoseconds per increment within a socket and more across sockets.

That makes the design obvious. Give every thread its own block of counters on its own cache lines. Because each counter has exactly one writer, it doesn't need a read-modify-write at all:

use std::sync::atomic::{AtomicU64, Ordering::Relaxed};

/// One per thread, on its own cache lines. Exactly one thread writes it;
/// a background thread reads it.
#[repr(align(64))]
pub struct ThreadCounters {
    pub orders_in: AtomicU64,
    pub rejects: AtomicU64,
    pub acks_out: AtomicU64,
}

#[inline(always)]
pub fn bump(c: &AtomicU64) {
    // Single writer: no RMW needed. On x86 this is a plain load and a plain
    // store: no lock prefix, no barrier.
    c.store(c.load(Relaxed) + 1, Relaxed);
}

The reader sees each counter at some recent value, never a torn one (aligned 64-bit atomics are single-copy atomic), and never anything newer than the truth. It may see orders_in from one moment and rejects from a slightly different one. For monitoring that is fine, and it's the price of zero coordination. If two numbers must be consistent with each other, derive one from the other when you read them instead of synchronizing the writer.

Counters scale by giving each thread its own. Histograms work the same way. Each gateway thread, the engine thread and the publisher own a histogram per thing they measure. The buckets are AtomicU64s written with the single-writer load-plus-store, so record is a bucket-index computation (a leading_zeros and two shifts for the log-linear layout from week 2) plus one load and one store. A background merger thread wakes every second or so, reads every thread's histograms and combines them.

Three rules make this correct.

Merge buckets, never percentiles. Percentiles don't compose. If one thread out of four has a tail, averaging the four p99s dilutes the tail by a factor of four and reports a number no order ever experienced. Adding the bucket counts and computing the percentile of the sum is exact, up to bucket resolution. This is also why Prometheus histograms can be aggregated across instances and Prometheus summaries (client-side quantiles) cannot.

The hot thread never resets. Clearing a histogram from the merger would make it a second writer. So per-thread histograms are cumulative forever, and the merger derives "what happened in the last interval" as the bucket-wise difference between this read and the previous one.

Detect resets. If a thread restarts, or a histogram is recreated, the new cumulative counts are smaller than the previous read. A naive subtraction underflows into nonsense. Treat any bucket going backwards as a reset, exactly as Prometheus does for counters, and skip that interval for that source.

One honest caveat: the merger's read of a histogram is not an atomic snapshot. It may see a bucket incremented while a total it reads separately is not, or the other way round. Avoid storing a separate total and compute it from the buckets you read, so the view is at least self-consistent.

// Log-linear buckets (given). Values 0..8 get exact buckets; above that
// every power of two is split into 8 sub-buckets, so a bucket is never
// wider than 1/8 of its lower bound (≤ 12.5% relative error).
pub const BUCKETS: usize = 8 + 61 * 8; // 496 buckets cover all of u64

pub fn bucket_of(v: u64) -> usize {
    if v < 8 {
        return v as usize;
    }
    let e = 63 - v.leading_zeros(); // floor(log2 v), >= 3
    let sub = ((v >> (e - 3)) & 7) as usize;
    8 + (e as usize - 3) * 8 + sub
}

/// Largest value that lands in bucket `i`.
pub fn bucket_upper(i: usize) -> u64 {
    if i < 8 {
        return i as u64;
    }
    let e = ((i - 8) / 8 + 3) as u32;
    let sub = ((i - 8) % 8) as u64;
    let lower = (8 + sub) << (e - 3);
    lower + ((1u64 << (e - 3)) - 1)
}

/// A cumulative histogram owned by ONE thread. A background merger reads
/// every thread's histogram and combines them.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Hist {
    pub counts: Vec<u64>,
    pub total: u64,
    pub max: u64,
}

impl Hist {
    pub fn new() -> Self {
        Hist { counts: vec![0; BUCKETS], total: 0, max: 0 }
    }

    pub fn record(&mut self, v: u64) {
        // TODO: bump the bucket, the total and the max.
        let _ = v;
        todo!()
    }

    /// Fold `other` into `self` (the merger does this once per thread).
    pub fn merge_from(&mut self, other: &Hist) {
        // TODO: add bucket by bucket; total adds; max is the max of maxes.
        let _ = other;
        todo!()
    }

    /// What happened between an earlier snapshot of this same cumulative
    /// histogram and now. Returns None if any bucket (or the total) went
    /// DOWN: the source was reset or restarted, so the interval is unknown.
    /// The interval's true max is not recoverable from two snapshots; use
    /// the upper bound of the highest non-empty bucket (0 if empty).
    pub fn delta_since(&self, earlier: &Hist) -> Option<Hist> {
        let _ = earlier;
        todo!()
    }

    /// Nearest-rank quantile: rank = ceil(q × total), clamped to 1..=total.
    /// Walk the buckets until the running count reaches rank and return
    /// min(bucket_upper(i), self.max). Empty histogram → 0.
    pub fn percentile(&self, q: f64) -> u64 {
        let _ = q;
        todo!()
    }
}

Telemetry leaves a hot thread through three channels, from cheapest to most expensive:

1. Aggregates: the counters and histograms above. Written in place, read by the merger. Cost on the hot path: a few nanoseconds.
2. Event records: fixed-size binary structs pushed into a per-thread SPSC ring (week 1, week 5) that a telemetry thread drains. Use them for things an aggregate can't answer, such as which session sent the order that took 80 µs, or the stage timestamps of every order in a slow second. Cost: one ring push, tens of nanoseconds.
3. Logs: formatted text, produced only by the telemetry thread or by cold code (session logon, configuration, recovery). Never on the hot path.

The ring has one hard rule: the hot path never waits for telemetry. If the ring is full, the record is dropped and a telemetry_dropped_total counter is bumped. A trading system that slows down because its monitoring is slow has inverted its priorities, and a monitoring outage must not become a trading outage.

#[derive(Clone, Copy)]
#[repr(C)]
pub struct OrderRecord {
    pub session: u32,
    pub kind: u8,          // accepted / rejected / filled …
    pub reason: u8,        // reject reason code, 0 if none
    pub _pad: [u8; 2],
    pub order_id: u64,
    pub stamps: Stamps,    // see below
}

#[inline]
pub fn emit(ring: &Producer<OrderRecord>, dropped: &AtomicU64, rec: OrderRecord) {
    if ring.try_push(rec).is_err() {
        dropped.store(dropped.load(Relaxed) + 1, Relaxed); // single writer
    }
}

Stage timestamps are the most valuable records you will emit. Carry them in the order's message as it moves through the pipeline, so each stage adds one stamp and nobody has to correlate across threads afterwards:

#[derive(Clone, Copy, Default)]
#[repr(C)]
pub struct Stamps {
    pub gw_in: u64,     // bytes read off the socket
    pub risk_out: u64,  // decoded and risk-checked, handed to the engine queue
    pub eng_in: u64,    // dequeued by the engine
    pub match_out: u64, // matching done
    pub ack_out: u64,   // ack written to the socket
}

The differences between consecutive stamps are the per-stage latencies; their distributions are what week 24 attributes the tail with. For now the stamps can come from Instant; week 21 makes reading the clock cheap enough to do five times per order.

One boundary to keep sharp: telemetry timestamps never feed back into the engine's decisions. They ride alongside the command and are not part of the journaled input or the state hash. The moment a wall-clock reading influences matching, week 9's replay determinism is gone.

Prometheus pulls metrics: every scrape interval (typically 10–60 seconds) it does an HTTP GET on /metrics and parses a plain-text format. Producing that text is cold-path work for your merger thread. The format is simple and strict:

# HELP exch_orders_total Orders accepted by the gateway.
# TYPE exch_orders_total counter
exch_orders_total{session="A",side="buy"} 1200
exch_orders_total{session="B",side="sell"} 7
# HELP exch_order_latency_seconds Gateway-in to ack-out latency.
# TYPE exch_order_latency_seconds histogram
exch_order_latency_seconds_bucket{stage="match",le="0.00001"} 5
exch_order_latency_seconds_bucket{stage="match",le="0.0001"} 12
exch_order_latency_seconds_bucket{stage="match",le="+Inf"} 14
exch_order_latency_seconds_sum{stage="match"} 0.0123
exch_order_latency_seconds_count{stage="match"} 14

The conventions matter because dashboards and alerting rules assume them. Counters end in _total. Values use base units: seconds and bytes, not nanoseconds and kilobytes. Your hot path records integer nanoseconds and the exporter divides. Histogram buckets are cumulative (each le bucket counts every sample at or below its bound) and end with +Inf, which equals _count.

Choose export buckets deliberately. Your internal histogram has hundreds of buckets, and every exported bucket is a separate time series in the database. Export a dozen or so bounds that match your SLOs (say 1, 2, 5, 10, 20, 50, 100, 200, 500 µs, 1 ms) by summing internal buckets, and keep the full-resolution histogram for your own reports. Recent Prometheus versions also support "native histograms" with exponential buckets, closer to what you built; check what your monitoring stack runs before relying on them.

Cardinality is the classic way to take down a monitoring system. Each distinct label combination is a separate series. stage, side, reason and even session (hundreds) are fine. order_id or client_order_id creates a new series per order. That's millions of series by lunchtime, and both your exporter and Prometheus fall over.

// Prometheus text exposition format (0.0.4), the subset an exchange needs:
//
//   # HELP <name> <help>        help escapes \ as \\ and newline as \n
//   # TYPE <name> counter | gauge | histogram
//   <name>{<k>="<v>",...} <value>
//
// Label values escape \ as \\, " as \" and newline as \n. A series with no
// labels has no braces at all. Every line, including the last, ends in \n.
//
// A histogram series is cumulative buckets, then _sum and _count:
//   <name>_bucket{<labels>,le="<bound>"} <count of samples ≤ bound>
//   ...one line per bound, then le="+Inf" whose count is the total
//   <name>_sum{<labels>} <sum>
//   <name>_count{<labels>} <total>
// Format f64 values with Rust's `{}` (0.0005 → "0.0005", 1.0 → "1").

pub fn escape_label_value(v: &str) -> String {
    let _ = v;
    todo!()
}

pub fn escape_help(v: &str) -> String {
    let _ = v;
    todo!()
}

/// `{a="1",b="2"}` in the given order, or "" when there are no labels.
pub fn format_labels(labels: &[(&str, &str)]) -> String {
    let _ = labels;
    todo!()
}

/// One counter family: HELP and TYPE once, then one line per series.
pub fn render_counter(name: &str, help: &str, series: &[(Vec<(&str, &str)>, u64)]) -> String {
    let _ = (name, help, series);
    todo!()
}

/// One histogram family with a single series.
/// `bounds` are ascending upper bounds (seconds). `counts` is NOT
/// cumulative and has bounds.len() + 1 entries; the last is the overflow
/// bucket (samples above the largest bound).
pub fn render_histogram(
    name: &str,
    help: &str,
    labels: &[(&str, &str)],
    bounds: &[f64],
    counts: &[u64],
    sum: f64,
) -> String {
    let _ = (name, help, labels, bounds, counts, sum);
    todo!()
}

You still need logs: for session logon and logout, configuration at startup, recovery and failover, reject summaries, and anything an operator has to read during an incident. Use the tracing crate and make them structured (session = %sid, reason = ?r), so they can be filtered and correlated rather than grepped. Know what each part costs:

• A disabled callsite is nearly free. tracing caches each callsite's "interest". A debug! that is filtered out costs roughly a load and a branch. The crate's max_level_* and release_max_level_* features remove callsites below a level at compile time.
• An enabled event is not. Formatting the fields, possibly allocating, and with a synchronous writer a write(2) per line puts you in microseconds. tracing-appender's non-blocking writer moves the I/O to a worker thread behind a channel. Make sure it drops rather than blocks when that channel is full, for the same reason as Day 3's ring.

The policy for the exchange, then: the engine thread emits no tracing events at all, only counters and event records. Gateway threads emit events for session lifecycle and anomalies, never per order. Every event carries the identifiers you'll need at 3 a.m.: session id, client order id, exchange order id, sequence number.

There is also one thing logs are not: the audit trail. The authoritative record of what the exchange did is the journal (week 9, week 18), which is sequenced, checksummed and replayable. Logs are for humans and can be lossy. The journal must not be.

Build the telemetry crate below, then run the w07 engine benchmark with telemetry enabled and disabled. If the difference shows up in p50, something on the hot path is doing more than a load and a store.