Time and Clocks

Linux exposes several clocks through clock_gettime, and Rust wraps two of them:

• CLOCK_REALTIME, which is SystemTime in Rust. Wall time, roughly UTC. It can jump: an operator runs date -s, NTP steps a large error, a leap second is applied by stepping. Never compute a duration from it.
• CLOCK_MONOTONIC, which is Instant on Linux. Never goes backwards, starts at an arbitrary point (boot), doesn't count suspend. NTP may slew its rate slightly to converge on true time, so a monotonic second isn't exactly an SI second, but it never jumps.
• CLOCK_MONOTONIC_RAW: monotonic and not slewed. It's the hardware's rate, including the hardware's error.
• CLOCK_TAI: International Atomic Time, without leap seconds, if your time daemon has told the kernel the TAI-UTC offset.

On x86 with the TSC as the kernel's clocksource, clock_gettime for the common clocks runs in the vDSO: a small piece of kernel-provided code mapped into your process that reads the TSC and the kernel's conversion parameters from shared memory, with no system call. That's where the ~20 ns comes from. If the kernel has decided the TSC is unreliable (you'll see "Marking TSC unstable" in dmesg, and some virtual machines do this), it falls back to a clocksource like hpet or acpi_pm, every read becomes a real system call plus a slow device access, and the cost grows by an order of magnitude or more. One file tells you which you have:

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Put that check in your deployment checklist and your startup code. A host that silently fell back to hpet will make every latency number you collect wrong in an interesting way: dominated by the cost of measuring it.

rdtsc reads the processor's 64-bit Time Stamp Counter. On any x86 CPU from the last decade or so it is invariant: it ticks at a constant rate regardless of frequency scaling, turbo or sleep states. That rate is the TSC frequency, usually close to the nominal base clock. CPUID leaf 0x80000007 reports this in EDX bit 8, and Linux shows it as the constant_tsc and nonstop_tsc flags in /proc/cpuinfo. Reading it costs roughly 20–40 cycles depending on the microarchitecture, a few nanoseconds, and it never enters the kernel.

#[inline(always)]
pub fn rdtsc() -> u64 {
    // SAFETY: rdtsc has no preconditions on x86_64.
    unsafe { core::arch::x86_64::_rdtsc() }
}

Two properties to respect:

rdtsc is not serializing. The CPU may execute it before earlier instructions have finished, or let later instructions start first. Intel's manual spells out the fixes: an lfence before rdtsc waits for earlier instructions to complete, an lfence after it stops later instructions from starting early. rdtscp waits for earlier instructions, though not later ones, and also returns a processor id (Linux puts the CPU number there). For timing a 30-cycle code fragment the fences matter. For stage timestamps on an order path measured in microseconds they cost more than they buy, so use plain rdtsc.

Ticks are not nanoseconds. You need the TSC frequency, and the honest way to get it is to measure it against the kernel's monotonic clock at startup. The conversion on the hot path should then be one multiply and one shift: ns = ns0 + ((ticks − tsc0) × mult) >> shift. That is exactly how the Linux kernel's own clocksource code converts cycles (its mult and shift fields). Two traps:

• The product overflows 64 bits after only a few seconds of GHz ticks. Widen to u128. On x86-64 that's still a single mul producing a 128-bit result.
• A single calibration reading can be interrupted between reading the reference clock and reading the TSC, and an interrupted reading has a misleading midpoint. Bracket each TSC read with two reference reads and keep the reading with the narrowest bracket. It's the one least likely to have been interrupted. Then use a long window (hundreds of milliseconds or more) between the start and end bursts, because the frequency error is the endpoint error divided by the window.

/// One calibration reading: the reference clock (CLOCK_MONOTONIC, ns) read
/// immediately before and immediately after one TSC read.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Sample {
    pub before_ns: u64,
    pub tsc: u64,
    pub after_ns: u64,
}

/// ns(tsc) = ns0 + (((tsc - tsc0) * mult) >> shift)
/// One multiply and one shift on the hot path: no division, no floats.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Calibration {
    pub tsc0: u64,
    pub ns0: u64,
    pub mult: u64,
    pub shift: u32,
}

pub const SHIFT: u32 = 32;

/// Pick the reading with the NARROWEST [before_ns, after_ns] bracket (the
/// one least likely to have been interrupted between the two clock reads)
/// and return (its tsc, the bracket midpoint). Ties go to the earliest.
pub fn best_point(burst: &[Sample]) -> (u64, u64) {
    let _ = burst;
    todo!()
}

/// Calibrate from a burst taken at the start and a burst taken at the end
/// of a window (the longer the window, the smaller the error).
///   tsc0, ns0 = best point of `start`
///   mult      = round((Δns << SHIFT) / Δtsc), computed in u128
pub fn calibrate(start: &[Sample], end: &[Sample]) -> Calibration {
    let _ = (start, end);
    todo!()
}

impl Calibration {
    /// Ticks → ns. Must stay right for hours: (tsc - tsc0) * mult overflows
    /// u64 after a few seconds of GHz ticks, so widen to u128. A tsc earlier
    /// than tsc0 maps to ns0.
    pub fn to_ns(&self, tsc: u64) -> u64 {
        let _ = tsc;
        todo!()
    }

    /// Estimated TSC frequency in ticks per ns (= GHz).
    pub fn ghz(&self) -> f64 {
        (1u128 << self.shift) as f64 / self.mult as f64
    }
}

Put the clock behind a trait and make every component generic over it, so the hot path inlines the read (static dispatch, week 37 goes deeper) and tests can control time exactly as week 17's token bucket did:

#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Default, Debug)]
pub struct Ticks(pub u64);

pub trait Clock {
    fn now(&self) -> Ticks;
    fn to_ns(&self, t: Ticks) -> u64;
}

/// Production on x86_64 with an invariant TSC the kernel also trusts.
pub struct TscClock { cal: Calibration }

/// Fallback: Instant, expressed as ns since an origin.
pub struct MonoClock { origin: std::time::Instant }

/// Tests and simulation: time moves only when the test says so.
pub struct ManualClock { now: std::cell::Cell<u64> }

pub struct Gateway<C: Clock> { clock: C, /* … */ }

Choose the implementation once, at startup: TSC only if the CPU reports an invariant TSC and the kernel's clocksource is tsc, since the kernel has information about cross-core synchronization and virtualization that a CPUID bit doesn't. Otherwise fall back to MonoClock and log loudly that you did.

Where the stamps go matters as much as how they're taken. Stamp where an order changes hands: when the read syscall returns with its bytes, when it's pushed to the engine queue, when the engine pops it, when matching finishes, when the ack's write returns. A gap between two stamps is then attributable to exactly one owner. Even the first stamp is late, though. The packet waited in the NIC and the kernel before your read returned. The SO_TIMESTAMPING socket option can give you kernel or NIC receive timestamps for that earlier moment, and week 33 comes back to hardware timestamping.

Budget it: five rdtsc reads cost a few tens of nanoseconds per order, five Instant::now() around a hundred. Both are acceptable for an order path measured in microseconds. The second is not acceptable for a matching path measured in hundreds of nanoseconds, which is one more reason the engine itself takes only one stamp, or none.

Two machines agree on time the same way whether the protocol is NTP or PTP: a two-way exchange of four timestamps. The client sends at t1 (its clock), the server receives at t2 and replies at t3 (its clock), the client receives at t4. Then:

offset = ((t2 − t1) + (t3 − t4)) / 2      server clock minus client clock
delay  = (t4 − t1) − (t3 − t2)            round-trip time on the wire

The derivation assumes the path takes the same time in each direction. If it doesn't, the offset estimate is wrong by half the asymmetry, and nothing in the four timestamps can reveal it. Asymmetry comes from queueing in switches, from different routes each way, and from software timestamping (time spent in the kernel before the timestamp is taken). Two defenses:

• Filter by delay. Of many exchanges, the one with the smallest round trip had the least queueing, so the least room for asymmetry. NTP's clock filter works on this principle, and so does the drill.
• Take the timestamps in hardware and correct for the network. PTP (IEEE 1588) uses the same math, but with timestamps taken by the NIC as packets cross the wire, and with switches that either act as boundary clocks (terminate and re-originate time) or transparent clocks (add their own residence time into the packet). On Linux, linuxptp's ptp4l disciplines the NIC's hardware clock to the grandmaster and phc2sys disciplines the system clock to the NIC.

Rough orders of magnitude: NTP over the internet gives milliseconds. NTP on a LAN with a good daemon (chrony) gives tens of microseconds to better. PTP with hardware timestamping and PTP-aware switches gives sub-microsecond, down to tens of nanoseconds with good hardware. The grandmaster is usually disciplined by GNSS with a pulse-per-second signal.

This isn't optional for a European venue. Under MiFID II's RTS 25, trading venues whose gateway-to-gateway latency is under a millisecond, and members using high-frequency algorithmic techniques, must keep business clocks within 100 µs of UTC and timestamp with 1 µs or finer granularity. You can't meet that with NTP across the internet, and you can't demonstrate it without monitoring your own offset.

/// One request/response exchange between a client and a time server:
///   t1  client sends     (client clock)
///   t2  server receives  (server clock)
///   t3  server replies   (server clock)
///   t4  client receives  (client clock)
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Exchange {
    pub t1: u64,
    pub t2: u64,
    pub t3: u64,
    pub t4: u64,
}

/// offset = server clock − client clock = ((t2 − t1) + (t3 − t4)) / 2
/// delay  = time on the wire, both ways  = (t4 − t1) − (t3 − t2)
/// Use signed arithmetic (the server may be BEHIND the client, so t2 < t1
/// is normal) and i64 division by 2.
pub fn offset_and_delay(x: &Exchange) -> (i64, i64) {
    let _ = x;
    todo!()
}

/// NTP's clock filter in miniature: of several exchanges, trust the one
/// with the SMALLEST delay (least queueing, so least room for asymmetry).
/// Skip exchanges whose delay is negative (a bad timestamp). Returns that
/// exchange's (offset, delay), or None if nothing is usable.
pub fn best_offset(xs: &[Exchange]) -> Option<(i64, i64)> {
    let _ = xs;
    todo!()
}

/// One-way latency of a message sent at `sent_a` on host A's clock and
/// received at `recv_b` on host B's clock, given offset = B − A.
pub fn one_way(sent_a: u64, recv_b: u64, offset_b_minus_a: i64) -> i64 {
    let _ = (sent_a, recv_b, offset_b_minus_a);
    todo!()
}

Everything above is for measurement. The matching engine has a stricter rule: it never reads a clock. If matching logic consults now() (order expiry, auction end, the week 17 throttle refill, a "minimum resting time" rule), then replaying the journal tomorrow gives different answers, and week 9's replay, week 18's recovery and week 20's standby all silently diverge.

Instead, time enters through the log like everything else:

• The sequencer stamps each command with a transact time, read once from a PTP-disciplined realtime clock when the command is sequenced. That timestamp is journaled with the command, replicated with it, and is the only "now" the engine knows about. It is also the time you send back to clients (FIX TransactTime, tag 60) and publish on market data.
• Timers are commands. "The opening auction ends", "GTD orders expire at 16:00", "advance the throttle clock" are injected into the sequenced stream by a timer source as ordinary commands with sequence numbers. The engine reacts to them, and on replay it reacts identically.
• Measurement clocks stay outside. TSC ticks for stage latencies ride alongside the command (week 19) but are never journaled as input and never touch state.

The sequencer's realtime clock can still misbehave. If NTP or PTP steps it backwards, the sequencer must not emit transact times that go backwards. Clamp to the previous value (and alarm), so the log's time is monotonic by construction. Leap seconds deserve one sentence in your runbook too. Smearing (spreading the second over hours) and stepping both exist in the wild, and the international standards bodies voted in 2022 to stop adding leap seconds by 2035, so this problem is slowly going away.

Build the clock layer below. Before you benchmark, check current_clocksource, the constant_tsc and nonstop_tsc flags, and dmesg for TSC warnings on the machine you measure on, and record them in the report.