Threads, Cores & Busy Polling

When a consumer parks and a producer later unparks it, the producer makes a futex wake syscall; the kernel marks the consumer runnable and, if it belongs on another CPU, sends that CPU an inter-processor interrupt; if that CPU was idle it may have to leave a deep C-state first; the scheduler then switches to the consumer, which returns from its own futex wait syscall into user space with partly cold caches. Each step is cheap by OS standards. Together they typically cost from a few microseconds to tens of microseconds, with a long tail — on the machine these drills were tested on, a park/unpark wakeup measured around 16 µs.

Compare a busy-polling consumer: it is already running, spinning on a cache line. The producer's store invalidates that line in the consumer's cache; the consumer's next load fetches it from the producer's core. That is a core-to-core cache-line transfer — tens of nanoseconds within a socket on current hardware, more across sockets.

Idle states deserve their own sentence: a core in a deep C-state can take tens of microseconds — over a hundred on some server parts — to wake. Check /sys/devices/system/cpu/cpu0/cpuidle/state*/latency on your machine. Low-latency hosts limit C-states (kernel parameters, or holding /dev/cpu_dma_latency open with a value of 0) precisely so that a core that does sleep wakes quickly.

The hybrid strategy escalates:

1. Spin with std::hint::spin_loop() — the x86 PAUSE instruction, which tells the core this is a spin-wait (saving power and, on SMT cores, giving issue slots to the sibling). Its latency varies a lot by microarchitecture: on Intel cores since Skylake it takes on the order of a hundred cycles or more, versus roughly ten on earlier ones, so "spin N times" is not a portable unit of time.
2. Yield with thread::yield_now() — a sched_yield syscall that lets another runnable thread on this CPU run, and returns immediately if there is none.
3. Park — sleep in the kernel until unparked.

The notifier wants to skip the expensive unpark syscall when the waiter is still spinning. So the waiter announces "I'm about to sleep" with a flag, and the notifier only unparks if it sees the flag. Here is the trap:

• Waiter: sleeping = true, then check ready; if not ready, park.
• Notifier: ready = true, then check sleeping; if set, unpark.

Each side stores one variable and then loads the other. That is the store-buffering pattern, and with anything weaker than sequentially consistent ordering both loads may see the old values — even on x86, where each core's store buffer lets a later load overtake an earlier store. The waiter sees ready == false and parks; the notifier sees sleeping == false and does not unpark. The notification is lost, and the waiter sleeps until the next one — or forever.

x.store(1, Relaxed);
let r1 = y.load(Relaxed);

y.store(1, Relaxed);
let r2 = x.load(Relaxed);

The fix is SeqCst on all four accesses (or SeqCst fences between each store and the following load), which forbids the outcome where both loads miss. And thread::park itself is forgiving in the right way: an unpark that arrives before the park leaves a token, so the park returns immediately; parks may also return spuriously, so the waiter always loops and re-checks.

use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Mutex;
use std::thread::{self, Thread};

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Step {
    Spin,
    Yield,
    Park,
}

/// Escalating wait strategy: spin a little, yield a little, then park.
pub struct Backoff {
    spin_limit: u32,
    yield_limit: u32,
    step: u32,
}

impl Backoff {
    pub fn new(spin_limit: u32, yield_limit: u32) -> Self {
        Backoff { spin_limit, yield_limit, step: 0 }
    }

    /// The first `spin_limit` calls return Spin, the next `yield_limit`
    /// return Yield, and every call after that returns Park.
    /// (Beware overflow when spin_limit is u32::MAX.)
    pub fn next(&mut self) -> Step {
        todo!()
    }

    pub fn reset(&mut self) {
        todo!()
    }
}

#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct WaitStats {
    pub spins: u64,
    pub yields: u64,
    pub parks: u64,
}

/// Auto-reset event for ONE waiting thread and any number of notifiers.
pub struct Event {
    ready: AtomicBool,
    sleeping: AtomicBool,
    waiter: Mutex<Option<Thread>>,
    unparks: AtomicU64,
}

impl Event {
    pub fn new() -> Self {
        Event {
            ready: AtomicBool::new(false),
            sleeping: AtomicBool::new(false),
            waiter: Mutex::new(None),
            unparks: AtomicU64::new(0),
        }
    }

    /// Block until notified, consuming the notification (ready -> false).
    /// 1. Register thread::current() in `waiter`.
    /// 2. Loop: if ready, consume it and return the stats. Otherwise act on
    ///    backoff.next(): spin (std::hint::spin_loop), yield, or park.
    /// 3. Before parking: set `sleeping`, RE-CHECK `ready`, and only park if
    ///    it is still false. Clear `sleeping` after waking. Count a park only
    ///    when you actually called thread::park(). Mind the orderings.
    pub fn wait(&self, backoff: &mut Backoff) -> WaitStats {
        todo!()
    }

    /// Publish readiness; unpark the waiter only if it announced `sleeping`,
    /// and count each unpark in `unparks`.
    pub fn notify(&self) {
        todo!()
    }

    /// How many times notify actually called unpark (a futex syscall each).
    pub fn unpark_calls(&self) -> u64 {
        self.unparks.load(Ordering::Relaxed)
    }
}

Rust's standard library has no affinity API; use libc::sched_setaffinity directly (or a small crate such as core_affinity) from inside the thread you want to pin, and name your threads so tools show them. Pinning alone only says where your thread may run. To keep everything else off those cores:

• isolcpus= removes CPUs from the scheduler's load balancing, so nothing lands there unless explicitly pinned. The kernel documents it as deprecated in favour of cpusets, but it remains very common on trading hosts; cpusets (or tuned's cpu-partitioning profile) achieve the same more flexibly.
• nohz_full= stops the periodic scheduler tick on those CPUs while they run a single task, removing a regular interruption (with a 1 kHz tick, one every millisecond). rcu_nocbs= moves RCU callback processing off them.
• Interrupts: set /proc/irq/*/smp_affinity (and irqaffinity= at boot) so device interrupts go to housekeeping cores, and stop irqbalance from undoing it. NIC queues are the exception you place deliberately (week 31).
• SMT siblings share L1, L2 and execution units. Two hot threads on sibling hyperthreads slow each other down unpredictably; most latency setups leave the sibling of a hot core idle or disable SMT.
• Housekeeping: something must run the OS, logging, metrics and the control plane. Reserve core 0 (and its sibling) for that.

Look at your machine before planning: lscpu -e lists every logical CPU with its core, socket and NUMA node; lstopo draws it; on many Intel servers CPU i and CPU i + N are siblings, but check /sys/devices/system/cpu/cpu0/topology/thread_siblings_list rather than assuming.

/// One logical CPU as Linux numbers it. SMT siblings (hyperthreads) share
/// the same `core`; `core` ids are unique across the whole machine.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Cpu {
    pub id: usize,
    pub core: usize,
    pub node: usize,
}

pub struct Topology {
    pub cpus: Vec<Cpu>,
    /// NUMA node the NIC is attached to (/sys/class/net/<if>/device/numa_node).
    pub nic_node: usize,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Role {
    /// Busy-polling, latency-critical: engine, gateway rx, market-data publisher.
    Hot,
    /// Needs its own CPU but tolerates a busy sibling: journal writer, risk refresher.
    Warm,
    /// Housekeeping: metrics, logging, control plane.
    Cold,
}

#[derive(Clone, Debug)]
pub struct ThreadSpec {
    pub name: &'static str,
    pub role: Role,
}

/// Return one logical CPU id per thread (same order as `threads`), or
/// Err(reason) when no valid plan exists.
///
/// 1. The housekeeping core is the physical core containing CPU id 0. Every
///    Cold thread runs on one of its logical CPUs; nothing else may.
/// 2. Each Hot thread gets a whole physical core on `nic_node`, outside the
///    housekeeping core: no other thread on that CPU or its SMT siblings.
/// 3. Each Warm thread gets a logical CPU nobody else uses, outside the
///    housekeeping core and every Hot core. Prefer `nic_node`: a Warm thread
///    may sit on another node only if no eligible CPU on `nic_node` is left.
pub fn plan(topo: &Topology, threads: &[ThreadSpec]) -> Result<Vec<usize>, String> {
    todo!()
}

On a multi-socket server (and some large single-socket parts) memory is attached to NUMA nodes. A load from the local node's DRAM is roughly the same cost as on a single-socket machine; a load from a remote node crosses the socket interconnect and typically costs 1.5–2× as much. Cache-line transfers between cores on different sockets are similarly more expensive than within a socket.

Three rules for an exchange:

• Put the hot path on the NIC's node. Packets are DMA'd into memory near the NIC; a gateway thread on the other socket pays the remote penalty on every packet.
• Memory goes where it is first touched. Linux's default policy places a page on the node of the CPU that first writes it. Allocate and prefault each thread's data (week 27) from that thread, after it is pinned — not from main before spawning.
• Keep pipelines on one socket. A queue whose producer and consumer are on different sockets pays cross-socket transfers for every cache line of every message.

numactl --hardware shows nodes and their distances; numactl --cpunodebind=1 --membind=1 runs a whole process on one node; numastat -p shows where a process's memory actually ended up.

Even a pinned, isolated, busy-polling thread is occasionally interrupted: device interrupts you did not move, inter-processor interrupts (TLB shootdowns when another thread of your process unmaps memory, for example), kernel worker threads, a residual timer tick, SMIs from firmware. Find them empirically:

• Hiccup measurement. Run a thread on the isolated core that does nothing but read the clock in a tight loop and record every gap longer than, say, 1 µs into a histogram. That is the noise floor of the core — your engine can never be better. (Gil Tene's jHiccup applies the same idea to JVMs.)
• Counters: perf stat -e context-switches,cpu-migrations -p <pid> should show zero of both for a pinned busy-polling thread; /proc/interrupts shows which interrupts still land on which CPUs.

Spinning has real costs, so choose it deliberately: a busy-polling thread uses 100% of its core forever, consumes power and thermal headroom (which can lower the turbo frequency other cores reach), and on SMT steals resources from its sibling. A typical exchange busy-polls the gateway receive path, the engine and the market-data publisher, and lets the journal writer, snapshotting, metrics and admin threads spin briefly and then park.