Kernel Networking Costs

Receiving a TCP segment on a standard Linux box:

1. The NIC DMAs the frame into a ring buffer in host memory and raises an interrupt — possibly after waiting to coalesce several frames (ethtool -C).
2. The interrupt handler schedules NAPI polling; a softirq pulls frames off the ring, possibly merging them (GRO), and runs IP and TCP processing.
3. The data lands in the socket's receive queue, an ACK is scheduled, and the thread blocked in recv/epoll_wait is woken — the futex-style wakeup path from week 30.
4. Your thread returns from the syscall and the kernel copies the bytes into your buffer.

Sending is the mirror image: a syscall, a copy from your buffer into socket memory, TCP output (which may decide not to send yet), the queueing discipline, the driver, and a DMA by the NIC.

Rough costs on current hardware, all highly dependent on CPU, kernel version and security mitigations: a trivial syscall's entry and exit is from under 100 ns to several hundred ns (mitigations such as KPTI on affected CPUs are the difference); a small TCP send or recv costs a few microseconds of CPU through the stack; a thread wakeup is several microseconds or more. A request/response round trip through the kernel on loopback typically lands in the tens of microseconds unless both ends busy-poll.

Step the packet through the kernel path, then switch to busy-poll and see exactly which hops disappear and which stay. Finally switch the reply to Nagle: one socket option outweighs everything else on the path.

Nagle's algorithm (sender side): if any sent data is still unacknowledged, hold new small writes in the send buffer until either the ACK arrives or a full segment's worth (MSS) has accumulated. Delayed ACK (receiver side): do not ACK every segment immediately; wait a little — on Linux at least 40 ms, at most 200 ms — in case there is response data to piggyback the ACK on, but ACK at once when a second full segment arrives.

Each is reasonable. Together they deadlock for the length of the delayed-ACK timer. The classic trigger is write-write-read: a client sends a request as two writes (say, a header and then a body) and waits for the response. The header goes out immediately. The body is small and the header is unacknowledged, so Nagle holds it. The server has received only a partial request, has nothing to send back, and is delaying its ACK. Forty milliseconds later the delayed-ACK timer fires, the ACK arrives, Nagle releases the body, and the request completes — about 40 ms late, on a network with a 100 µs round trip.

The fixes, in order of preference for a trading gateway:

• TCP_NODELAY on every socket: disable Nagle; small writes leave immediately. Universal in low-latency systems.
• Write whole messages: build the full message (or a batch) in one buffer, or use one vectored write (Day 3), so there is no second small write to hold back. Fewer syscalls, fewer packets.
• TCP_QUICKACK on the receiver disables delayed ACKs only temporarily — the kernel may re-enter delayed-ACK mode, so it has to be set again after reads. Useful, but not a substitute for the first two.

One reason this bug survives testing: Linux starts connections in a quick-ACK mode, ACKing immediately for the first few segments, so a short test may never show the stall that appears minutes into a production session.

use std::cmp::Reverse;
use std::collections::BinaryHeap;

#[derive(Clone, Copy, Debug)]
pub struct Net {
    /// One-way delay, each direction (wire + both stacks).
    pub one_way_ns: u64,
    /// Receiver's delayed-ACK timer (Linux: 40 ms minimum).
    pub delack_ns: u64,
    /// Bytes in a full segment.
    pub mss: u32,
}

#[derive(Clone, Copy, Debug)]
pub struct Write {
    pub at_ns: u64,
    /// <= mss. Writes are given in non-decreasing at_ns order.
    pub len: u32,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Outcome {
    /// When each write's bytes arrived at the receiver.
    pub delivered_ns: Vec<u64>,
    /// Data segments sent.
    pub segments: usize,
}

/// Simulate one TCP connection carrying small writes (one direction only).
///
/// Sender
///  - nagle == false: every write is sent at once as its own segment.
///  - nagle == true: a write is sent at once only if no sent segment is
///    unacknowledged AND nothing is buffered; otherwise it joins the buffer.
///    The buffer leaves as ONE segment when (a) it reaches mss bytes, or
///    (b) an ACK arrives that leaves no sent segment unacknowledged.
///  - A segment sent at t arrives at t + one_way_ns (segments stay in order).
/// Receiver
///  - On each segment arrival: if it now holds 2 un-ACKed segments, ACK at
///    once; otherwise start a delack_ns timer unless one is running.
///  - When the timer fires: ACK. Sending an ACK cancels any running timer.
///  - An ACK covers every segment received so far and reaches the sender
///    one_way_ns later.
/// Ties at the same instant are processed in this order: ACK arrivals at the
/// sender, then app writes, then segment arrivals, then timer expiries.
///
/// Hint: a BinaryHeap<Reverse<(time, kind, seq, payload)>> event queue.
pub fn simulate(net: &Net, writes: &[Write], nagle: bool) -> Outcome {
    todo!()
}

Every send is a syscall plus a trip through TCP output. If the engine produces five execution reports for one session in a single batch, five sends cost five syscalls and likely five packets; one write of all five costs one syscall and, with TCP_NODELAY, typically one packet (if it fits in a segment).

The tools:

• writev (Write::write_vectored in Rust): one syscall, many buffers — a header from here, a payload from there, no copying into a staging buffer. Linux allows up to IOV_MAX (1024) buffers per call.
• Coalescing copies: for small messages, copying them into one contiguous, reused buffer and issuing one write is often as fast as building an iovec array, and simpler. Use writev when payloads are large or live in separate places.
• sendmmsg / recvmmsg: many UDP datagrams per syscall — the market-data publisher's equivalent.
• MSG_MORE / TCP_CORK: tell TCP more data is coming so it should not send a partial segment yet — the opposite of TCP_NODELAY, useful for bulk transfers, dangerous on an order path.

The catch is partial writes. A non-blocking socket accepts only as many bytes as fit in its send buffer; writev returns how many it took, and that can end in the middle of one of your buffers. You must resume at exactly that byte, skip empty buffers, retry on EINTR, and treat "zero bytes accepted" as an error rather than spinning forever. Rust's IoSlice::advance_slices helps; a write_all_vectored exists in std but is still unstable (as of Rust 1.97), so production code writes this loop itself — which is the drill.

use std::io::{self, IoSlice, Write};

/// Write every byte of `bufs` with as few `write_vectored` calls as possible
/// (one call per partial write — never one call per buffer).
/// - A call may accept fewer bytes than offered, even stopping in the middle
///   of a slice: resume exactly there.
/// - Skip empty slices, and never call write_vectored with nothing to write.
/// - Retry on io::ErrorKind::Interrupted; Ok(0) while bytes remain is an
///   error of kind io::ErrorKind::WriteZero.
/// Return the total number of bytes written.
pub fn write_all_vectored<W: Write>(w: &mut W, bufs: &[&[u8]]) -> io::Result<usize> {
    todo!()
}

/// Frame each message as [u16 little-endian length][payload] into `out`
/// (clear it first, reuse its capacity) so a whole batch can go out in one
/// write. Return Err(index) for the first payload longer than u16::MAX.
pub fn frame_batch(msgs: &[&[u8]], out: &mut Vec<u8>) -> Result<(), usize> {
    todo!()
}

Socket buffers. SO_SNDBUF and SO_RCVBUF size the kernel queues on each side of a socket. Linux doubles the value you set (to account for bookkeeping) and caps it at net.core.wmem_max / rmem_max, and setting a buffer size explicitly turns off the kernel's autotuning for it. For throughput, bigger is better. For latency, a buffer is a queue: a full 4 MB send buffer draining at 10 Gb/s holds 3.2 ms of data, and any new order joins the back of it. In a gateway, keep send buffers modest and make backpressure explicit in the application (week 24): a session that cannot keep up should be throttled, conflated or disconnected — not silently queued for milliseconds.

Interrupt coalescing (ethtool -C rx-usecs …) makes the NIC wait to batch interrupts: less CPU, more latency. GRO/LRO merge consecutive segments into one large one: fewer passes through the stack, but a small packet may wait for company. Latency-tuned hosts reduce both on the trading NIC.

Steering. RSS hashes each flow to one of the NIC's receive queues, each with its own interrupt. Pin the interrupt of your gateway's queue to a core on the NIC's NUMA node — not the busy-polling engine core (week 30) — or, with busy polling (Day 5), process the queue from the gateway thread itself.

Two levels of busy polling:

• Application level: put the socket in non-blocking mode and spin on recv. No sleep, no wakeup — but every empty poll is still a syscall, and the data still has to be delivered by the interrupt/softirq path on whichever core handles the NIC queue.
• Kernel socket busy polling: SO_BUSY_POLL (or the net.core.busy_read / net.core.busy_poll sysctls for recv and poll/epoll) makes a receive with no data spin in the kernel for up to N microseconds, polling the device's queue directly from your thread. With "prefer busy poll" and deferred interrupts, a busy-polling application can drive the queue itself and interrupts stay off. Requires driver support (NAPI) and is per receive queue, so steering (Day 4) matters.

Both cost a core at 100%, which is why the gateway receive thread is in week 30's Hot class. Measure the difference — on loopback and, if you can, on a real NIC between two machines — before deciding it is worth it.

The standard microbenchmark is a ping-pong: client sends a small message, server echoes it, client measures the round trip; repeat a few hundred thousand times into the week 25 histogram (a ping-pong is closed-loop by nature, so it measures per-message latency at low load, not behaviour under load — say so in the report). Run it with blocking receives and with busy-polling receives, pinned and unpinned.

Know what loopback leaves out: there is no NIC, no DMA, no device interrupt or coalescing; packets still traverse TCP and IP and the softirq backlog, so the syscall, copy and wakeup costs are real. Loopback numbers are a floor for the kernel-stack part, not a prediction for a real network.

To split a round trip into parts, use timestamps: SO_TIMESTAMPING can report when the kernel (or, with NIC support, the hardware) sent and received each packet, so you can separate time-in-application from time-in-stack. ss -ti shows per-connection TCP state (RTT, congestion window, retransmits, delivery rate) and is the first thing to look at when a session is slow.