Kernel Bypass

Follow one market-data packet through an unmodified Linux box:

1. Wire → NIC. The frame is deserialised (a 64-byte frame takes 51.2 ns at 10 Gb/s and about 20 ns at 25 Gb/s), checked, filtered by MAC address, and hashed (RSS) to pick one of the NIC's receive queues.
2. NIC → memory (DMA). Each receive queue is a ring of descriptors in host memory. The driver pre-filled each one with the address of an empty buffer. The NIC writes the frame into the next buffer over PCIe, then writes the descriptor back with the length and a descriptor done bit. The PCIe transactions alone cost hundreds of nanoseconds.
3. Interrupt. The NIC raises an MSI-X interrupt, possibly after a delay, because interrupt coalescing (ethtool -C … rx-usecs) deliberately waits to batch interrupts. Defaults are tuned for throughput, not latency.
4. NAPI and softirq. The interrupt handler schedules NAPI and returns. In the NET_RX softirq the driver's poll function walks the completed descriptors, allocates an sk_buff for each, refills the ring, and hands packets up the stack (GRO, IP, UDP, socket lookup, any netfilter rules).
5. Socket queue and wakeup. The packet lands on the socket's receive queue and the kernel wakes the thread blocked in recv or epoll_wait. If that thread was asleep, the scheduler has to run it. If its core was in a deep C-state, waking the core alone can take tens of microseconds.
6. Copy out. recv copies the payload into your buffer, frees the sk_buff and returns.

Stages 3 to 6 run on the CPU, in the kernel, for every packet. On a carefully tuned server (coalescing off, busy polling, pinned and isolated cores, C-states limited) the whole path is typically a few microseconds. On a default configuration it's tens of microseconds, with a long tail. The work also costs throughput: a kernel stack that spends about a microsecond of CPU per packet can't keep up with a 10 GbE link of minimum-size frames (14.88 million packets per second) on one core, or on ten.

Switch to kernel bypass and watch stages 3 to 6 drop out of the path. Then put the bypass thread back to sleep on an interrupt and see how much of the saving was the wakeup.

Every bypass technology, and the kernel driver itself, speaks the same contract with the NIC. Only who polls the ring changes.

• Software owns a pool of buffers in DMA-able memory (a DPDK mempool of mbufs, an AF_XDP UMEM, an ef_vi registered memory region).
• Software posts buffers by writing their addresses into RX descriptors and then writing the new tail index to a NIC register. That's the doorbell: an MMIO write across PCIe, so drivers batch it.
• The NIC fills posted descriptors in order. For each frame it DMAs the bytes, then writes the descriptor back with the length, status flags and a done (DD) bit.
• Software polls the DD bit of the next descriptor. When it's set, the frame is in memory and the descriptor can be consumed.
• When the application is finished with a buffer, it goes back to the pool and gets re-posted.

Two failure modes fall straight out of this contract. You'll reproduce both in the drill.

The NIC drops in hardware when it runs out of posted descriptors. If your poll loop stalls (a page fault, a log flush, a slow path) for longer than the ring takes to fill, the NIC has nowhere to put the next frame and discards it. It increments a counter like rx_missed_errors in ethtool -S or imissed in DPDK, and nothing else tells you. At 10 million packets per second, a 512-entry ring fills in about 50 µs.

Holding buffers starves the ring. Zero-copy means the application reads the frame in the DMA buffer. If it keeps buffers (queues them for another thread, or batches them for later), they aren't in the pool, so refill can't post them and the ring drains even though the poll loop is running. Size the pool for ring size plus the maximum your pipeline can hold in flight, and treat "pool empty" as an alarm, not a slow path.

pub const BUF_SIZE: usize = 2048;

/// RX descriptor as the NIC sees it: which buffer to DMA into, plus the
/// fields the NIC writes back (length, "descriptor done").
#[derive(Clone, Copy, Debug, Default)]
pub struct Desc {
    pub buf: u32,
    pub len: u16,
    pub dd: bool,
}

/// A received packet handed to the application. Its bytes still live in
/// the DMA buffer: nothing was copied.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Pkt {
    pub buf: u32,
    pub len: u16,
}

pub struct RxRing {
    descs: Vec<Desc>,
    bufs: Vec<[u8; BUF_SIZE]>,
    free: Vec<u32>, // the buffer pool (a DPDK mempool, an AF_XDP fill queue)
    mask: u32,
    hw_head: u32, // next descriptor the NIC fills
    sw_next: u32, // next descriptor software checks for dd
    tail: u32,    // descriptors posted to the NIC are [hw_head, tail)
    missed: u64,  // frames dropped because no descriptor was posted
}

impl RxRing {
    /// `ring_size` is a power of two. All `pool_size` buffers start free;
    /// post as many as fit, as a driver does at start-up.
    pub fn new(ring_size: usize, pool_size: usize) -> Self {
        todo!()
    }

    /// Driver: post free buffers into empty descriptor slots and advance the
    /// tail (the doorbell write). A slot is reusable once software has
    /// consumed it. Returns how many descriptors were posted.
    pub fn refill(&mut self) -> usize {
        todo!()
    }

    /// NIC: DMA one frame into the next posted descriptor and write back len
    /// and dd. No posted descriptor -> drop it and count it in `missed`.
    /// Frames longer than BUF_SIZE are truncated.
    pub fn nic_receive(&mut self, frame: &[u8]) -> bool {
        todo!()
    }

    /// Application poll loop: hand out up to `max` completed packets in
    /// order, clearing dd. No syscall, no copy.
    pub fn rx_burst(&mut self, max: usize, out: &mut Vec<Pkt>) -> usize {
        todo!()
    }

    /// Zero-copy view of a packet's bytes.
    pub fn data(&self, p: Pkt) -> &[u8] {
        todo!()
    }

    /// The application is done with the packet: its buffer returns to the pool.
    pub fn release(&mut self, p: Pkt) {
        todo!()
    }

    pub fn missed(&self) -> u64 {
        self.missed
    }
}

DPDK (the Data Plane Development Kit) is the maximalist option. You unbind the NIC from its kernel driver and bind it to vfio-pci, which uses the IOMMU so the device can only DMA into memory you mapped for it. A user-space poll-mode driver then programs the hardware directly. Packet buffers come from mempools in hugepages (2 MB or 1 GB pages, so a few TLB entries cover the whole pool), and one or more cores spin in rte_eth_rx_burst, pulling up to a burst (commonly 32) of mbufs per call. That gives you no interrupts, no system calls on the data path and line rate on a handful of cores. It also means the kernel no longer sees the NIC: no ip addr, no tcpdump (DPDK ships its own capture tool), no kernel TCP. You implement or link whatever protocols you need, down to ARP and IGMP for multicast joins. Some NICs, notably NVIDIA/Mellanox ConnectX via the mlx5 driver, use a bifurcated model where the kernel keeps the interface and DPDK takes only the flows you steer to it.

AF_XDP (Linux 4.18 and later) is the kernel's own answer, and it's the pragmatic middle ground. You register a UMEM, a region of your memory split into fixed-size frames, and create an AF_XDP socket with four rings: fill (buffers you give the kernel for receiving), RX (filled frames coming back), TX, and completion (TX buffers you can reuse). Those are the descriptor-ring contract again, one level up. A small XDP program (eBPF, run by the driver before any sk_buff exists) decides per packet whether to redirect it into your socket or let it continue up the normal stack. With driver support in zero-copy mode, the NIC DMAs straight into your UMEM. Without it, copy mode still skips most of the stack. The NIC stays a normal Linux interface, so SSH, monitoring and everything else on that port keep working, and you bypass only the multicast group or port that matters.

The trading world's default for a long time has been Solarflare NICs (Solarflare was bought by Xilinx in 2019, and Xilinx by AMD in 2022), largely because of their software stack:

• Onload is a complete TCP/UDP stack in user space that implements the BSD socket API. You run an unmodified binary under it (onload ./gateway) and it intercepts socket calls via LD_PRELOAD, so recv reads straight from the NIC's rings in your process. You keep epoll, TCP semantics and your code. You can make it spin instead of sleep (EF_POLL_USEC). It's the easiest bypass to adopt, and order-entry gateways that need real TCP often use it.
• ef_vi is the layer underneath: a low-level API to a virtual interface with its own RX/TX descriptor rings and event queue. You install filters (for example "this multicast group to my VI") and receive raw Ethernet frames, so parsing IP and UDP and handling IGMP are your job. Feed handlers that want the absolute minimum between NIC and parser use it.
• TCPDirect is a minimal, zero-copy TCP/UDP API built on ef_vi. It's faster than Onload because it drops socket-API compatibility.

Other vendors have equivalents (NVIDIA's VMA for ConnectX, the ExaNIC libraries from Exablaze, now Cisco), and beyond software bypass sit FPGAs that parse the feed and fire orders in hardware, with tick-to-trade measured in hundreds of nanoseconds. The pattern is consistent. Market data (UDP multicast, read-only, parse-heavy) goes to the lowest-level API the team can maintain. Order entry (TCP, stateful, must survive retransmits) goes through Onload or TCPDirect, because writing a correct TCP stack is a project of its own.

What changes in your code when you adopt any of these:

• The receive loop becomes a poll loop on a pinned, isolated core that never sleeps (week 30). Idle costs a whole core.
• Data arrives in NIC-owned buffers. Parse it in place (week 11's zero-copy views), and copy out only what must outlive the buffer.
• Batching is natural: a poll returns every frame that has arrived (week 32's lesson).
• NUMA matters. The NIC hangs off one CPU socket's PCIe lanes, so poll from a core on that socket and allocate buffers in its memory, or every packet crosses the interconnect.

Now do the arithmetic. The visualiser below lays out a typical kernel receive path next to a bypass path. Change the stage costs and watch which ones dominate. The drill then turns the same arithmetic into functions: wire time, line rate, cores needed and a go/no-go verdict.

#[derive(Clone, Copy, Debug)]
pub struct Stage {
    pub name: &'static str,
    pub ns: u64,
    pub in_kernel: bool,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Verdict {
    KernelIsFine,
    BypassNeeded,
    BypassNotEnough,
}

/// Time to clock a frame of `frame_bytes` onto a `gbps` link, in ns,
/// rounded up.
pub fn serialization_ns(frame_bytes: u64, gbps: u64) -> u64 {
    todo!()
}

/// Maximum frames per second on the link. Each frame also occupies 8 bytes
/// of preamble + start-of-frame delimiter and a 12-byte inter-frame gap.
/// Rounded down.
pub fn line_rate_pps(frame_bytes: u64, gbps: u64) -> u64 {
    todo!()
}

/// Cores needed to absorb `pps` if each packet costs `per_packet_ns` of CPU
/// on one core. Rounded up.
pub fn cores_needed(per_packet_ns: u64, pps: u64) -> u64 {
    todo!()
}

/// Wire-to-application latency. With `bypass`, the in-kernel stages vanish
/// and the user-space receive path (`user_rx_ns`: poll + protocol parse)
/// is added instead.
pub fn path_ns(stages: &[Stage], bypass: bool, user_rx_ns: u64) -> u64 {
    todo!()
}

/// Fraction of the end-to-end time (path + application work) that bypass
/// removes: (kernel_total - bypass_total) / kernel_total.
pub fn bypass_saving(stages: &[Stage], user_rx_ns: u64, app_ns: u64) -> f64 {
    todo!()
}

/// Decide, given the end-to-end target for path + application work.
pub fn verdict(kernel_path_ns: u64, bypass_path_ns: u64, app_ns: u64, target_ns: u64) -> Verdict {
    todo!()
}

Bypass claims are claims about time, so measure them at the right place. A software timestamp taken when your code reads the packet includes everything you're trying to measure. Modern NICs can timestamp each frame in hardware as it passes the MAC, using their own clock: the PTP hardware clock (PHC).

On Linux:

• Enable it on the device with the SIOCSHWTSTAMP ioctl (an RX filter, often "all packets", plus TX on), then ask for it per socket with SO_TIMESTAMPING, using SOF_TIMESTAMPING_RX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE and the TX equivalents.
• RX timestamps arrive as ancillary data on recvmsg (an SCM_TIMESTAMPING control message carrying three timespecs, where the raw hardware stamp is the third). TX timestamps come back later on the socket's error queue (MSG_ERRQUEUE), because the frame is timestamped when it actually leaves, after send has returned.
• Bypass APIs expose the same stamps directly. ef_vi can deliver an RX timestamp with each receive event, and DPDK exposes NIC-dependent RX timestamps through mbuf fields.

A timestamp is only as good as its clock. PTP (IEEE 1588) synchronises the NIC's PHC to a grandmaster clock over the network (ptp4l from linuxptp), and phc2sys then steers the system clock from the PHC. With hardware timestamping on every hop and PTP-aware switches, sub-microsecond agreement is routine. Over a network that isn't PTP-aware, queueing jitter dominates. Regulators care too: under MiFID II (RTS 25), firms using high-frequency algorithmic trading must keep business clocks within 100 µs of UTC with 1 µs timestamp granularity.

The strongest measurements don't use the trading host's clock at all. Firms put passive optical taps on the links into and out of the box and timestamp both directions with an FPGA capture device on one disciplined clock. "Packet in" to "order out", as seen on the wire, is the only tick-to-trade number that can't flatter the software (week 46 builds tooling on exactly those captures).

Your exchange has two network paths that matter. Market data out is UDP multicast, published by the exchange and consumed by every client. Order entry in is TCP sessions from clients to the gateway. A real exchange also cares about the client side of the same links. A firm connecting to you bypasses on its side, and your publisher's jitter becomes its problem.

Write the memo as if a CTO will approve or reject spending on it. For each path, state the current measured latency and its tail, the budget from week 39's targets (draft them now if you haven't), which bypass option you'd use (none, Onload, ef_vi/TCPDirect, AF_XDP, DPDK), what it costs (NICs, cores burned polling, engineering time, lost tooling, vendor lock-in), how you'd measure the result, and how you'd fall back if the bypass stack misbehaves in production. "Don't bypass yet, and here's the number that would change my mind" is a legitimate conclusion.