HFT-52 · Weeks 31 and 33 · Kernel networking and kernel bypass

One packet, wire to wire

An order arrives at the gateway and an acknowledgement goes back. Follow the packet through the NIC, the kernel and your code, then take the kernel out of the path and see which costs disappear and which are physics.

Order in, ack out, on one host
Where the packet is, layer by layer NIC & wireKernelUser spaceSkipped
Round trip to scale
Receive pathWire → appApp → wireRound trip

The numbers are illustrative orders of magnitude for a tuned server (interrupt coalescing off, pinned cores, C-states limited) and match the guides: the week 33 drill's receive path (wire + NIC + DMA 800 ns, IRQ + NAPI 1.5 µs, IP + socket 1.2 µs, wakeup 2.5 µs, recv copy 400 ns, so 6.4 µs through the kernel and 0.95 µs with bypass) and the kernel-vs-bypass budget (send through the stack 3 µs, bypass send 800 ns, application 600 ns). Busy-poll in the kernel means SO_BUSY_POLL: the thread never sleeps and runs the NAPI poll itself, so the interrupt and wakeup go, but the protocol stack and the copy stay. Bypass means ef_vi, DPDK or AF_XDP zero-copy with a user-space stack: your loop polls the descriptor ring. Bypass with interrupts means sleeping on the NIC's event queue, which brings the wakeup back. Kernel sockets copy small messages either way; MSG_ZEROCOPY only pays above about 10 KB. Nagle shows week 31's write-write-read: the reply is written as header then body, and Linux's delayed ACK holds the body at least 40 ms. A user-space TCP stack such as Onload implements Nagle too.