io_uring & Batched I/O

epoll is a readiness API. You register interest once with epoll_ctl, then each loop iteration asks "who is ready?" and does the I/O yourself:

// Shape of an edge-triggered epoll loop (pseudo-Rust over libc).
loop {
    let n = epoll_wait(epfd, &mut events, -1);       // syscall 1
    for ev in &events[..n] {
        let s = &mut sessions[ev.token];
        loop {
            match recv(s.fd, &mut s.rx) {            // syscall 2..k
                Ok(0) => { s.close(); break }
                Ok(n) => s.on_bytes(n),
                Err(WouldBlock) => break,            // edge-triggered: drain to EAGAIN
                Err(e) => { s.fail(e); break }
            }
        }
    }
    flush_pending_sends();                           // more syscalls
}

Count the crossings for one wakeup with 10 ready sockets under edge-triggered mode: one epoll_wait, ten successful recvs, ten more recvs that return EAGAIN to prove each socket is drained, plus the sends. Level-triggered mode lets you skip the EAGAIN probe, but then the next epoll_wait reports the socket again if you left bytes behind.

io_uring is a completion API, the model Windows IOCP has used for decades. You hand the kernel the operation and the buffer up front: "receive into this buffer on this socket and tell me when it's done". The kernel queues the operation, completes it when data arrives, and posts a completion record carrying a 64-bit tag you chose (user_data). Your loop stops asking "who is ready?" and starts asking "what finished?":

loop {
    // 0 or 1 syscalls: submit everything queued since last time and wait for >= 1 completion
    ring.submit_and_wait(1)?;
    for cqe in ring.completion() {
        let (session, op) = decode_tag(cqe.user_data());
        match op {
            Op::Recv => { sessions[session].on_bytes(cqe.result()); queue_recv(session) }
            Op::Send => sessions[session].on_sent(cqe.result()),
        }
    }
}

Three consequences follow, and the rest of the week builds on them:

1. The buffer belongs to the kernel while the operation is in flight. You can't reuse, move or free it until its completion arrives. This is why Rust io_uring runtimes (tokio-uring, glommio, monoio) take buffers by value and hand them back with the result. A borrowed &mut [u8] can't express "the kernel may write here after this future is dropped".
2. Completions can arrive in any order. Two recvs on two sockets complete when their data arrives, not in submission order. user_data is your only correlation key, so encode the session and operation in it. (If you need ordering between operations, IOSQE_IO_LINK chains them.)
3. Syscalls become a batching knob. Submission and reaping are memory operations on shared rings. When you actually enter the kernel is now your decision, not a side effect of every I/O.

io_uring_setup(2) creates the instance and returns a file descriptor. You mmap three regions from it: the submission queue ring, the array of SQEs it points into, and the completion queue ring (on newer kernels the two rings share one mapping). Each ring is a power-of-two array plus a head and a tail:

The indices are free-running 32-bit counters. The slot is index & ring_mask, the fill level is tail - head with wrapping arithmetic, and "full" means tail - head == entries. Free-running counters let you tell full from empty without sacrificing a slot, and masking is a single AND because the size is a power of two. io_uring_setup rounds your requested size up to a power of two, and by default the CQ gets twice as many entries as the SQ because more operations can be in flight than you submit at once.

The memory ordering is exactly your week 5 SPSC ring. When you publish SQEs, you fill in the entries and then store the SQ tail with release semantics. The kernel reads tail with acquire semantics, so it sees fully written entries. On the completion side you load the CQ tail with acquire, read the CQEs, then store the CQ head with release so the kernel doesn't overwrite an entry you're still reading. liburing and the io-uring crate do this for you. Write it once by hand anyway, because one day you'll debug it.

// The io-uring crate (tokio-rs/io-uring): one read, submitted and reaped.
let mut ring = IoUring::new(8)?;
let read = opcode::Read::new(types::Fd(file.as_raw_fd()), buf.as_mut_ptr(), buf.len() as _)
    .build()
    .user_data(0x42);
// unsafe: the kernel will write into `buf` later. You promise it outlives the op.
unsafe { ring.submission().push(&read).expect("SQ full") };
ring.submit_and_wait(1)?;                 // one io_uring_enter
let cqe = ring.completion().next().expect("CQ empty");
assert_eq!(cqe.user_data(), 0x42);

What happens when the CQ is full? Older kernels dropped the completion and bumped an overflow counter. Since Linux 5.5 (IORING_FEAT_NODROP) the kernel keeps overflowed completions on an internal list and may refuse new submissions with EBUSY until you reap. Either way, a full CQ means your design is wrong: size the CQ for the maximum number of operations you allow in flight, and reap before you submit more.

You'll now build the model: a generic free-running ring, and an Uring that pairs an SQ with a CQ and counts how many times you "enter the kernel". The kernel is played by a closure. Our model is stricter than the real thing in one way: submit stops consuming SQEs when the CQ has no room, so nothing is ever dropped or buffered behind your back.

/// One submission queue entry. The real SQE is 64 bytes; we keep the
/// fields that matter for the model.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct Sqe {
    pub opcode: u8,     // 0 = NOP, 1 = READ, 2 = WRITE ...
    pub fd: i32,
    pub len: u32,
    pub user_data: u64, // echoed back in the CQE: your only correlation key
}

/// One completion queue entry (16 bytes in the real ABI).
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct Cqe {
    pub user_data: u64,
    pub res: i32, // bytes transferred, or -errno
}

/// Single-producer / single-consumer ring with free-running u32 indices,
/// like io_uring's SQ and CQ: the slot is `index & mask`, the fill level
/// is `tail.wrapping_sub(head)`.
pub struct Ring<T: Copy + Default> {
    slots: Vec<T>,
    mask: u32,
    head: u32, // consumer's next read
    tail: u32, // producer's next write
}

impl<T: Copy + Default> Ring<T> {
    /// Capacity rounds up to the next power of two (io_uring_setup does too).
    pub fn new(entries: u32) -> Self {
        Self::with_start(entries, 0)
    }

    /// Start both indices at `start` so tests can exercise u32 wrap-around.
    pub fn with_start(entries: u32, start: u32) -> Self {
        todo!()
    }

    pub fn capacity(&self) -> u32 {
        todo!()
    }

    pub fn len(&self) -> u32 {
        todo!()
    }

    /// Producer side. Returns false when the ring is full.
    pub fn push(&mut self, v: T) -> bool {
        todo!()
    }

    /// Consumer side.
    pub fn pop(&mut self) -> Option<T> {
        todo!()
    }
}

pub struct Uring {
    sq: Ring<Sqe>,
    cq: Ring<Cqe>,
    enters: u32,
}

impl Uring {
    /// SQ gets `entries` (rounded up to a power of two); the CQ gets twice
    /// that, which is the kernel's default.
    pub fn new(entries: u32) -> Self {
        todo!()
    }

    /// Queue an SQE in shared memory. No syscall. False if the SQ is full.
    pub fn prep(&mut self, sqe: Sqe) -> bool {
        todo!()
    }

    /// One io_uring_enter(2): exactly one syscall however many SQEs it
    /// consumes. `exec` plays the kernel and returns `res`. Consume SQEs in
    /// order while the CQ has room for the completion. Returns the number
    /// of SQEs consumed.
    pub fn submit<F: FnMut(&Sqe) -> i32>(&mut self, mut exec: F) -> u32 {
        todo!()
    }

    /// Drain every available CQE into `out`; returns how many were reaped.
    pub fn reap(&mut self, out: &mut Vec<Cqe>) -> usize {
        todo!()
    }

    pub fn enters(&self) -> u32 {
        self.enters
    }

    pub fn sq_pending(&self) -> u32 {
        self.sq.len()
    }
}

Batching cuts the number of kernel entries. The next set of features cuts the work the kernel repeats for every operation.

Registered files (IORING_REGISTER_FILES, then IOSQE_FIXED_FILE on each SQE). A normal operation resolves the fd to a struct file and takes and drops a reference count on it, which is an atomic increment and decrement on a cache line other threads may touch. With a registered file table the SQE carries an index into a table the kernel already holds references for.

Registered buffers (IORING_REGISTER_BUFFERS, with READ_FIXED/WRITE_FIXED). Normally the kernel must pin your user pages for the duration of each I/O and unpin them afterwards. Registering a buffer region pins it once. This matters most for storage I/O such as the journal. For network receives the next feature usually matters more.

Provided buffer rings (Linux 5.19). If you post a recv with its own 4 KiB buffer on each of 2,000 mostly idle sessions, 8 MB of memory sits pinned waiting for data that rarely comes. With a provided buffer ring you give the kernel a pool, and it picks a buffer only when data actually arrives, reporting which one in the CQE flags. Memory then scales with active traffic, not open sockets.

Multishot operations. A multishot accept (5.19) or multishot recv (6.0) is one SQE that keeps producing CQEs, each flagged IORING_CQE_F_MORE while the request stays armed. For a recv loop that means one submission for the lifetime of the connection instead of one per message.

SQPOLL (IORING_SETUP_SQPOLL). A kernel thread polls your SQ, so submitting is just writing memory. In steady state your loop makes no syscalls at all: it writes SQEs, advances the tail and spins on the CQ. The costs are a core burned by the poller, which you should pin with IORING_SETUP_SQ_AFF next to your thread, and an idle timeout (sq_thread_idle) after which the thread sleeps and sets IORING_SQ_NEED_WAKEUP. You then have to call io_uring_enter with IORING_ENTER_SQ_WAKEUP to restart it. Very old kernels required elevated privileges for SQPOLL. Check what yours requires.

Newer kernels (6.0/6.1) also added IORING_SETUP_SINGLE_ISSUER and IORING_SETUP_DEFER_TASKRUN, which defer completion processing until your thread enters the kernel instead of interrupting it at arbitrary moments. For a single-threaded, pinned event loop (the shape you want) they cut background interference.

Every wakeup has a fixed cost: waiting, entering the kernel, the scheduler, cache refills. It also has a per-message cost: decode, risk check, hand-off to the engine. With F fixed and c per message, a wakeup that handles B messages takes F + B·c, and one loop sustains B / (F + B·c) messages per unit time. As B grows, throughput approaches 1/c and the fixed cost is amortised away.

Latency moves the other way. There are two ways to batch, and they behave very differently:

• Fixed batching waits until B messages have accumulated, or until a timeout fires. At low load the first message in a batch waits for strangers to arrive. This is Nagle's algorithm again, and it's how you lose milliseconds in a system that should take microseconds.
• Natural batching, which Martin Thompson calls smart batching, never waits. When the loop wakes it takes whatever is already queued, up to a cap, processes it, flushes all replies with one submission and goes back. At low load batches are size 1 and latency is minimal. Under load, messages pile up while the loop is busy, so batches grow exactly when amortisation is needed. The LMAX Disruptor's batch-consumer design rests on this effect.

With natural batching capped at B, the worst case for one message is: it arrives just after a wakeup began, waits for that batch to finish, then rides as part of the next full batch whose replies flush at the end. That's 2·(F + B·c). Two constraints now define a window of acceptable caps. The cap must be large enough that capacity at the cap covers the offered rate, or the queue grows without bound (Little's law: L = λ·W, so if arrivals outpace service, W explodes). It must also be small enough that the worst case fits the budget. If the window is empty, no batch size saves you, and you have to shrink F (SQPOLL, busy polling, kernel bypass) or shrink c.

In the drill you'll turn that model into the numbers you'd put in a design review. The figures in the tests are illustrative but realistic in proportion: about 1.5 µs of fixed cost per wakeup when it involves a syscall and a sleep/wake, about 0.3 µs when nothing sleeps, about 200 ns of per-message work, and about 700 ns extra per message when every message needs its own recv.

/// Time to process one wakeup that handles `batch` messages: fixed
/// per-wakeup overhead plus per-message work.
pub fn service_ns(fixed_ns: u64, per_msg_ns: u64, batch: u64) -> u64 {
    todo!()
}

/// Messages per second one loop sustains if every wakeup carries `batch`
/// messages. Round down. (Use u128 for the intermediate product.)
pub fn capacity_per_sec(fixed_ns: u64, per_msg_ns: u64, batch: u64) -> u64 {
    todo!()
}

/// Smallest batch (at least 1) whose capacity is at least `rate_per_sec`.
/// None if no batch size keeps up (per-message work alone saturates the core).
pub fn min_batch_for_rate(fixed_ns: u64, per_msg_ns: u64, rate_per_sec: u64) -> Option<u64> {
    todo!()
}

/// Worst-case latency with natural batching capped at `batch`: a message that
/// lands just after a wakeup started waits for that batch, then rides in the
/// next full batch, and its reply is flushed when that batch ends.
pub fn worst_latency_ns(fixed_ns: u64, per_msg_ns: u64, batch: u64) -> u64 {
    todo!()
}

/// The batch caps you may configure: from the smallest that keeps up with
/// the rate to the largest whose worst case fits the budget. None if empty.
pub fn batch_window(fixed_ns: u64, per_msg_ns: u64, rate_per_sec: u64, budget_ns: u64) -> Option<(u64, u64)> {
    todo!()
}

Look at the last two cases once they pass. Moving from a recv per message to completions that are already waiting, or from a sleeping loop to one that polls, doesn't just make things a bit faster. It turns a configuration that cannot meet the budget at any batch size into one with plenty of room. That's the argument you'll make for io_uring in the gateway, and next week for kernel bypass on the market-data path.

There's no universal winner. Here's how the options compare for the three I/O paths in your exchange:

That points to a split:

• Order-entry gateway (thousands of TCP sessions, bursty): io_uring with multishot recv, provided buffers and natural batching of replies. The per-message syscall savings are real and the batching window is wide.
• Market-data hot path (one or two multicast sockets, every nanosecond counts): a pinned thread busy-polling a non-blocking socket beats both models on latency, and kernel bypass beats that (week 33). Batching here only helps under bursts.
• Journal (sequential writes plus durability): io_uring fits naturally. A write linked to an fsync with IOSQE_IO_LINK expresses "durable, in order" as a single submission, and registered buffers plus O_DIRECT remove the per-I/O page pinning.

Then there's the operational side. io_uring gives the kernel a large new attack surface. In 2023 Google's security team reported that most of the kernel exploits submitted to its kCTF vulnerability reward program targeted io_uring, and it restricted io_uring on ChromeOS, for Android apps and on its production servers. Linux 6.6 added a kernel.io_uring_disabled sysctl, and some container runtimes' default seccomp profiles block the io_uring syscalls. Before you design around io_uring, confirm that the boxes you'll deploy on (week 47) allow it, and keep the epoll backend as a working fallback rather than dead code.

The gateway from week 13 reads and writes sockets through Tokio. This week you build an I/O layer that speaks in submissions and completions, implement it twice, and run the same session code on both. That session code must not know which backend it's running on. In the repo it is a framing-layer echo server, echo_loop, and the gateway itself stays on Tokio, so the comparison measures the I/O model and nothing else.

The interface. Model it on io_uring even for the epoll backend. Completion is the more general model, because you can emulate completions on top of readiness but not the other way round. The repo's shape (gateway::io):

pub enum Submission {
    Accept { user_data: u64 },
    Read { conn: ConnId, buf: Vec<u8>, user_data: u64 },  // the backend owns buf until it completes
    Write { conn: ConnId, buf: Vec<u8>, user_data: u64 },
    Close { conn: ConnId, user_data: u64 },
}
pub struct Completion { pub user_data: u64, pub result: i64, pub buf: Option<Vec<u8>> } // result < 0: -errno

pub trait IoBackend {
    fn register(&mut self, stream: TcpStream) -> io::Result<ConnId>;
    fn listen(&mut self, listener: TcpListener) -> io::Result<()>;
    fn push(&mut self, sub: Submission) -> Result<(), Submission>; // no syscall; handed back if full
    fn submit_and_wait(&mut self, min_complete: usize, timeout: Option<Duration>)
        -> io::Result<usize>;                                     // one wakeup
    fn drain(&mut self, out: &mut Vec<Completion>) -> usize;     // no syscall
    fn stats(&self) -> IoStats;                                   // wakeups, syscalls, ...
    fn name(&self) -> &'static str;
}

The epoll backend turns readiness into completions: submit_and_wait tries each pending read and write itself, and when one would block it waits in epoll_wait for the socket and retries, appending completions as operations finish. In the repo that is ReadinessBackend<P>: EpollBackend on Linux, and PollBackend, a portable fallback that rescans every socket. The ring backend can be a simulated io_uring. The repo's RingBackend is the SQPOLL model: the SQ and the CQ are the SPSC rings from weeks 1 and 5, shared with a "kernel" thread that performs the operations with its own poller, so the caller enters the (simulated) kernel only to wake that thread or to block for completions. That keeps the capstone runnable on machines and containers where io_uring isn't available. If you're on Linux 5.19 or later, put a real backend behind a cargo feature using the io-uring crate in your fork; the repo stops at the simulation because Tokio is its only external dependency. A real backend is the version worth talking about in an interview.

The loop. Natural batching: wait for at least one completion, drain all available completions, run the session logic for each, push every resulting send and the replacement recvs, then submit once. Replies to the same session within one batch should coalesce into one send (the writev lesson from week 31). Count submit_and_wait calls and messages, and report messages per wakeup next to throughput. That ratio is the whole story of this week. The repo's echo_loop(io, conns, &EchoOptions) is this loop. EchoOptions::max_batch caps the completions handled per wakeup (usize::MAX for natural batching, 1 for one at a time), and the returned EchoStats has per_wakeup() and msgs_per_sec().