The memory hierarchy, with numbers you should be able to recite (they vary by part; the ratios are what matter):

• Registers: zero extra latency.
• L1d: 32–48 KB per core, ~4–5 cycles (about 1 ns).
• L2: 256 KB–2 MB per core, ~12–16 cycles (3–5 ns).
• L3: shared, several MB to hundreds of MB, ~40–70 cycles (10–20 ns), more on big mesh parts.
• Local DRAM: ~80–100 ns.
• Remote NUMA DRAM: often 130–150 ns or more.

Everything between the core and DRAM moves in lines: 64 bytes on current Intel and AMD x86 cores and most ARM server cores, 128 bytes on Apple's M-series. Read one u8 and the whole line comes in; write one u8 and the core must own the whole line exclusively first.

The rules you need:

• Every type has a size and an alignment (a power of two). A value's address is always a multiple of its alignment, and its size is always a multiple of its alignment — that is what lets [T; N] place element i at i * size_of::<T>().
• repr(C) lays fields out in declaration order. Each field goes at the next offset that is a multiple of its alignment, the struct's alignment is the largest field alignment, and the size is rounded up to it. The gaps are padding: real bytes that occupy cache lines and carry no information.
• repr(Rust), the default, makes no promises. Today rustc reorders fields (roughly largest alignment first) to remove interior padding and exploits niches: Option<NonZeroU64>, Option<Box<T>> and Option<&T> are 8 bytes, while Option<u64> is 16 because every u64 bit pattern is a valid value.
• #[repr(align(N))] raises the alignment to at least N (and so the size to a multiple of N). Use it to start a value on a line boundary.
• #[repr(packed)] removes padding by allowing misaligned fields. Rust will not let you take a reference to a misaligned packed field; you copy it out or use read_unaligned. It is for matching wire formats (week 11), not for speed.

A typical first-draft order:

#[repr(C)]
struct NaiveOrder {
    side: u8,     // offset 0, then 7 bytes of padding
    id: u64,      // 8
    is_ioc: bool, // 16, then 7 bytes of padding
    price: i64,   // 24
    qty: u32,     // 32
    owner: u16,   // 36
    flags: u8,    // 38, then 1 byte of tail padding
}                 // size 40, align 8 — 15 of 40 bytes are padding

Sorted by alignment the same fields fit in 32 bytes: two orders per line instead of one and three-fifths. That is the whole trick, and repr(Rust) usually does it for you; with repr(C) (or any layout you must control) it is your job. You can pin the result in a test with const _: () = assert!(std::mem::size_of::<Order>() == 32); so a later field addition that blows the budget fails the build instead of the benchmark.

/// A struct layout as the compiler computes it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Layout {
    pub size: usize,
    pub align: usize,
    /// Byte offset of each field, in declaration order.
    pub offsets: Vec<usize>,
}

/// Round `n` up to the next multiple of `align` (a power of two).
pub fn align_up(n: usize, align: usize) -> usize {
    // Hint: one add and one mask. No division, no branches.
    todo!()
}

/// Lay out `fields` exactly like `#[repr(C)]`. Each field is `(size, align)`.
///   - fields stay in declaration order
///   - each field starts at the next offset that is a multiple of its align
///   - the struct's align is the largest field align (1 if there are no fields)
///   - the size is rounded up to the struct's align (so arrays stay aligned)
pub fn repr_c_layout(fields: &[(usize, usize)]) -> Layout {
    todo!()
}

/// The smallest size any ordering of `fields` can achieve — what you get by
/// reordering by hand, and what rustc's default `repr(Rust)` usually finds.
pub fn min_reordered_size(fields: &[(usize, usize)]) -> usize {
    todo!()
}

/// How many 64-byte cache lines does an object of `len` bytes that starts at
/// byte address `addr` touch? A zero-length object touches none.
pub fn cache_lines_touched(addr: usize, len: usize) -> usize {
    todo!()
}

The harness checks your repr_c_layout against size_of, align_of and offset_of! on real #[repr(C)] structs, so you are predicting the compiler, not a made-up model. For min_reordered_size, convince yourself why sorting by alignment, largest first, can never leave interior padding when every alignment is a power of two.

Each core has private L1 and L2 caches, so the hardware runs a coherence protocol to keep copies consistent: MESI and its variants (Intel uses MESIF, AMD MOESI). A line is Modified, Exclusive, Shared or Invalid in each cache. The rule that matters: to write, a core must hold the line exclusively. If another core has a copy, the writer sends a read-for-ownership (RFO), the other copy is invalidated, and if it was Modified the data travels core to core.

Now put two counters next to each other in one struct, and give each one its own thread:

struct Stats {
    orders_in: AtomicU64,  // written only by the gateway thread
    fills_out: AtomicU64,  // written only by the engine thread
}

No data is shared, yet every increment by one thread invalidates the line in the other core, and the next increment there has to pull it back. That is false sharing: the line ping-pongs between cores at tens of nanoseconds per transfer, and both threads run at the speed of the interconnect instead of the speed of L1. With true sharing (both threads updating the same counter) you get the same traffic and also need a locked read-modify-write; with false sharing you get the traffic for nothing.

Things that make it worse or confusing in practice:

• Atomic RMWs are not free even without contention. A lock xadd on an L1-resident line is roughly 20 cycles on recent Intel cores, versus about one for a plain store. If a counter has exactly one writer, a Relaxed load plus store avoids the locked instruction entirely.
• The adjacent-line prefetcher. Intel's L2 spatial prefetcher pulls in 128-byte-aligned pairs of lines, so two hot variables 64 bytes apart can still interfere. That is why crossbeam pads to 128 bytes on x86-64.
• SMT siblings share L1. Two hyperthreads of one physical core do not ping-pong a line, so a false-sharing benchmark pinned to sibling hyperthreads shows nothing. Pin to separate physical cores.

The pattern is one line of attribute and a Deref:

#[repr(align(128))]
pub struct CachePadded<T> { value: T }

Because size_of is always a multiple of align_of, every CachePadded<T> occupies at least 128 bytes, so neighbours in a Vec or a struct cannot share a line. Vec<T> allocates with T's alignment, so the first element starts on a boundary too. The price is memory: 8 bytes of counter in 128 bytes of space. Spend it on the handful of variables that different threads write; never on data that is read-mostly, where sharing a line is free.

The other design move is one writer per line: rather than one global counter every thread bumps, give each thread its own padded slot and sum them when someone asks. Writes become core-local; the reader pays a few cache misses once per scrape instead of every writer paying on every increment. The per-thread histograms and counters of week 19 use exactly this shape.

use std::ops::{Deref, DerefMut};
use std::sync::atomic::{AtomicU64, Ordering};

/// Wraps a `T` so that no two `CachePadded` values ever share a cache line.
// TODO: one attribute on this struct does all the work. Which, and why 64
// (or 128 — read crossbeam's comment before you choose)?
pub struct CachePadded<T> {
    value: T,
}

impl<T> CachePadded<T> {
    pub fn new(value: T) -> Self {
        todo!()
    }

    pub fn into_inner(self) -> T {
        todo!()
    }
}

impl<T> Deref for CachePadded<T> {
    type Target = T;
    fn deref(&self) -> &T {
        todo!()
    }
}

impl<T> DerefMut for CachePadded<T> {
    fn deref_mut(&mut self) -> &mut T {
        todo!()
    }
}

/// One counter per thread. Slot `i` is written only by thread `i`; any
/// thread may read the total.
pub struct PaddedCounters {
    slots: Vec<CachePadded<AtomicU64>>,
}

impl PaddedCounters {
    pub fn new(n: usize) -> Self {
        todo!()
    }

    /// Add one to slot `i`. Only the owning thread calls this.
    pub fn incr(&self, i: usize) {
        todo!()
    }

    pub fn get(&self, i: usize) -> u64 {
        todo!()
    }

    /// Sum of every slot (a metrics scraper calls this).
    pub fn total(&self) -> u64 {
        todo!()
    }

    /// Address of slot `i`'s counter — the harness checks the spacing.
    pub fn slot_addr(&self, i: usize) -> usize {
        todo!()
    }
}

The order of fields inside one struct is half the story. The other half is how you arrange many of them.

• Array of structs (AoS): Vec<Level> where each Level holds price, total quantity, order count and queue pointers. A loop that touches every field of each element uses every byte it pulls in.
• Struct of arrays (SoA): prices: Vec<i64>, qtys: Vec<u64>, counts: Vec<u32>, …. A loop that reads only quantities — "how much is available within 10 ticks of the touch?" — streams through 8 bytes per level instead of 32 or more, gets 8 levels per line instead of 2, and is trivially vectorizable (week 29).

Neither wins in general; the access pattern decides. Matching touches one level and one order at a time and uses all of their hot fields, so AoS with a carefully packed hot struct is right there. Depth aggregation and market-data snapshots scan one field across many levels, so a SoA view (or a dense ladder of quantities, week 28) is right there.

The intermediate move is a hot/cold split. The fields the matching loop needs — price, remaining quantity, id, queue links — go in a compact hot struct; the client order id string, account, entry timestamp and other reporting data go in a parallel cold table indexed the same way. The hot struct then fits two to four orders per line, and the cold data is only touched when you emit an execution report.

#[repr(C)]
struct OrderHot {          // 32 bytes: two per line
    price: i64,
    qty: u64,
    id: u64,
    next: u32,
    prev: u32,
}

struct OrderCold {         // touched on reports and audits, not on matching
    client_order_id: [u8; 20],
    account: u32,
    entered_ns: u64,
}

Padding everything is not a strategy: it bloats memory, spreads hot data across more lines and can make a cache-friendly design worse. Pad the variables different threads write frequently, verify with perf c2c or a before/after benchmark, and leave the rest alone.