Schema-Driven Binary Protocols

An SBE message is four regions, always in this order:

1. Message header, 8 bytes in the standard schema: blockLength: u16, templateId: u16, schemaId: u16, version: u16. templateId says which message this is, schemaId which schema it belongs to, and version which version of the schema the sender used. blockLength says how many bytes the sender's root block occupies.
2. Root block: the message's fixed-size fields at fixed offsets. No tags, no lengths, no delimiters. A field is wherever the schema says it is.
3. Repeating groups, each with a small dimension header (typically blockLength: u16 and numInGroup: u16; CME's MDP 3.0 uses a one-byte count) followed by numInGroup entries of blockLength bytes each. Groups can nest.
4. Variable-length data: length-prefixed byte strings, last.

Fields are little-endian by default (the schema can say otherwise), and SBE doesn't align anything unless the schema asks for it with explicit offsets. Optional fields don't have a presence bit. A designated null value of the field's type means "absent": the maximum for unsigned types, the minimum for signed ones.

The schema itself is XML. Here's a NewOrder in the shape you'll decode on Day 3 (the field ids are the FIX tag numbers, which is SBE convention):

<sbe:message name="NewOrder" id="1">
    <field name="clOrdId" id="11"  type="uint64"/>
    <field name="price"   id="44"  type="int64"/>
    <field name="qty"     id="38"  type="uint32"/>
    <field name="side"    id="54"  type="Side"/>
    <field name="minQty"  id="110" type="uint32" presence="optional" sinceVersion="2"/>
    <group name="parties" id="453" dimensionType="groupSizeEncoding">
        <field name="partyId" id="448" type="uint32"/>
        <field name="role"    id="452" type="uint8" presence="optional" sinceVersion="2"/>
    </group>
</sbe:message>

The design consequence that matters most: access is streaming and single-pass. Root fields are at constant offsets, so reading price is one load. A group's position, though, depends on the root block's length, and the second group's position depends on the first group's length. Decoders therefore walk the message front to back, and generated APIs make you read groups in order. In exchange for that discipline you get a decoder that's a few loads and no branches per field.

A code generator's first job is the layout: walk the fields in declaration order, place each one where the previous ended (or at an explicit offset, leaving padding), and record the root block's length. That's the same arithmetic you did by hand in week 11, but now it's done by a function that can be tested.

Alignment is a deliberate choice. SBE doesn't require it. On x86 an unaligned 8-byte load costs nothing extra unless it crosses a 64-byte cache-line boundary, and even then it costs only a few cycles. Crossing a 4 KiB page boundary is worse. Explicit offsets let you put 8-byte fields at multiples of 8 so a message never splits a hot field across lines, at the price of a few padding bytes. CME's schemas are packed, while many in-house protocols align. Make the choice once, in the schema, and let the layout function enforce it.

Evolution constrains the function. The extension rules boil down to one principle: a field's offset, type and meaning never change once published. It follows that:

• New fields go at the end of the root block or of a group entry, tagged with sinceVersion. The sequence of sinceVersion values along a block never decreases.
• Fields are never removed or reordered. You can deprecate one (stop populating it and send the null value), but its bytes stay.
• New groups go after existing groups, and new variable-length fields after existing ones.

Because the layout function sees every field's sinceVersion, it can compute the root block length for each version: the length a v0 encoder writes, the length a v1 encoder writes, and so on. That's exactly what a conformance test needs to construct "a message from an old client".

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Prim {
    U8,
    U16,
    U32,
    U64,
    I64,
    Char(usize), // fixed-length character array
}

#[derive(Clone, Copy, Debug)]
pub struct FieldSpec {
    pub name: &'static str,
    pub prim: Prim,
    pub since: u16,            // sinceVersion
    pub offset: Option<usize>, // explicit offset; padding before it is allowed
}

#[derive(Debug, PartialEq, Eq)]
pub enum SchemaError {
    Overlap(&'static str),      // explicit offset lands inside an earlier field
    VersionOrder(&'static str), // a field older than one before it
}

#[derive(Debug, PartialEq, Eq)]
pub struct Layout {
    pub offsets: Vec<(&'static str, usize)>,
    pub block_length: usize,
}

pub fn prim_size(p: Prim) -> usize {
    todo!()
}

/// Lay the fields out in declaration order. A field without an explicit
/// offset starts where the previous one ended; an explicit offset must not
/// overlap the previous field. sinceVersion must never decrease: new fields
/// are only ever appended. Report the first offending field.
pub fn layout(fields: &[FieldSpec]) -> Result<Layout, SchemaError> {
    todo!()
}

/// The root block length an encoder of `version` writes: the end of the
/// last field that exists in that version (0 if none). Schema errors
/// propagate.
pub fn block_length_at(fields: &[FieldSpec], version: u16) -> Result<usize, SchemaError> {
    todo!()
}

A decoder compiled against schema version S reads a message whose header says version V, the acting version. Two directions must both work:

• New decoder, old message (V < S): fields with sinceVersion > V don't exist in this message. Their accessors return the null value (or None in an idiomatic Rust API) without reading the bytes, because those bytes belong to whatever comes next.
• Old decoder, new message (V > S): the root block has fields the decoder has never heard of. It reads the fields it knows at their known offsets and then finds the groups at header + blockLength, the sender's length and not its own idea of the block size. The same holds inside every group: step through entries by the group's blockLength, not your own entry size.

Then be strict about malformed input. If the acting version says a field exists but blockLength is too short to contain it, the message is corrupt, and decoding garbage would be worse than rejecting it. A buffer that ends before the header, a root block or a group entry ends is Short. A decoder on an exchange's order-entry path faces hostile and buggy input all day, so it must reject cleanly and never panic.

Two smaller traps:

• Null sentinels in the wire format, Option in the API. u32::MAX meaning absent is a wire convention. Convert at the boundary, so min_qty: Option<u32> can't be mistaken for a 4-billion-lot order.
• Enums grow too. Adding an enum value (a new TimeInForce) is a schema change that old decoders see as an unknown byte. Decode into an enum with an explicit Unknown(u8) variant, or reject, and never transmute a byte into a Rust enum: an out-of-range discriminant is undefined behaviour.

pub const HEADER_LEN: usize = 8;
pub const TEMPLATE_NEW_ORDER: u16 = 1;
pub const SCHEMA_ID: u16 = 7;

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Party {
    pub id: u32,
    pub role: Option<u8>,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct NewOrder {
    pub cl_ord_id: u64,
    pub price: i64,
    pub qty: u32,
    pub side: u8,
    pub min_qty: Option<u32>,
    pub parties: Vec<Party>,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum DecodeError {
    Short,              // the buffer ends before something it must contain
    WrongTemplate(u16),
    WrongSchema(u16),
    BlockTooShort,      // a block too small for the fields its version promises
}

/// Schema id 7, NewOrder template 1. Version 2 is current.
///
///   header : blockLength u16 | templateId u16 | schemaId u16 | version u16
///   root   : clOrdId u64 @0 | price i64 @8 | qty u32 @16 | side u8 @20  (since v1)
///            minQty u32 @21                          (since v2, null = u32::MAX)
///   group  : blockLength u16 | numInGroup u16, then numInGroup entries,
///            each blockLength bytes
///   entry  : partyId u32 @0 (since v1) | role u8 @4 (since v2, null = 255)
///
/// All integers little-endian. Rules:
///   - read the whole header first (Short if < 8 bytes), then check
///     templateId, then schemaId
///   - root blockLength < 21, or < 25 when version >= 2 -> BlockTooShort;
///     entry blockLength < 4, or < 5 when version >= 2 -> BlockTooShort
///   - a field exists iff version >= its sinceVersion
///   - the group starts at HEADER_LEN + blockLength (the sender's!)
///   - entries are blockLength apart; a partial entry -> Short
///   - null sentinels decode to None
pub fn decode(buf: &[u8]) -> Result<NewOrder, DecodeError> {
    todo!()
}

The drill decodes into an owned NewOrder with a Vec so the tests are easy to read. Your capstone decoder must not do that, because the allocation alone would blow the latency target. Tomorrow is about the zero-copy shape.

What should a generated decoder look like? The SBE reference implementation answers with a flyweight: a thin view over the bytes that knows the acting version and the block length and decodes fields on access. In Rust:

pub struct NewOrderDecoder<'a> {
    buf: &'a [u8],     // validated once: header + root block fit
    version: u16,
    block_len: u16,
}

impl<'a> NewOrderDecoder<'a> {
    pub fn wrap(buf: &'a [u8]) -> Result<Self, DecodeError> { /* one length check, header checks */ }

    #[inline]
    pub fn price(&self) -> i64 {
        i64::from_le_bytes(self.buf[HDR + 8..HDR + 16].try_into().unwrap())
    }

    #[inline]
    pub fn min_qty(&self) -> Option<u32> {
        if self.version < 2 { return None; }
        match u32::from_le_bytes(self.buf[HDR + 21..HDR + 25].try_into().unwrap()) {
            u32::MAX => None,
            v => Some(v),
        }
    }

    pub fn parties(&self) -> PartiesIter<'a> { /* starts at HDR + block_len */ }
}

Each accessor is a load from a constant offset. Validate the lengths once in wrap (header, root block, and if you can, the whole message including groups), and per-field slicing becomes a bounds check the optimiser can often remove. If it can't, generated get_unchecked behind a validated constructor is the one place unsafe earns its keep. Check the assembly (week 37 is all about this).

Three ways to produce that code:

• build.rs + a schema file. The build script parses the schema (XML, TOML, or your own format), writes Rust into OUT_DIR, and the crate pulls it in with include!(concat!(env!("OUT_DIR"), "/messages.rs")). It's the most flexible: the same parse can emit docs, test vectors or a C header for clients. Remember cargo:rerun-if-changed=schema/…, or edits won't trigger rebuilds.
• macro_rules!. The schema becomes Rust tokens: message! { NewOrder = 1 { cl_ord_id: u64, price: i64, … } }. Nothing to parse, and offsets can come from const fn arithmetic over the field list. Error messages and versioning metadata get awkward, though, and nobody else can read the schema.
• A derive / procedural macro. You annotate a Rust struct and generate the codec. It reads naturally, but compiles slowest, and the struct definition quietly becomes the wire contract.

For an exchange whose clients are written in other languages, a schema file that isn't Rust wins: you hand it to clients. macro_rules! is a fine way to start, and the spine allows either.

A codec is correct when its bytes are right, not when encode-then-decode round-trips. A buggy encoder paired with the matching buggy decoder round-trips perfectly. A conformance suite has four layers:

1. Golden vectors. Checked-in hex files, one per message and version, produced once and reviewed by a human, ideally cross-checked against an independent implementation such as the reference SBE tool. Your encoder must produce them byte for byte, and your decoder must decode them to known values.
2. The version matrix. For each pair of (encoder version, decoder version) you support, encode with one and decode with the other. Include a synthetic future version with extra root and group bytes, as in the drill, so old-decoder-new-message is tested before a future version exists.
3. Property tests. For random valid messages (week 12's generators), decode(encode(m)) == m, and the encoded length equals what the layout predicts.
4. Fuzzing. Feed the decoder arbitrary bytes (cargo fuzz) and require that it never panics, never reads out of bounds and never loops. Your gateway parses bytes from the internet, so this isn't optional.

It's also worth knowing what you didn't choose, because interviewers ask:

Protobuf's varints and tag-per-field parsing cost tens to hundreds of nanoseconds per message. That's fine for configuration and admin RPCs, and wrong for a gateway's hot path. SBE wins there by refusing flexibility: no optional reordering, no sparse fields, fixed shapes.

Before you write the generator, decide:

• Schema format. XML that mirrors SBE (familiar to anyone who has seen CME's schemas), a TOML file, or a macro_rules! DSL. If clients will ever see it, avoid the DSL.
• What gets generated. At least: constants (template ids, offsets, block lengths per version), a decoder flyweight per message with Option accessors for versioned fields, group iterators, an encoder that writes into &mut [u8] and returns the length, and a Message enum to dispatch on template id.
• Migration of week 11. Your existing messages should be expressible in the schema and produce identical bytes as version 1. Make that the first golden test. (The repo chose not to. Its schema messages have their own 4-byte header and packed fields, and week 11's codec stays as it is beside them. See the capstone.)