Taking protobuf out of a 1,000-crate Rust monorepo

Ryle · Aug 12, 2026

One of my side projects is a Rust monorepo that's grown to just under a thousand crates. For most of its life, every stored record and most service APIs were defined in protobuf. At the peak that was about 500 .proto files, roughly 75 build.rs scripts running prost and tonic code generation, and around 30 gRPC services. Then over about a week, coding agents working through a numbered list of migration steps converted all of it to plain Rust structs with serde derives. Data goes over the wire and into row storage as CBOR, anything hashed goes through a canonical CBOR encoder, and cold storage is an Arrow projection of the same structs, written to Lance files. Protobuf is gone from the first-party code, and a CI gate keeps it out.

I'd do it again for this repo, but a few things got worse and some bugs took a while to show up, so here are both.

Why protobuf went

The main reason was hashing. The repo is built around an append-only store where each record is identified by a hash of its content, and some records are also signed. Protobuf has no canonical encoding, and the docs say so. Field order, map entry order, whether default values get written and what happens to unknown fields are all left to the implementation. Hashing message.encode_to_vec() works as long as every writer runs the same prost version with the same schema. I couldn't promise that across Rust services, a WASM build and a TypeScript client.

So the hash moved first. About two months before the protobuf removal, I added a small canonical CBOR crate and switched record hashes from prost bytes to canonical CBOR of the same structs' serde form. Protobuf stayed on as the wire format. Once that was done, protobuf wasn't giving me much beyond code generation, and code generation was the second problem.

In a pilot measurement, one prost-generated schema crate pulled in nearly 90 crates and took about two minutes to build cold. The same type written as a plain serde struct needed about 10 crates and well under 20 seconds. Every build server and CI image needed protoc. Each type also existed three times: in the .proto, in generated code under OUT_DIR that nobody reviews, and usually in a hand-written impl module on top of that. Code review saw the .proto and the wrapper but never the code the compiler actually built.

The usual argument for keeping protobuf is cross-language clients, so I counted. Outside Rust there was one small Python gRPC server, a few Python eval scripts that used generated _pb2 modules, and the browser UI, which decoded with protobufjs. All of them lived in the same repo, and nobody outside it depended on the schema, so they could all change along with the Rust side.

What replaced it

A schema is now an ordinary struct in a small schema crate that depends on serde and nothing else:

#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
#[serde(default)]
pub struct PrRecord {
    pub id: String,
    pub repo: String,
    pub number: i32,
    pub created_at_ms: i64,
    // added after release, so it's Option forever
    pub merged_at_ms: Option<i64>,
}

Most of the converted types look like this, because the bulk conversion kept prost's field shapes. Proto3 scalars became plain fields, and #[serde(default)] on the struct keeps proto3's "missing means zero" behavior. Fields that were optional or nested messages in the proto became Option. In the converted modules, about 3,100 of roughly 10,000 fields are Option.

To stop the call sites from churning, encode and decode go through a blanket trait that keeps prost's method names:

pub trait Message: Serialize + DeserializeOwned + Default + Sized {
    fn encode_to_vec(&self) -> Vec<u8> {
        to_cbor(self).expect("schema types always encode as CBOR")
    }
    fn decode(bytes: impl AsRef<[u8]>) -> Result<Self, cbor::Error> {
        from_cbor(bytes.as_ref())
    }
}

impl<T: Serialize + DeserializeOwned + Default> Message for T {}

That trait is a big part of why the final deletion could touch thousands of files mostly mechanically. Foo::decode(bytes) and foo.encode_to_vec() still compile. They just mean CBOR now.

Canonical CBOR for anything that gets hashed

Plain serde CBOR is fine for transport, but it isn't a hash input. The canonical encoder follows RFC 8949 §4.2 deterministic encoding and adds a few rules on top:

• integers use the shortest form, and lengths are always definite;
• map keys must be text, they're sorted bytewise on their encoded form, and duplicate keys are an error;
• CBOR tags are rejected;
• null map values are stripped after the duplicate check, so null and absent encode to the same bytes.

There are two profiles. The signing profile also rejects floats and byte strings, so binary data has to travel as base64 text and no other implementation can disagree about NaN or about how bytes are represented. The content profile used for record hashes allows them. Here's the core of the map rule:

entries.sort_by(|a, b| a.encoded_key.cmp(&b.encoded_key));
if entries.windows(2).any(|w| w[0].encoded_key == w[1].encoded_key) {
    return Err(Error::DuplicateKey);
}
// nulls go only after the duplicate check: {"a": null, "a": 1} is still an error
let kept = entries.into_iter().filter(|e| !e.value.is_null());

The null rule is what makes adding a field hash-safe. An Option field that's None produces exactly the bytes the record had before the field existed, so old hashes and signatures stay valid. Hashing itself is a blanket ContentHash trait over anything Serialize, so the schema crates never depend on the hasher.

Arrow and Lance from the same type

Cold storage is columnar. serde_arrow traces an Arrow schema straight from the Rust type, and the same struct becomes a RecordBatch that gets written to a Lance part file (I wrote about indexing those files in the Lance sidecar post). Nothing in that path had a prost bound in the first place, so it needed no changes.

The mapping still has sharp edges:

• serde_arrow traces String as LargeUtf8, but some older hand-written schemas used Utf8. Both become TEXT in SQL. In Lance, though, mixing them is a part-file schema change, and my compactor treats drift between part files as a hard error.
• Non-Option fields trace as NOT NULL, while prost's generated serde made every column nullable. Keeping the prost field shapes kept the Arrow schemas of existing datasets identical.
• Option<SomeEnum> traces as a nullable union. Arrow unions have no validity buffer, though, so the builder emits a non-nullable field and RecordBatch::try_new rejects the mismatch. A small schema walk forces union fields to non-nullable.
• Lance rejects Arrow Map columns, so maps go through a List<Struct> codec. Decode and reconstruct also have to use identical tracing options, or the round trip breaks.
• serde serializes Vec<u8> as an array of integers. Migrated byte fields became lowercase hex text through a newtype, which is what the generated code's serde was already doing, so the bytes didn't move.

How it was staged

The pilot was one four-field type. I kept its prost-generated twin as a dev-dependency and wrote a parity test: same canonical CBOR bytes, same hash, same Arrow schema. That parity test was the gate for every swap that followed. Then the agents went component by component:

1. Writers build the native struct, convert it to the prost twin at the wire, and a test pins the bytes as identical to the old path. Readers don't change.
2. The record envelope gets an encoding field, empty for protobuf and "cbor" for the new rows. One decode function handles both. Readers deploy before writers, every time.
3. Each generated schema crate converts in place, keeping its package name and public module path, and its .proto gets deleted.
4. The last step was one big deletion: the remaining few hundred .proto files, the codegen crates, the buf workflow, descriptor artifacts, and every first-party gRPC service, replaced by HTTP routes with CBOR bodies and WebSocket streams. That PR changed well over two thousand files, adding close to 200,000 lines and removing more than 250,000.

I should be honest about that last step. CI was switched off for the migration, the agents merged their own PRs, and the deletion merged before a full workspace compile had finished. The handoff note literally says the first post-merge build is the integration compile. Four repair PRs landed the same day. It worked because I'd decided historical rows were disposable: readers skip rows they can't decode, and I didn't keep a protobuf reader for most old data. I wouldn't recommend that shortcut for a system with real users.

Over those nine days, about 70 of the roughly 900 commits in the repo were part of the migration, and around half of those were fixes.

Schema evolution without field numbers

Without field numbers, compatibility comes down to a few rules:

• A schema crate keeps its package name and public module paths, so consumers don't churn.
• A field added after release is Option forever. Later turning it into a required field with a default would change the re-encoded bytes, which breaks hashes.
• A breaking change gets a new versioned type and dataset. You backfill it, switch readers, and only then delete the old one.
• There's one type per record. No second permanent "v2" schema living next to the original, and no wire struct shadowing a domain struct.
• Every schema change gets a round-trip test in both directions:

let old = Row { value: Some("x".into()) };
let new = EvolvedRow { value: Some("x".into()), revision: Some(7) };
assert_eq!(EvolvedRow::decode(old.encode_to_vec())?.revision, None);
assert_eq!(Row::decode(new.encode_to_vec())?, old);

The part I miss is the mechanical breaking-change check protobuf gives you almost for free. buf breaking catches a reused field number or a changed type in seconds, although I'd already demoted it to a manual job a few weeks before the migration. I also had a Rust gate that diffed .proto files for persisted datasets. Both went out with the .proto files, and I don't have a shape-snapshot gate for serde types yet. Today these rules are enforced by review, including an agent review checklist. Renaming a field compiles fine and orphans every stored row that used the old name. That's the biggest hole I have right now.

Keeping it out

A CI gate runs on every PR. It fails on any tracked .proto file. It fails on any of these dependencies: prost, prost-build, prost-types, prost-reflect, protobuf, protobuf-codegen, protoc-bin-vendored, tonic, tonic-build and tonic-prost. It fails on source strings like include_proto!, prost::Message, prost_build:: and tonic::. And it fails on any build.rs that runs protoc.

There's a short allowlist. Arrow Flight is gRPC underneath, so a Flight server and a shuffle transport built on Flight still use tonic, and an indexer that reads SCIP (a protobuf format) uses the protobuf crate. Cargo.lock still contains prost and tonic through Lance, Arrow Flight and OpenTelemetry, and protoc went back into the build images the day after the deletion, because Lance's encoding crate needs it at build time. The gate covers first-party code. It doesn't pretend protobuf is gone from the dependency tree.

Bugs it turned up

These were the ones that cost real time:

• Empty input. Protobuf decodes zero bytes as a message with every field at its default, but CBOR treats it as an error. A handful of call sites depended on the protobuf behavior, so the decode helper went back to returning the default for empty input.
• A reader left behind. One writer flipped to CBOR while four readers were still running prost over the raw bytes. One of those readers sat on an identity-resolution path. For hours it logged about 40 warnings a minute and resolved nothing: zero mapping rows in the two hours before the fix, 69 in the 15 minutes after. Some test probes made it worse by wrapping decoding in if let Ok(..), so they reported CBOR rows as "no matching rows". Readers that only have bytes now sniff the first byte, a CBOR map header versus a protobuf tag.
• Crash loop on boot. A routing store read old protobuf rows at startup, and one ? took the service down on every boot. Those rows were a cache, so the fix skips them and lets the next registration pass rewrite them.
• Durable subscriptions. The snapshot source hard-failed on any pre-migration row, so a durable subscription couldn't work against any dataset with history, which meant all of them. It now skips those rows and still advances its cursor past them.
• Hand-computed sizes. A change page carried its own encoded size, computed with protobuf tag and varint math. After the switch that number never matched CBOR's encoded_len(), and every change-page read failed. Since the size field is itself part of what gets measured, the fix iterates until the value stops changing.
• Key order. The TypeScript client built maps in field declaration order, which was a protobuf habit, while the server re-encodes in canonical name order. The byte comparison failed and valid requests got rejected.
• Floats and text that collided. So that float-bearing records could be hashed without retyping their fields, my first content profile rendered floats as decimal text and bytes as base64 text. That meant 1.5 and "1.5" hashed the same, and so did a byte string and its base64 spelling. An audit caught it after the migration. The second version keeps the CBOR types, and the first version only survives as an explicit option for historical records whose version is known.

Build time

These are the pilot numbers, single runs on the same machine with fresh target directories:

After that, serde_derive and syn dominate, and trimming dependencies further changed the shape of the graph more than the wall-clock time. I don't have a clean before-and-after number for the whole workspace. The same week I also collapsed 364 integration-test binaries into inline test modules, and that moved build times more than this did, so I'm not going to make up a combined figure.

One cost I didn't expect: serde is generic, so Foo::decode instantiates the whole CBOR deserializer for Foo in every crate that calls it. With prost, the decode code lived once in the generated crate. The fix is to make the schema crate export one shared copy. It takes function pointers to the codec instances in a #[used] static, and under share-generics (rustc's default at opt-level 0 and 1) downstream crates link that copy instead of building their own:

#[macro_export]
macro_rules! share_cbor_codecs {
    ($($ty:ty),* $(,)?) => { const _: () = {
        #[used]
        static ROOTS: &[$crate::CodecRoot] = &[$($crate::CodecRoot::of::<$ty>()),*];
    }; };
}

On the first crates I tried, one leaf crate's rustc time went from 1.8s to 1.1s, and its monomorphized code shrank by 63%. The schema crate pays roughly 60 ms per listed type, so I only list types that two or more crates decode. The bigger build problem turned out to be layout, not encoding. An old aggregate schema crate re-exported dozens of schema crates, and 163 crates depended on it, so editing any schema rebuilt a big slice of the repo. Moving those consumers to the small crates they actually used took its dependents from 163 to basically none. That fix had nothing to do with protobuf.

Wire size

I didn't measure this on real traffic. As a rough illustration, I computed both encodings for two made-up rows. A ten-field record that's mostly strings came out at 203 bytes in protobuf and 296 in CBOR, which is 46% bigger, and 92 of those bytes are field names. A row of seven integers went from 30 bytes to 70. Arrow and Lance store field names once in the schema, so cold storage doesn't pay this, but the wire and hot rows do. Compression probably wins most of it back, but that's a guess. I could switch to integer keys, but at that point I'd be reinventing field numbers.

What got worse

• No language-neutral schema. The Python service moved to HTTP and I maintain its types by hand. The TypeScript side is also hand-maintained, which is where the key-order bug came from.
• No grpcurl or server reflection. HTTP endpoints with CBOR or JSON bodies work with ordinary tools, but you can't read CBOR by eye.
• Unknown fields get dropped. Protobuf carries unknown fields through a decode and re-encode. serde ignores them on decode and drops them on re-encode, so an old binary doing read-modify-write strips any field it doesn't know about. As far as I know I haven't hit this, and deploying readers before writers mostly covers it, but it's a real difference.
• More checked-in code. About 2 MB of .proto in the schema crates became about 4.9 MB of Rust. The generated code always existed somewhere, but now it's in review.
• Old data. I chose to drop most of it. That was fine for a side project, and it wouldn't be for most systems.
• The library. The serde CBOR crate I started with is unmaintained. I switched to a maintained fork after comparing output bytes across every half-float bit pattern, a sample of doubles and the integer and string length boundaries. No stored bytes changed.

Would I do it again

Yes, for a mostly single-language repo whose records need stable hashes. If your data really does cross a lot of languages, protobuf's generated clients are worth a lot. In that case keep it and canonicalize only what you hash.

If I did it again, I'd change three things: build the shape-snapshot gate before deleting the buf check, never merge the big deletion before the workspace compiles, and before flipping any writer, find every reader that decodes raw bytes, not just the ones that go through the shared decode function.

Related posts

• Indexing Lance files without the Dataset API
• Agents write tests that can't fail
• CI that tests only what changed
• The importance of remote build servers for agents
• Point reads from Lance files, and the approximate algebra that picks which files to read