CI that tests only what changed

Ryle · Aug 17, 2026

One repo I work on is a Rust monorepo of a little under a thousand crates, and most of the commits to it come from coding agents that push, get reviewed, push a fix, and push again. Running the whole workspace on every push was never going to work. What I ended up with has two layers. There's a planner that picks only the units downstream of a change, and a cache that skips a unit's tests entirely when an earlier run already passed them at the same content fingerprint. I'll cover both, plus a week of cost data that changed my mind about which one mattered more.

Selecting downstream units

CI work is grouped into units. A unit is roughly one deployable image and the crates it builds, plus a few special ones. The planner diffs the branch against its merge base, maps changed files to crates and to each image's trigger paths, and emits the units whose inputs changed. Library crates that no image owns go into a catch-all unit. I added that because an earlier version of the planner routed only by trigger paths and let library changes merge without ever being compiled.

Two kinds of change get extra coverage. A diff that only touched Cargo.lock used to compile nothing at all, so now it compiles three deep binaries whose dependency closure covers most of the workspace. Changes under the shared schema and common-library directories add a reverse-dependency compile gate, where up to three downstream deployable binaries get compiled, ranked by how much I care about them breaking. The cap of three is about cost. 784 of the 959 crates in the workspace at the time depended on at least one schema crate, so "check everything downstream" on a schema change basically means checking everything.

Each unit runs its crates as one aggregate cargo nextest invocation. If that fails, up to 16 subset reruns binary-search for the crate responsible. A passing diagnostic rerun never turns the result green. If attribution runs out, the unit reports an unattributed failure and stays red.

The waste the planner leaves

The planner is incremental at the scope of a PR. It diffs the merge base against the tip, so every push re-derives the full unit set for the whole PR. A review-and-fix loop with three to six pushes ran the same units three to six times, even when each fix touched one file in one crate. The superseded run got cancelled and its partial test work was thrown away. Then the post-merge run on the integration branch ran the same units again, usually at identical content.

The obvious fix is to keep the previous run alive and only test HEAD~1..HEAD. I decided against it. That diff is only sound if the previous run finished green against the same base, and a red run, a rebase or a two-commit push all break that. On top of that, the stale runs would hold runner slots, which were already the bottleneck.

A cache keyed by content

The design I went with leaves cancellation, run topology and merge-gate semantics alone. Before a unit's test phase, it computes a fingerprint of everything that can affect the outcome and looks for a green result at that fingerprint. If it finds one, the unit reports success through the normal result path and moves on to the build.

The fingerprint is a blake3 hash over a sorted (mode, blob sha, path) listing plus some scalars. The listing comes from a single git ls-files -s, which works on blobless partial clones because the index has the hashes without the blob content, so nothing gets rehashed. It covers:

• every tracked file under the unit's crates and their transitive workspace dependencies (normal, dev and build edges, taken from cargo metadata), unioned with the unit's image trigger paths
• the root Cargo.toml and Cargo.lock, the workspace test configuration and the sqlx offline query data
• the unit's own definition: crate list, images, smoke tests
• rustc -V and cargo -V, an image digest when the environment exposes one, a manual salt, and a format version

Since the key is based on content, the cache also dedupes across PRs, content-identical rebases, and the pair of PR run and post-merge run. Results go into ClickHouse next to the CI metrics the runners were already writing. Any failure counts as a cache miss (ClickHouse unavailable, cargo metadata failing, a git error), and the unit just runs like it would have without the cache. There's one exception that goes the other way. An unparseable index line, or a unit that matches zero files, refuses to fingerprint at all, because folding a blank blob sha into the hash would let two different contents collide.

Trust rules

Only green results get written. A red result always reruns, since it might have been a flake. Beyond that:

• A green result is trusted for a limited time. I started with 30 days and later cut it to 7.
• Each week, 5% of trusted hits run anyway. The selection hashes (week, unit, fingerprint), so retrying the same commit makes the same choice and doesn't multiply cost. The recorded outcome works as an ongoing check that cached verdicts still hold.
• UI units and units with smoke tests always run, since they involve a browser or a running service.
• Units with database-backed suites use a separate key that only runs with the database available can read or write.
• Full-rebuild runs turn the cache off for the whole process. A repository variable turns it off everywhere without a code change.
• A skipped unit posts an extra commit status naming the source run, so someone reading the check list can tell nothing re-executed. It's a separate context because the forge posts each job's own status after the job ends and would overwrite an update made during the job.

The tradeoff I accepted is flake grandfathering. If a flaky test passed once at a fingerprint, it stays green until its inputs change. The TTL, the validation sample and a nightly workspace check are the backstops. I haven't measured how well they actually catch intermittent failures.

Lesson one: I skipped the shadow soak

The plan had four phases: measure, run in shadow mode for about a week (log decisions but never skip), go live, then extend. Going live was supposed to require zero false hits during the shadow window. A false hit is a unit that would have been skipped but then failed, which means some fingerprint input is missing.

What actually happened is all the phases went out live in one change on the same day. What saved me was that the shadow rows kept being written. Every unit execution still recorded its would-skip decision next to its actual outcome, so validation turned into something I checked after the fact.

Two days after going live, I found the closure was missing build dependencies, which can change test behavior through build scripts and generated code. The fix added a format version to the hash so rows written under the old input contract can never match again. I'd put that version in from the start next time.

Sixteen days in, the shadow table had 11,939 rows, 3,631 hits and 3 false hits. All three were on commits that also had a passing row for the same sha, so they look like flakes and not coverage gaps. Based on that, I let the three cross-crate gate binaries use the cache. They had been force-run because shared-schema breakage had reached the integration branch twice, and the heaviest unit had been a miss 22 times out of 23 on the PR lane. Taken at face value, 3,631 of 11,939 is about a 30% would-skip rate, but that number mixes live hits, shadow-era hits and exempt units, so I treat it as rough.

Going live early worked out this time, but I wouldn't recommend it as a policy. The thing that made it recoverable was writing the comparison rows no matter what mode the cache was in.

Lesson two: the cost data pointed somewhere else

The cache had been live for about two weeks when I finally sat down with a week of runner metrics, which was the first real measurement I'd done:

Three quarters of runner time went to runs that ended red, and the cache never stores a red result. It only helps with repeated green work. The bigger pools of time were failing runs and compile fan-out from a small set of crates.

The per-crate metric only records the 50 slowest compilation units in each build, so cheap crates that are used everywhere mostly don't show up in it (and that's why first-party plus third-party adds up to less than 433). It's good for ranking the top of the list and not reliable for totals.

The 53% came from a build-script watch. Each schema crate's build ran codegen (protobuf at the time) and told cargo to rerun on changes anywhere in the 455-file definition tree. One crate emitted 940 watch lines for the 29 files it compiled. Deleting the per-crate watch lines did nothing, because the real watch came from a shared helper that printed one for every file it read. Emitting watches only for the files a crate actually compiles fixed it. On a fixed 245-package check, touching an unrelated package's definitions went from rebuilding 60 units to 9. Narrowing the one crate that compiled 167 definitions down to exposing 50 modules took that to 0. That change cuts compile work on every run, whether the cache hits or not.

The table also has a scope problem. During that same week, PR CI moved off the Actions runners onto shared workspace pods, so the 1,409 hours describe a lane that doesn't run anything anymore. The compile numbers still carry over, because the same unit runner emits them wherever it runs. I need to recompute the failure share on the new lane before acting on it.

Lesson three: migrations drop properties you got for free

On the old lane, the forge cancelled a PR's previous run when a new push came in, and the cache design assumed that would keep happening. The new pod lane had nothing like that. In one snapshot I saw four legs running for a single PR, each holding a lane permit and a warm target directory, and three of them were computing answers nobody would ever read. The oldest had been running for 1.4 times its timeout.

So if you move CI to new infrastructure, it's worth writing down what the old system did for you without being asked (cancellation on push, timeouts, cleanup) and checking each of those on the new one.

What I'd keep and change

I'd keep the content key, the fail-open behavior, green-only writes and the always-on comparison rows. With those in place, every other mistake I made was recoverable. What I'd change is the order: measure first, since failing runs and fan-out from a few crates turned out to matter more than repeated green runs. I'd also measure the compile cache before relying on it. When I went to look, the primary compile cache had no hit/miss telemetry at all.