You should be spending as much on CI as you do on AI
Most of the code in the repos I work on now gets written by coding agents. Generating a change got cheap fast. Knowing whether that change is correct didn't. Agents make a lot of mistakes, they make them confidently, and they open pull requests faster than anyone can read them. The only thing I've found that keeps up is a very large amount of automated testing, and the part that catches the most is the testing that happens after the code is deployed. If you're paying for agents to write code, I think you should expect to spend something comparable on the machinery that checks it, in money and in engineering time.
I've written before about what self-hosted CI costs, testing only what changed and tests that can't fail. This post is the case for why all of that is worth it, and what it looks like once you take it seriously.
How much testing
The largest repo has about a thousand Rust crates and around 46,000 test functions, plus roughly 6,000 UI unit tests and 145 browser end-to-end specs. CI ran about 2,800 jobs over 548 distinct commits in the last week I measured, between roughly 110 and 800 jobs a day depending on how busy the agents were. On top of that a nightly rotation runs the full suite in 14 shards, one every six hours, so the whole thing gets covered every three and a half days even where the per-PR planner skipped it. After every deploy there are change-selected smoke checks, and a set of synthetic probes runs every hour against the live system whether anything changed or not.
That sounds like a lot, and it is. About a quarter of all commit subjects in that repo start with "fix". Agents fix their own mistakes quickly once something tells them about it. The whole job of the test system is to be that something.
Pre-merge CI can't be the correctness gate
This was the biggest change in how I think about it. When I went through my incident notes, unit test failures barely showed up. The failures that actually reached users were mostly ones no pre-merge check could see:
- An agent doing a rename sweep edited comments inside SQL migrations that had already been applied. The checksums stopped matching, the new pods crash-looped on startup, and the old pods kept serving code with the feature removed. Every test passed.
- One PR added a single key to a secret sync that had no value behind it. The sync tooling fails the whole secret when one key is missing, so an unrelated token stopped reaching image builds.
- Two PRs merged within minutes of each other with the same migration version number. Each was fine alone.
- A new startup step fetched from git with a token that didn't work, so the service crash-looped after deploy.
- A login flow worked in the browser, but the native mobile client got tokens with a different issuer that the backend rejected. It only showed up on a phone.
- Chat replies were being saved under the wrong owner. My read-only smoke checks stayed green through it, because reads still worked. That's why the smoke test now writes.
So I stopped trying to make pre-merge CI catch everything. It stays fast and narrow: static gates, a compile check, tests for the changed units. The real gate is post-deploy verification, and pre-merge CI exists to keep obviously broken code from reaching it.
What post-deploy verification checks
After a merge, the publish step builds the changed images, renders the deploy manifests, syncs them, restarts what changed and waits for it to become available. Then it runs smoke checks selected by what the change touched: pod health, database connectivity, network egress rules, a browser journey through the UI and so on. Every one of them is fatal. I had a list of non-fatal checks once, and the list is empty now, because a check that's allowed to fail is one nobody reads.
The checks I trust most are the ones that don't believe anything the build says about itself. A publish once reported success while the old image was still serving. Another time an image carried a fresh revision label over a stale binary. A manifest render step skipped silently on at least three separate merges, so the PR merged green and the deploy repo never changed. The verifier I use now reads the digest each running pod actually has, compares it to the registry, and checks by git ancestry that the build contains the merge. It doesn't use labels or the version string baked into the binary, since those have both been wrong.
Then there's one real request. The release smoke gate sends a fresh conversation through the live UI to a cheap model: at most three turns, a hard token cap, and a quarter-dollar ceiling, with about two cents as the typical cost. The reply has to arrive and still be there after a cold reload. It's the only check that exercises the whole path the way a user does, and it has caught things that passed everything else, including one PR that passed CI and then timed out on the live round trip twice.
A few hourly probes keep running after that, with no change involved: the public stack, the UI, an authenticated browser session that checks a real account's conversations render, and the public mirrors. One trap they guard against is a 200 that's actually the HTML app shell where JSON was expected. That looks healthy to a status-code check and is completely broken.
Checks have to prove they ran
With this many tests, the failure mode that hurt most was a check that quietly stopped checking. A workflow step had never passed once in its life, and nobody noticed because it was marked continue-on-error. A publisher job was red on every commit for 13 days while the build gates stayed green. UI tests returned success when the package manager wasn't installed. A test aggregate failure got absorbed as a pass with a log line. An edit to one registry file reached no CI lane at all, and that happened three different ways in one week.
So every layer has to show evidence that it executed something. Test filters that should match run with --no-tests=fail. A database test with no database fails instead of skipping. A test-cache hit gets reported as a cache hit, never as a new run, and 5% of hits are re-executed each week to check the cache is honest. The hourly smoke gate checks how old its own newest result is, and if it's older than twice its period it reports itself as dead. A probe for a service I'd decommissioned was failing every hour, and I removed it, because a guaranteed failure trains you to ignore the alerts.
Failures need automated follow-ups
Finding the break isn't the end, since nobody is reading the dashboards all day. Every failure path files something an agent will pick up. A failed post-deploy smoke check opens a ticket keyed by the commit, and skips that if one is already open. Three red runs in a row on the integration branch raise one escalation per streak. A failure aggregator collects failures from the probes, CI and the review bots every two hours, dedupes them, and dispatches one fixer that can propose a change but never merge it.
Merges get checked too. A verifier walks every PR marked merged in the last two days and confirms its commits are actually on the integration branch. It found dozens of PRs the forge reported as merged whose code never landed. Before that, finding them was luck, usually someone noticing a feature missing days later. Now each one gets an automatic re-land PR, which a person or agent still has to merge.
And 45 minutes after a merge, which is enough time for the image rebuild and the GitOps sync, an agent gets the original ticket text and checks that the deployed system does what was asked. The verdict is verified, not resolved, or inconclusive, and the deterministic image check sets a floor the model can't override: a stale image means not resolved no matter what the agent thinks it saw. Only verified closes the ticket. Not resolved sends it back with the diagnosis.
All of these follow-ups are bounded. Each fixer is single-flight per issue with a cooldown, verification gets a fixed number of dispatches per merge, and nothing in the follow-up path calls a model more than once per round. Without those limits the follow-ups become their own incident.
Agents inflate the cost of testing
The other side of running this much testing is that agents make it expensive in ways people don't. If you leave CI alone for a few weeks, its cost grows on its own, so optimizing CI turned out to be continuous work and not something I did once.
Agents add tests freely, and they add them in whatever shape is easiest. At one point there were 364 separate test binaries across 142 crates, and one crate's 44 binaries cost about five and a half minutes of a single test run just in linking. Agents push again and again on the same PR, and each push used to re-run the whole PR's units. Several CI runs could be going at once for different heads of one PR, because nothing cancelled a run when the head moved, and killing those stale runs took one node's load average from 29 to 9. Agents also wait badly. In one nine-hour window, about 62 of 80 busy runner-hours went to review and fix jobs that were mostly sitting in a polling loop while real jobs queued behind them.
Some of the waste was mine. A lint pass wrapped in || true gated nothing and was still about 41% of CI lane time. A static-gate block failed fast, so one PR found out about three drifted generated files one CI round at a time, paying a twenty-minute compile each round. The compile cache reported a 54% hit rate, but weighted by compile cost it was 43%, and one cache key that accidentally included a build directory path dropped a builder's hit rate below 1%. The most expensive unit ran cold on 22 of 23 runs. Pods reported 32 CPUs from nproc under a cgroup quota of 16, so anything that sized itself from nproc oversubscribed.
The fixes were all fairly unglamorous. Remote build servers with warm per-slot target directories, so agent workspaces don't compile at all. A content-keyed test cache so an unchanged unit doesn't re-run. Cancelling superseded runs. Running every cheap check at once and reporting the whole list. Moving agent review and fix jobs off the runner pool so they wait without holding a slot. Scoping widely shared changes to a compile check of the services that actually ship. Turning rules like "put tests inline" into tool hooks that block the action, because agents don't reliably follow rules that are only written in the prompt. And measuring all of it per job, with both the self-hosted cost and what a hosted runner would have charged, so I can see when it starts drifting again.
Where the money should go
If I were starting over with agents writing most of the code, I'd build the verification side first. Keep pre-merge CI fast and narrow. Put the real gate after deploy, with checks that read what's actually running, one real request through the live system, and hourly probes that keep going when nothing changed. Make every check prove that it ran. Route every failure into a ticket an agent will pick up, with hard limits on retries. Then keep measuring CI cost, because the agents will grow it faster than you expect.
The models are the part everyone budgets for. In my experience the testing around them is what decides whether their output is worth anything.