What I learned running a fleet of AI coding agents

Ryle · Oct 5, 2026

For a while now, almost every change in my hobby projects has been written by an AI coding agent running in a Kubernetes pod on my cluster. My part is writing tickets, answering questions and reading the evidence the agents leave behind. The agents branch, edit, test, merge, publish and then check the deployed result. After a few months of this across several projects, most of what I learned is about the system around the agents: how work gets routed, how loops end, and which boundaries an agent will go around if you let it.

The scale

I pulled these numbers from the integration branch's git history in the repository where most of the agent work happened. The branch has 13,268 commits, and 9,707 of them are squash merges with a PR number. The first PR in that history is #7 and the highest is #10,830, so the agents opened roughly 10,800 pull requests over a few months.

Here's how authorship breaks down. A CI bot wrote 3,450 commits, nearly all of them deploy-manifest and image-tag updates (2,085 are "update deploy manifests, no image change"). A forge bot that does merges on the agents' behalf wrote 2,580, all of them from partway through the third month. Another 6,924 (52%) are under per-slot agent identities, which is how merges were attributed before the forge bot. So about 45% of commits are under bot identities and most of the rest are under agent slot identities. Git authorship can't prove who typed what, so "almost every change is agent-written" is an inference from how the pipeline is set up, but I'm fairly confident in it.

The second month was the peak, and a lot of what follows comes from that month and the one after.

How a change flows

Each agent gets a slot, which is a pod with a persistent workspace (a repository checkout plus scratch space) and a session I can attach to. A request starts as a ticket, and related tickets roll up into a longer-lived objective that I call a epic. An agent picks up the ticket, branches from the tip of the integration branch, makes one coherent PR, and runs a CI plan scoped to the crates its diff touches. Then it merges, runs the incremental publish for whatever images changed, and checks that the deployed service actually does what the ticket asked. Deployment is GitOps: CI renders manifests and ArgoCD syncs the cluster, so no agent applies anything by hand outside a debug namespace.

Where the finish line sits turned out to matter more than I expected. For a while "PR opened" counted as done, and the queue filled up with PRs nobody owned once the session that wrote them ended. "Merged" wasn't enough either, because if the publish never ran, the old behavior keeps serving. Now an agent owns its change through deployed verification. I also read agent, issue and deployment state separately, since "agent stopped" doesn't mean "issue done."

Where agents fail

The failures that caused real trouble were rarely bad code on its own, and review caught a lot of real bugs. In one snapshot of eight stuck PRs, the automated reviewer had blocked them for good reasons: a service that accepted any token signed by the identity provider because the audience and issuer weren't pinned, a heredoc that expanded a tracing endpoint to an empty string, and a SQL parser that silently dropped LIKE ... ESCAPE instead of failing. The problems came after that.

Silent success

The most expensive pattern I saw was a step that exits green without doing anything. In that same snapshot, the automated fixer ran for 24 minutes 54 seconds on one PR and pushed no commit. On another it pushed a CI tweak and ignored the critical finding. It exited green when its model backend was unreachable or when it saw "no fix signal," and a silent green fixer looks exactly like "nothing to fix," so I couldn't see that it wasn't converging until I listed the stuck PRs by hand.

I kept running into the same shape. A dispatch path picked a target pod from rows marked "running" without checking their heartbeats. One snapshot had 15 of those rows and one live pod, so messages went to pods that no longer existed and the UI could only say that nobody picked them up. A keepalive loop spun forever at debug log level because its own identity couldn't read back its own row. And when a model plan hit its weekly limit (access was blocked for six days), slots got stuck on raw error transcripts instead of recognizing a dead turn and releasing the slot.

What I keep having to relearn is that every terminal state needs its own recorded outcome, and "succeeded with no payload" should never read as "fine." Most of my guardrail work has been turning silence into a named state.

Loops that don't converge

The worst stretch was when both subscription coding plans had been rate-limited for over a day, so every fix attempt fell back to a paid API model. That model was also returning 429s, so legs never exited cleanly, a runner killed them at 30 minutes, and the orchestrator re-dispatched the whole leg with the full kickoff prompt. I counted 64 kickoff resends in six hours on a single PR, and roughly 11,000 to 20,000 paid fallback calls per day for three days.

Under that was a less visible bug. When the fixer hit its round limit, it posted a hand-off comment and returned success with no result. The orchestrator took that as a harmless yield and dispatched again. Each retry exited in about ten seconds and posted another "deferred without a result" comment, until a dispatch cap marked the fix as given up. Then a policy for exhausted PRs auto-merged one with failing CI. Each piece did what it was designed to do, and put together they burned a lot of tokens and then merged red code.

Shared state when merges pile up

With dozens of agents merging into one branch, one bad merge breaks everyone. One time a PR broke compilation of a test crate in another service, and every PR that triggered a workspace check inherited the failure. Another time a burst of merges let one PR through with red CI, and the occasional "passes" after that were runs where the diff selector skipped the broken unit, so it looked like flakiness. Agents are reasonably good at fixing their own breakage. They're bad at noticing that the failure in front of them came from someone else's merge an hour ago.

Guardrails that held up

One pass of each expensive step

After that bad stretch I replaced the multi-round review/fix loop with a straight line. For an orchestrated PR it goes: execute once, review once, run CI once, do exactly one fix pass that gets both the CI failures and the review findings, recheck with build and tests only, then merge. Every stage that calls a model is capped at one, and there's no loop anywhere. I did this because of cost per issue. Second and third fix rounds on the same failure almost never converged.

The part people might object to is that the merge at the end is unconditional. If a failure survives one focused fix pass, it gets merged and caught by the dev deployment. The next agent starts from a real broken state instead of a stalled PR, and nothing sits half-done waiting for me. That only works because the blast radius is the dev environment and because the publish and verify steps run in the same pass. I wouldn't do it against production. Agents I drive directly, outside the orchestrator, keep iterating until the change works and then merge. The one-pass cap is only for the automated lane, where nobody is watching each turn.

Tests the CI can trust

When agents are pushed to get to green, they'll write tests that can't fail, with no assertion, no error path, and nothing that could ever turn red. Fairly early on I added a lint that rejects tests with no failure signal, and it does catch real cases. That's a big enough topic for its own post, so I'll leave it there.

The other half was CI scope. Running every crate on every change cost too much, so CI checks the changed crates and the images they feed. Schema-level changes trigger a compile-only check of every deployed service downstream, and I wrote down the coverage that drops out.

Pauses and approvals the agents can't undo

This one surprised me the most. I went to pause the PR orchestrator once and found it running, even though I thought it had been paused for days. Most likely a debugging session resumed it, or a redeploy went back to the newest enabled configuration. Either way, if any session with the right command can reverse a pause, it's really only a suggestion.

Other boundaries leaked in the same way. A readiness review of the epic system found that manual claims got past paused epics during cold start, because the pause check ran before the slow startup path instead of right before spawning. Merge permissions had a similar hole. Force-merge was restricted to one role, but the role was an environment variable the caller set, so a rejected caller could set it and resend the same request under the same key. I changed the capability to an allowlist of principals held on the server, and the role is now only an audit label.

For anything that costs real money, I made the rule very simple. The GPU autoscaler stays paused unless I explicitly say otherwise, and general task authorization, queued work or a finished lifecycle don't count as saying so. That rule came from an incident where about six rented GPU machines were up at once, one of them rented on-demand at two to three times the bid rate, on a provider with no native hourly spending cap. The fix was an aggregate dollars-per-hour cap enforced by the reaper, plus a standing instruction that agents report the blocker and wait. I also wrote down that asking another agent to do something your own session was denied counts as a violation. An agent that gets denied and has a cooperative peer one message away has an obvious workaround, so the rule has to spell it out.

The undo can't depend on the thing being changed

When I went over the workflows that change cluster infrastructure (node firewall rules, cluster provisioning), the design rule I came out with was that an executor inside the blast radius of its own operation can't report the outcome. If a firewall change cuts the path to the API server, the runner dies with it and you get silence, which looks the same as "still running." The firewall workflow already mostly had the right shape, with a revert timer on the node itself, but its persist mode killed that timer before anything outside confirmed the node was still reachable.

Tracking that reflects reality

Tickets and epics are how I steer, and agents create them freely, which turned into its own problem. One readiness snapshot had 145 epic roots, 106 of them still proposals; 172 tickets marked in progress, 150 of those with no assignee; and 32 agents reporting 6 busy, 21 idle and 5 unknown. None of those counts tells you what is actually running. So now new epics are a rare last resort that has to justify itself against the existing ones, unavailable state gets reported as unknown instead of empty, and a dispatch claim has to come with delivery evidence.

What I'd tell someone starting

I'd decide what "done" means before adding a second agent, and make it the deployed result. I'd give every step an explicit terminal outcome and treat "no output" as a failure until shown otherwise. I'd cap retries on anything that calls a model and measure spend per issue, because loops add up faster than you'll notice from reading transcripts. Pauses, spending limits and merge rights should live on the server, tied to credentials the agent doesn't have. And I'd keep automatic merges in an environment you don't mind breaking.

Going back through my incident notes, a lot of the plumbing I built is there because something exited green while doing nothing.