The importance of remote build servers for agents

Ryle · Oct 26, 2026

My coding agents run in Kubernetes pods, and for a long time those pods also compiled and tested the Rust they were writing. That turned out to be the hardest resource-sizing problem I've had on the cluster. An agent spends most of its life waiting for a model to answer, using about a gigabyte of memory and a fraction of a core. Then it runs cargo test and wants a dozen cores and 15 GB for a few minutes. Kubernetes asks you to describe a pod's needs ahead of time, and there's no honest description of a pod that does both.

What fixed it was taking compilation out of the agent pods entirely. Agents now send build and test commands to a small fleet of remote build servers and get the output back. The agent pods went from a 16 GB memory request and a 48 GB limit to 2 GB and 4 GB, and the build servers became something I can actually size on purpose. This post is about how I got there, and how I decide what the build servers get.

I wrote about the build server design and its compile cache in the CI cost post. This one is about the agent side, the machines and the sizing.

Two very different workloads in one pod

Kubernetes schedules on requests: the CPU and memory a pod says it needs. Limits are the ceiling a pod gets killed or throttled at. The scheduler packs nodes by adding up requests, so a request that's too low overpacks a node and one that's too high wastes it.

Here's what the two halves of an agent pod looked like when I measured them:

Early on I had sized agent pods for the build, at 2 CPUs and 12 GB requested. Four of those reserved 48 GB of my autoscaled pool while actually using a few gigabytes, and the CI runners couldn't get scheduled. So I cut the request to 1 CPU and 4 GB to match what the agents really used. That was correct for the agents and wrong as soon as they compiled.

Then I moved CI into the agent pods

The reasoning was sensible on paper. The agent pods already had warm Rust target directories on fast local disk, and the separate CI runners were rebuilding from colder caches. So in late July I gave each agent pod four warm target directories and moved pull request CI onto them. The next week was mostly incidents.

CPU. One agent pod sustained 13.68 CPUs on a node with 19 allocatable. That node ran 152 pods in total, including the GitOps controller and the LLM gateway, and during a CI leg the rest of them shared about 2 CPUs. The scheduler kept putting more pods there because it saw a 1-CPU pod. Inside the container, nproc and the load average report the host, so the pod thought it had 32 cores when its quota was 16. Every "we have 32 cores" reading taken from inside a pod was wrong.

Memory. One pod's memory cgroup sat pinned at its 32 GB limit, with 368 recorded limit breaches. Before any build ran, the agents and their tools already held 7.8 GB that couldn't be reclaimed. Four builds at their p95 peak need 28 GB on top of that. The pod got OOM-killed, so I raised the limit to 48 GB and the request from 4 to 16 GB. The note on that change says the 4 GB request "told the scheduler the node had ~90Gi free when it had ~10Gi". The first try at 48 GB was rejected by a namespace limit capped at 32 GB, which also had to move.

Disk. A pod's 128 GB workdir actually held 382 GB, because it was a directory on the node's local disk and nothing enforced the size. The kubelet only attributed part of it to the pod, so the first sign of trouble was node-level disk pressure: eleven DiskPressure transitions in eleven hours on one machine, and an agent pod evicting itself. When I added an ephemeral storage request of 24 GB, the next build went to 35 GB and the eviction killed four in-flight CI legs. Incremental compilation state alone was 76% of one release build tree.

Placement. To keep warm target directories, pods were pinned to the node holding their disk. So a pod couldn't be moved off a struggling node without throwing away the thing that made it fast.

The bad part is what an OOM kill means in this setup. A build that's killed for memory reports a red CI result on code that's fine, and an agent will then try to fix code that isn't broken. At one point a git repack running next to three builds got a shared CI pod OOM-killed about every 13 minutes, and no PR in the fleet could hold a green check long enough to merge.

Every fix pulled in the same direction. To make builds fit, agent pods needed bigger requests, bigger limits and more disk, and every idle agent paid for them. By the end a standard agent pod requested 16 GB with a 48 GB limit, while the agent in it used about 2 GB most of the time. One of my autoscaled cloud nodes has 29 GB of allocatable memory in total, so that limit was 19 GB more than the whole node.

Route the operation, not the task

You can't tell ahead of time which agent task will compile. Almost all of them do eventually, and many only run cargo check twice. So deciding per task where an agent should run doesn't help. What worked was deciding per operation: an agent never needs to be on a compile machine, it needs compile results.

A build server takes a commit, an overlay of the agent's uncommitted files and a cargo command. It runs the command in a slot with its own warm target directory and streams back the output and exit code, plus any binaries the agent asked for. Each slot runs one build at a time with fixed memory and parallelism, so if a build runs out of memory it fails one request, and the slot comes back with its cache intact. The first soak test ran 40 of 40 builds, and the compile cache hit rate was 92.9% where the in-pod CI lane had been getting 54.3%. Warm builds of service crates took 19 to 77 seconds. The in-pod lane's worst legs had taken 99 minutes.

Then I made it the only path. Agent pods have a wrapper in front of cargo that sends everything to a build server, and rustc is replaced with a stub that prints a message and exits. Before that, the remote path fell back to compiling locally whenever it couldn't find a credential for a build server, which brought the original problem back in a way that took a while to notice. A build that can't reach a build server now fails with a message saying why.

After that, agent pods could be sized for agents:

At one point I measured four development agent pods together using 0.63 CPUs and 3.2 GB, with the busiest peaking at 1.44 GB. Six agent pods used about 7 GB combined, less than one busy build server. They also became cheap to move. Each pod now holds one conversation. They fill spare room on the dedicated machines first, leaving 15% headroom on each node, and overflow to a pool of small cloud nodes (8 vCPU, 16 GB) that's kept separate from the burst build nodes so they don't fragment them.

Build servers want dedicated machines

The build side has the opposite profile: large, long-lived state on fast disk. A warm target directory for the big repo measured 250 to 270 GB, and each build server keeps a 250 to 300 GB compile cache next to it. A new cloud node starts with none of that, and the first heavy build on a fresh one took about 20 minutes even with cache hits. Copying a warm 100 GB target directory over the network costs more than the cold build it would save.

So the main build servers run on two dedicated servers that keep their disks between builds. The larger one has a 16-core Ryzen 7950X3D, 128 GB of RAM and two 1.92 TB NVMe drives. The smaller one has a 14-core Core i5-13500 with 64 GB. With the small control-plane machine, the three dedicated servers cost about $195 a month. Each of the two build machines runs one build server pod, pinned to its node with a node selector.

Autoscaling for bursts

Dedicated machines handle the normal load. Some days ten agents push at once, and for those there are burst build servers on autoscaled cloud nodes (16 vCPU, 32 GB, each with a 1 TB volume for its target directories). I covered the scale-down bugs in the earlier CI post. The trigger turned out to be just as hard to get right.

There are two layers. A builder scaler decides how many burst build server replicas should exist. Their pods start out Pending, and the node autoscaler sees those and buys cloud nodes for them. Both layers have caps. There are at most three burst builders, and my cloud account has a hard limit of 112 shared cores that the burst pool and the agent pool together have to fit inside.

The trigger went through three versions:

• Queue depth of at least 2 on every dedicated build server for 5 minutes. Both machines almost never had a queue at the same moment, so this rarely fired. My notes from the time call it effectively dead code.
• The oldest waiting build is 30 seconds old, escalating after 5 minutes. This fired, but the saturation check counted the compile jobserver's tokens being in use as "busy". So a long build with an empty queue behind it kept resetting the release timer, and burst nodes stayed up with nothing waiting.
• At least 4 builds waiting, the oldest for at least 30 seconds, sustained for 5 minutes for the first burst builder and 15 for the rest. Release happens after 10 idle minutes for the builder and then 15 for the node, and no node goes away younger than 50 minutes.

The current version is deliberately lazy, because a cold burst builder is only worth it if the backlog lasts longer than its warm-up.

How I size the build servers

A build server has five knobs, and each one is limited by a different resource:

• Slots: concurrent builds, each with its own warm target directory. Slots cost disk, not CPU.
• Jobserver tokens: how many compiler processes may run at once across the whole machine. All slots on a node share one GNU make jobserver, so several builds can't each start 16 compilers.
• CPU limit: the pod's cgroup quota.
• Memory limit.
• Disk thresholds: a build server only takes new work with 160 GB free, and treats 140 GB as critical.

The rules I ended up with are simple. Cargo ignores -j and CARGO_BUILD_JOBS once MAKEFLAGS names a jobserver; I checked that on the build servers before relying on it. Each running cargo also holds one implicit token on top of the shared ones. So the real compile width is tokens plus slots, and that sum is held at the CPU limit. Memory is planned at 2.7 GB per token plus room for the linker, and tokens times 2.7 GB has to stay under the pod limit. The 2.7 GB is a provisional planning number, not a calibrated measurement of rustc. The number of slots comes from disk: how many warm target directories fit next to the compile cache with the free-space margin intact.

Then I measure, and move one number at a time. Every build request records how long it waited, how long it ran, peak memory and whether it timed out. Over the months the big machine went from 20 tokens and one build at a time, to 8 slots and 14 tokens, to half of that once agent pods started sharing the node, to 6 slots and 6 tokens with a CPU limit of 12. The burst builders run 8 slots on 4 tokens, plus a memory governor that sets memory.high at 22 GB and holds back tokens when memory is above 20 GB or the pressure signal goes above 5%. That's 12 compile processes at most, under a CPU limit of 14.

Here's a week of that telemetry for the two dedicated build servers from late September:

Memory requests sit a little above the measured p95, with the limit leaving room for the linker. Six agent pods use less memory than one busy build server, so the next machine I buy should be a build server (16 cores, 128 GB, a few TB of NVMe) and not more room for agents.

What I'd tell someone starting

Don't let agents compile where they think. The two workloads have different shapes, and any pod sized for both is wrong most of the time. Give the build servers dedicated machines with big local disks, because their value is warm state, and keep the agents on small, movable nodes. Make the remote path the only path, and have it fail loudly when it can't reach a build server instead of falling back to a local build. Use a shared jobserver per machine and size tokens by memory, slots by disk and the sum by CPU. Autoscale for sustained backlog, not momentary saturation, because a cold build server is slow for its first 20 minutes. And record queue wait, peak memory and timeouts for every request, because that's the only way to tell which machine to buy next.