RL on a coding model with cargo test as the reward: most of the bugs were in the grader

Ryle · Jul 12, 2026

As part of the fine-tuning side project, one training stage was reinforcement learning on code. The idea is simple. Hide a function from one of my own Rust repos, ask the model to write it back, and grade the answer by actually compiling it and running its tests. No judge model and no human labels. The reward is whatever cargo test says.

Most of what went wrong had nothing to do with the RL algorithm. The grader passed every answer that compiled, whether or not the tests passed. A shared grading service kept getting restarted by other jobs, which zeroed the rewards of whatever was running. And at the end, I never got a clean measurement of whether the RL stage made the model better. This post covers all three.

The terms

GRPO (Group Relative Policy Optimization) is the RL method from DeepSeek's papers. For each prompt the model generates a small group of answers, called rollouts. Each rollout gets a score, the reward. GRPO then pushes the model toward the rollouts that scored above their group's average and away from the ones below it. The "advantage" is just (reward − group mean) / group std. There's no separate critic network, which is why it's popular for tasks with a checkable answer. I used the GRPOTrainer from Hugging Face's TRL.

LoRA means I'm not changing the base model's weights. I train small low-rank adapter matrices next to them. The model is Qwen3.6-35B-A3B, a mixture-of-experts model, and the training ran on one 96 GB GPU. The other post covers the hardware side.

Where the tasks come from

I mark functions in the repo as held out with comment markers. Each one has to have a test that calls it directly, a body of 10+ lines of real logic, and not churn much. Tests get a matching marker that links them to the function. The training-data builder strips the held-out bodies from everything the model is trained on, and the same IDs drive both an eval and the RL tasks. The dataset I trained on had 290 of them.

The prompt is the file with the function's body cut out, the signature and doc comment left in, plus the tests. The model has to output only the body.

Two things about the tasks turned out to matter more than I expected.

The bodies leaked into the training data anyway. Agent conversation logs were part of the corpus, and they contain pasted code. A contamination check now drops any held-out function that shows up in the corpus. Its code stays in the training data but stops being a task, because otherwise the RL just rewards recall.

The prompts were too big. With whole-file context they were about 4,800 tokens at the median and 8,700 at the max. Several rollouts of that through a 35B model, in a generation pass that isn't gradient-checkpointed, ran out of memory on the 96 GB card. Truncating the prompts to about 400 tokens made it fit, which proved length was the problem, but then the model can't see the code it's supposed to write against and the rewards go to about 0. What I shipped keeps 24 lines on each side of the function, drops any task whose prompt is still over about 1,200 tokens (logged, not silent), and generates 4 rollouts per prompt instead of 8. Four is about the floor. With a group of 2 or 3, the group statistics are mostly noise.

How a rollout gets graded

The rented GPU box has no Rust toolchain, and I didn't want to build my workspace there. So grading is a separate service on a CPU box in my cluster, which I'll call the verifier. Each GRPO step, the trainer's reward function sends the whole batch of completions to it in one HTTP call. For each completion the verifier:

1. makes a git worktree of the repo at a pinned commit,
2. splices the body into the masked function,
3. runs cargo build and then cargo test for that crate,
4. scores tests passed over tests total, with 0.0 for a build failure or timeout,
5. deletes the worktree.

Partial credit is there because some functions have several tests. Some also have negative controls: tests marked #[ignore] that are written to fail on the correct body and pass only on an obvious cheat, like a function that always returns the same constant. The runner inverts those, so a cheating body loses points. There are only a handful of them, though.

The trainer side has one more rule: if the HTTP call fails for any reason, every rollout in the batch gets 0.0. My reasoning was that if every reward in a group is equal, every advantage is zero, so a verifier outage wastes a step instead of training on garbage. That turned out to be only half true (more below).

The grader that passed everything that compiled

This was the worst bug, and it's the same bug as tests that can't fail, just with the RL loop doing the exploiting instead of an agent.

The task data stores the test's bare function name, like bar_works. The verifier ran cargo test -p <crate> bar_works -- --exact. But the test lives inside mod tests, so its real path is tests::bar_works, and --exact matches nothing. Cargo runs zero tests and exits 0. The verifier scored on the exit code. So every completion that compiled got a reward of 1.0, and the reward was really "does it compile".

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

For RL that's worse than a noisy reward. Compiling is much easier than being correct, and GRPO is very good at finding the easiest way to score. The fix drops --exact and scores on the named test's own result line in libtest's output, matching the last path segment exactly. If no line for that test shows up, because the filter missed or the test target didn't build, it doesn't count as a pass. Ignored tests don't count as having run. As far as I can tell, that also covers a body that calls std::process::exit(0) before the test can report.

The fix landed before either of the full training runs I describe below. But a lot of my early checks happened before it. In particular, a hand-run grade I'd been using as proof the verifier worked ("131 to 148 seconds, reward 1.0") was taken under the old scoring. I don't actually know if that test passed or just compiled.

What still isn't covered: the tests are in the prompt, so a model could hardcode the exact values the tests check for. The negative controls catch some shapes of that, not all. I didn't see it happen, but with roughly 38 steps of training (see below) I wouldn't expect to.

The shared verifier

At first there was one verifier for everything, a single-replica Kubernetes Deployment. Every fine-tuning or benchmark run started by pointing it at that run's commit: it patched the commit into the verifier's config and restarted it, so the verifier would check out the same code the run's tasks were built from. The Deployment used the Recreate strategy, which kills the old replica before starting the new one.

That's fine with one run at a time. With two overlapping runs, the second run's startup restarts the verifier under the first run's GRPO phase. Any batch the first run had in flight either got Connection refused or sat until the 1,200-second HTTP timeout, and either way that step's reward was 0.0 for every rollout. On one day the shared verifier restarted three times in 80 minutes, and two of those restarts landed inside a GRPO run.

The grading itself was fine. The failure was availability, and the logs made it look like the model was failing: a reward of 0 from a dead verifier looks exactly like a reward of 0 from a body that doesn't compile.

A rolling update wouldn't really have fixed it either, as far as I can tell. Even with no downtime, the second run had moved the first run's grader to a different commit, so the first run's later rollouts were graded against code its prompts weren't built from.

I hit a smaller version of the same thing from the scheduler. The verifier asked for 2 CPUs, and on a full node the Recreate replacement sat in Pending, which also dropped it mid-run. Now it requests little and bursts to its limit while grading, since it's idle most of the time.

One verifier per run

The fix is boring: each run gets its own verifier, and nothing else can restart it.

• At run start, the orchestrator copies the live shared verifier's Deployment and Service under a per-run name. Copying the live object instead of writing a second manifest keeps the image, volumes and resources in step with whatever the shared one ships.
• The commit goes in as a literal environment variable at creation, not a reference to the shared config. A later patch to the shared config can't move it.
• The GPU box reaches the verifier over a WireGuard mesh VPN, and the copy gets its own hostname on it. The box pins that exact name in /etc/hosts. The old lookup matched by substring, and the shared verifier's name is a substring of every per-run one, so with both online it could resolve the wrong one.
• Kubernetes names are capped at 63 characters, and the replica's name adds about 17 characters of generated suffix, so the run-ID part of the name is capped at 30.
• Teardown runs in the one function every exit path goes through. A hard crash of the orchestrator can still leak a pair, so they carry a label, and one kubectl delete -l cleans them up.

There's a switch to go back to the shared verifier when I know nothing will overlap and want its warm cache.

What isolation cost: a cold cache

A fresh verifier starts cold, and in Rust cold is expensive. In two early measurements on cheap crates, the first grade against a cold target directory took 324 seconds, and the next one, on a different crate with the dependencies already built, took 21. Grading speed was already the bottleneck. The first verifier ran 4 grades at a time against one shared cargo target directory, and cargo takes a lock on the target directory while it builds, so concurrent grades mostly waited on each other. My estimate for one pass over the tasks at 8 rollouts each was about 22 hours of grading with the GPU idle.

The next version gave each of 12 workers its own target directory that stays warm between grades, and restricted RL to tasks in cheap-to-build crates (about 170 of the 297 targets I had then). Tiering crates by how many internal dependencies they have turned out to be a poor guide, since a crate with three dependencies could still take minutes. So the task list is now filtered by measured build and test times.

Then the cold start bit again. A brand-new per-run verifier cold-builds all 12 target directories at once while GRPO is hammering it. Cold, each one is about 22–26 GB with intermediate artifacts. That burst went past the 200 GB disk cap, and the kubelet evicted the verifier in a loop during the GRPO phase of a real run. The cap is now 320 GB, and the replica reserves its disk at scheduling time, since an emptyDir size limit isn't something the scheduler looks at.

Copying the live object also copied its bugs. After I moved git hosting, the shared verifier's config still pointed at the old, frozen repo. The shared one didn't notice, because its commit was old enough to exist there. The first per-run verifier was pinned to a new commit and crashlooped with fatal: unable to read tree.

A reward of 0.0 isn't "no signal"

The verifier's reward wasn't the only reward. TRL adds up several reward functions, and I'd kept two cheap ones from the scaffolding stage: 1.0 for a <think>…</think> block and 0.5 for a completion between 50 and 2,000 characters. The idea was to keep some gradient on batches where every rollout failed and to stop the output format from drifting.

That means when the verifier was down, the step didn't do nothing. The verifier part of every reward was 0, but the format and length rewards still differed between rollouts, so that step trained on format and length alone. As I read TRL's code, the KL penalty against the reference model also keeps producing gradient. And a format reward worth as much as a fully passing test is a big weight for something that easy.

It also broke the check I used afterward. A later review concluded the verifier had been reachable because the GRPO loss and gradient norms weren't flat at zero. With the format and length rewards and the KL term in there, a non-zero loss doesn't show that. The per-rollout rewards were never logged to my metrics, only the loss, so I have no direct record of how many steps got real grades.

Results

Two full runs went all the way through (continued pretraining, SFT, DPO, KTO, GRPO, then a short recovery SFT). Both ran after the scoring fix, with per-run verifiers.

Run 2's GRPO phase ran at about 417 seconds per step, so its 263 minutes is roughly 38 optimizer steps. That's not much RL. Run 2's first step blew up, and the model came out of GRPO damaged enough that recovery SFT started at a loss of 14.5. I don't know why the first step exploded. The second run is also the one that had the verifier eviction loop during GRPO.

Run 2's adapter is the one that got deployed. On Terminal-Bench 2.1 it solved 6 of 89 tasks against 22 for the base model. A lot of that gap was serving failures. About 30 tasks per run died on empty responses from a cold or preempted GPU endpoint, and mostly different tasks died in each run. On the 37 tasks both models actually worked through, the base solved 16 and the fine-tune 4. The investigation put most of that on the SFT data, which was conversational coding help rather than finishing tasks on its own. That's a regression for the whole pipeline. I can't pin it on the GRPO stage, and I can't credit GRPO with anything either.

I also never measured the thing GRPO was optimizing: the pass rate on the held-out functions, before and after the RL stage, on the same tasks. I couldn't find a recorded number for it. I set up one more short run to get a clean, instrumented GRPO step with per-rollout rewards logged, and it got stuck on GPU quota before a box was created, at no cost. So honestly: the RL loop ran, and I have no evidence it helped.

What I'd do differently

• Test the grader before the trainer. Feed it the correct body, an empty body, a body that compiles and fails, and one that returns a constant, and check the four rewards. That would have caught the zero-tests bug on day one.
• Log every reward, not just the loss: per rollout, with the status (pass, test failed, build failed, timeout, verifier unreachable). A verifier outage should be a different number from a wrong answer, even if the trainer gets 0 for both.
• Never let one job reconfigure a service another job is using. If a run needs a particular commit, it gets its own copy.
• Measure the target before running the expensive part. A before/after pass rate on a fixed set of held-out tasks costs a fraction of a GRPO run and would have answered the only question that mattered.
• Keep cheap shaping rewards small relative to the real one, or turn them off once the real one works.

Related posts

• Fine-tuning a 35B mixture-of-experts model on rented GPUs
• Agents write tests that can't fail. A lint catches some, mutation testing finds the rest