Fine-tuning a 35B mixture-of-experts model on rented GPUs

Ryle · Jun 29, 2026

For a side project I've been LoRA fine-tuning Qwen3.6-35B-A3B, a mixture-of-experts model with 35B total parameters and about 3B active per token, on a tool-calling SFT dataset plus a few preference phases. I started on AWS spot instances with one RTX PRO 6000 Blackwell (96 GB) and later moved to rented boxes on vast.ai. Most of what I learned had little to do with which GPU was fastest. The recipe ended up setting the speed more than the card did. Separately, one library import quietly turned off training for the routed experts, and most of my rental problems came from hosts that looked fine in the listing.

The setup

The base model stays in BF16, unquantized. I use LoRA rank 32, alpha 64, on attention and on the routed experts, with batch 2, 8 steps of gradient accumulation and 2,048-token packed sequences. The expert layers use a LoRA-aware grouped-matmul backend registered with transformers, so all the experts' base and LoRA matmuls run as grouped calls instead of a Python loop. A run is a cascade of phases (continued pretraining, SFT, then preference phases including DPO, KTO and GRPO), and each one checkpoints to object storage so a new box can resume.

B200 and H200 were the same speed

I benchmarked the same recipe on three cards to decide whether it was worth leaving AWS. Steady-state step times on the continued-pretraining phase:

Both vast.ai cards came out about 1.55x faster than AWS, and the B200 matched the H200 within noise. On paper the B200 has roughly 2.3x the dense BF16 compute of the H200 and well over 1.5x its memory bandwidth, so whatever set the step time didn't scale with the card. The one part that did scale was weight loading. The H200 pulled the 67 GB of BF16 weights in about 13 seconds, against 30+ on AWS.

My suspects at that point were kernel launch overhead in the grouped experts path, attention falling back to plain PyTorch because flash-attn wasn't in the image, and the PCIe variants of both cards. I never isolated how much each one cost. The cost side was clear, though. For a full run, AWS spot came to about $18 (~30 hours at $0.60), the H200 about $24 (~20 hours at $1.20), and the B200 about $76 (~20 hours at $3.80). Faster hardware didn't buy me anything until the software changed, so I stayed on the cheap card.

The fallback on Blackwell workstation cards

Later on, a different version of the recipe (SFT at rank 64 with 16 accumulation steps, so the numbers below don't line up with the table above) ran at 112 s/step on an RTX PRO 6000 at 32 to 55% GPU utilization, against 28.5 s/step and about 80% on a B200. My first theory was an FP8 upcast on every forward pass, and that was wrong, since the run trained the plain BF16 base.

The real cause was in PyTorch. Through 2.11, the CUDA grouped matmul only takes its CUTLASS fast path on sm_90 (Hopper) and sm_100 (B200). The RTX PRO 6000 is sm_120, and on that architecture it silently runs a fallback: it copies the group offsets to the CPU, which forces a device-to-host sync, and then runs a serial loop of ordinary matmuls, one per group. I timed it directly at the MoE shapes:

H200 and B200 are both on the fast path, which fits with them tying in the earlier benchmark. I believe the AWS card was on the fallback then too, but I didn't instrument that run.

First I tried building PyTorch 2.11 with an open upstream PR that enables the CUTLASS path on sm_12x. On Linux sm_120 it failed at kernel initialization with two CUTLASS versions. Its schedules use tcgen05 MMA instructions that sm_120 doesn't have, so a real fix has to wait on an unmerged CUTLASS change. The wheel just turned the slow fallback into a runtime error, so I never deployed it.

What did work was the Triton grouped GEMM kernel that ships inside unsloth, on stock PyTorch. Forward was 0.72 ms, 4.9x faster than the fallback, and a full MoE-LoRA layer forward plus backward went from 22.9 ms to 8.0 ms. Numerics and gradients matched the reference path, including empty and uneven expert groups. On a real run the SFT phase dropped from 112 to about 57 s/step. At $1.23/hr that's about $0.020 per step against $0.041 on a B200 at $5.20/hr, so I went back to the 96 GB card as the default and kept the B200 for when wall-clock time matters.

An import that froze the routed experts

Then I found out the routed experts hadn't been training. A full seven-phase run of about 33 hours finished cleanly, and the attention LoRA file changed every phase. The routed-expert LoRA file, about 7.4 GB, was byte-identical after continued pretraining, SFT, DPO and KTO, and only changed at GRPO. The training metrics agreed: exactly 80 of 700 LoRA B matrices stayed at their zero initialization through those four phases, then dropped to 0 during GRPO. Those 80 were the routed experts.

I got the diagnosis wrong at first. GRPO was the one phase pinned to the PyTorch path and the frozen phases had used Triton, so I blamed the kernel's weight gradient and forced everything back to the slow path, even though the kernel had already passed a gradient-equivalence test.

The actual cause was the import. Pulling in the Triton kernel with from unsloth.kernels.moe.grouped_gemm.interface import grouped_gemm runs unsloth's patching pass, and that pass replaces the Qwen MoE experts' forward with its own dispatcher. That dispatcher never looks at the transformers registry where my LoRA-aware backend was registered. So the base expert matmul still ran, the LoRA delta was never computed, and the routed LoRA parameters got exactly zero gradient. GRPO only trained because its code path never ran that import.

A two-process test on a tiny Qwen MoE model settled it. Without the import, my backend ran and routed LoRA B had a nonzero gradient in 4 of 4 checks. With it, my backend ran zero times and 0 of 4 had any gradient. The check I should have done earlier: with unsloth loaded, setting LoRA B to nonzero values didn't change the loss, which means the delta was missing from the forward entirely.

The fix re-installs my experts forward after the import and makes Triton the default on sm_12x again. A 20-step overfit on a fixed batch then grew the routed LoRA B norm from 4.9 to 26.2 and took the loss from 8.0 to about zero. I already had a gradient sanity check, and it missed all of this because it summed over every trainable parameter, so the healthy attention LoRA kept the total above threshold. Now it checks the routed experts separately and aborts at step 1.

One consequence I haven't verified: the 57 s/step figure came from the same import path, so it almost certainly ran without the routed LoRA math, and the real step time should be somewhat higher. I haven't remeasured it yet.

The rental tooling

I ended up writing a few small Rust tools against the vast.ai API. The fine-tuning launcher searches offers, rents one box, drives the phases over SSH, and destroys the box when the run fails. A fleet manager for inference boxes ranks verified offers by a rough throughput score per dollar-hour. A scheduled reaper destroys anything past a maximum age (12 hours by default), anything with no training or serving process after a 30-minute grace period, and serving boxes that fail three probes of /v1/models, since a hung vLLM still looks alive to a process check. It also caps total account spend per hour by destroying the newest boxes first.

Most of the filters in the offer search are there because a box failed without them:

• Match on VRAM. GPU name strings vary ("H200" against "H200 NVL"), so each tier is a memory range: 90–100 GB for the RTX PRO 6000, 130–150 for H200, 170–190 for B200.
• Hard price cap. Picking the cheapest rentable offer got me $1.90–2.50/hr boxes three times while $1.00–1.20 offers were listed. The default cap is now $1.30/hr.
• Driver version. The training image ships CUDA 13 PyTorch. A host whose driver tops out below that rents fine and then fails CUDA init with error 803.
• Disk. 250 GB minimum, for the 67 GB base download plus a merged BF16 copy (~67 GB) and an FP8 copy (~35 GB) for evaluation.
• Bandwidth both ways. The 67 GB prefetch has a 1,500-second budget (about 360 Mbps), and the downlink floor is 250 Mbps so the pool doesn't empty. I added the 200 Mbps uplink floor after one host uploaded the 7 GB expert checkpoint at 0.38 MB/s, which projected to 5 hours of idle GPU.
• Location. Advertised bandwidth is local. A US host that delivered its full 167 Mbps locally pushed to my European object store at about 30 Mbps. With checkpoints in Europe the search only takes European hosts, and with checkpoints in a US bucket the filter is off.
• Reliability 0.95. A 0.99 floor excluded nearly the whole cheap tier, since most cheap hosts sit between 0.95 and 0.99.

Lessons about renting

Rent on-demand for long stateful jobs. Interruptible (bid) instances are much cheaper, for example $1.34/hr against $2.93/hr for an H200 in Europe on one day, but I got preempted twice in a single session mid-build, and a reclaim in the middle of a phase means redoing that phase. For multi-day runs the redo cost was more than the savings. For short smoke tests the bid tier is fine. Also check what you actually rented: vastai show instances --raw reports is_bid.

Don't trust remote exit codes. In one case an unresolvable pin made pip install nothing, a | tail ate the return code, the SSH wrapper ignored it anyway, and three boxes in a row died later with No module named 'peft'. Now the install step ends with an import probe that has to print a sentinel. A related one: pkill -f <pattern> matches the SSH shell running it, since the pattern is in its own command line. Bracket a character (pgrep -fc "[b]uild-torch").

Don't share the training box. Compiling PyTorch on the same B200 at 96 parallel jobs slowed training from 42 to 108 s/step through memory bandwidth contention, and nice didn't help.

Make teardown the default. Every failure path destroys the instance unless a debug flag says to keep it around for SSH debugging. Check that kind of flag by value. Mine was checked for presence, it got left set in an env file, and failed boxes kept billing until someone destroyed them by hand.

The whole sm_120 investigation cost about $5–6 in rentals and a lot more in hours. The GPUs were the cheap part. Most of the time went into trusting a number, an exit code, or a gradient check before I'd verified what it actually measured.