Cheap LLM serving on consumer Blackwell: NVFP4, FP8 KV cache and MTP speculative decoding

Ryle · Jul 14, 2026

For my side projects I run a lot of coding and research agents against a self-hosted model: Qwen3.6-35B-A3B, a mixture-of-experts model with about 3B parameters active per token, plus a rank-64 LoRA adapter I trained on it. I spent a while working out how to serve many of those agents at once as cheaply as I could, on vLLM and single Blackwell GPUs, mostly rented as interruptible instances. The short version is that one serve flag (FP8 KV cache) got me 21% more agent throughput, 4-bit NVFP4 weights let the model and adapter fit on a 48 GB card that rents for about a fifth of what I was paying, and speculative decoding helped less than I expected.

Why the KV cache caps concurrency

A transformer generates one token at a time, and each attention layer compares the new token against a key and a value vector for every earlier token. Instead of recomputing those for the whole history every step, the server keeps them in GPU memory. That stored history is the KV cache.

Its size per token is roughly 2 (K and V) × attention layers × KV heads × head dimension × bytes per element. For an example dense model with 64 layers and 8 KV heads of dimension 128, stored in BF16, that's 2 × 64 × 8 × 128 × 2 bytes = 256 KiB per token. A single 100k-token conversation would then need about 25 GiB of KV cache on top of the weights.

The model I serve is a hybrid. Three out of every four layers are Gated DeltaNet (linear-attention) layers that keep a fixed-size recurrent state instead of a growing cache, and only the remaining quarter are full attention layers that carry per-token KV. vLLM reports its KV geometry at 10.56 KB per token with FP8 KV. That's small, and it's a big part of why this model is cheap to serve at long context.

After loading weights, vLLM turns the rest of the GPU memory (up to --gpu-memory-utilization) into a pool of KV slots, and every active conversation takes up slots in proportion to its context length. Once the active contexts add up to more than the pool, the scheduler starts evicting cached prefixes or preempting requests, and throughput drops off hard. Agents are especially exposed to this, because each one resends its whole growing conversation every turn and relies on the prefix cache so it doesn't get reprocessed.

On a 96 GB RTX PRO 6000 serving the FP8 checkpoint with a BF16 KV cache, the pool held about 1.65M tokens. I benchmarked N concurrent multi-turn agents seeded around a 24k-token mean context, with a tail out to 120k:

Throughput peaks at 24 agents. After that it collapses, since the combined working set stops fitting and the prefix cache starts thrashing.

The rule of thumb I got out of this is optimal agents ≈ 0.5 × KV-cache tokens / mean agent context, where the one-half leaves room for contexts growing between turns and for each agent's prefix to stay cached. It comes out high: 1.65M / 24k × 0.5 gives about 34, and the measured knee was 24, so I'd sweep downward from the estimate. Also benchmark with realistic contexts. My first run used ~1.8k-token contexts, never touched KV pressure, and looked like it scaled past 256 agents.

FP8 KV cache: the free 21%

The first thing I changed was storing the KV cache in FP8 instead of BF16, which halves the bytes per cached element. It's one flag, --kv-cache-dtype fp8, and the weights don't change. On the same 96 GB card the pool went from 1.65M to 2.70M tokens (1.64×; I didn't dig into why it comes in short of 2×). The agent sweep:

Peak aggregate throughput went from 431 to 521 tokens per second (+21%) and the knee moved from 24 to 32 agents. It passed my quality gate at 1.0, so I didn't see any measured loss, and published long-context retrieval results on a similar Qwen MoE also hold up well under FP8 KV. I use it by default everywhere I serve now.

Two gotchas I hit: at --gpu-memory-utilization 0.95 the server ran out of memory during inference (0.90 is stable), and on this hybrid model --calculate-kv-scales silently corrupts output.

I ruled out the more aggressive KV ideas on this model for now. Sub-4-bit KV quantization and CPU/NVMe offload both run into vLLM's page allocator limits for hybrid models. Sparse attention doesn't have much upside when only a quarter of the layers carry a cache, and sliding windows on those layers would remove the long-range recall agents need. So after FP8 KV, getting more agent throughput on one card means more KV memory or smaller contexts.

NVFP4: fitting on a cheaper card

NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell. Each weight is a tiny 4-bit float, and every block of 16 weights shares an FP8 scale factor (with one more scale per tensor), so the format keeps a useful dynamic range while costing a little over 4 bits per weight. Blackwell tensor cores run it natively. 4-bit NVFP4 checkpoints of my base model are published in two flavors: a regular one, and a "Fast" one that also quantizes activations to 4 bits (W4A4).

What I wanted to know was whether the base model plus my existing adapter could move off the 96 GB tier, which was costing me about $0.80–1.00/hr. The memory budget explains how that went:

• Weights: about 25.5 GB in VRAM for NVFP4 and 23.2 GB for Fast. That's more than the ~17.5 GB you'd get from 35B × 4 bits, because the checkpoint keeps embeddings and the router at higher precision.
• LoRA: vLLM preallocates about 5 GB of buffers for a rank-64 adapter that covers every expert.
• On a 32 GB card at 0.90 utilization you have about 29 GB, so weights plus adapter don't fit. The RTX 5090 ran out of memory at adapter initialization, so on that card I could only run the base model.
• On a 48 GB RTX PRO 5000 the same arithmetic leaves room. After weights, adapter, activations and CUDA graphs, vLLM had an 8.8 GiB KV pool at 64k context, which it reports as 12.6× concurrency at full 64k.

That 12.6× is easy to check: 8.8 GiB at roughly 10.56 KB per token is around 0.9M tokens, or about 13 full-length 64k conversations. The full 262k context also fit, with about 3.5× worst-case concurrency, and a real 246k-token request finished in about 41 seconds.

Single-stream decode speed (512-token generations) on the cheap cards:

CUDA graph mode matters a lot here. Eager mode on the 5090 only managed 19.9 tok/s. The Fast variant was 35% faster than regular NVFP4 with the same piecewise setting (218 vs 162) and 2 GB smaller. I haven't measured its accuracy, though, and 4-bit activations are where I'd expect a quality cost to show up.

For fleet throughput I ran a concurrency sweep with prompts around 5k tokens. A 96 GB card limited to a 44 GB budget, to emulate the 48 GB card, gave 647 output tok/s at 16 concurrent requests against 700 with the full budget, which is within noise. At that context length the limit is GPU compute. On a real PRO 5000 at 64 concurrent requests, utilization sat at 99% and power at the ~300 W cap, while the API server used 16% of one CPU core. The adapter costs about 10% of aggregate throughput.

What it costs

The 96 GB tier at $0.80–1.00/hr against the 48 GB card at $0.16–0.19/hr works out roughly 5× cheaper for workloads that fit in 64k context. As an inferred figure, 647 tok/s is about 2.3M output tokens per hour, so around 7 cents per million at $0.16/hr. That's before cold starts (I measured 9 to 21 minutes from rental to serving), idle time and preemption. The bigger caveat is quality. The adapter runs coherently over the 4-bit base and its parallel tool calls parse cleanly, but I haven't run the eval bench yet against the FP8 serve it was trained on.

MTP speculative decoding

Decoding one token for one request leaves most of a GPU idle, because each step does very little math and mostly just reads weights. Speculative decoding uses that slack. Something cheap guesses the next few tokens, the big model checks all the guesses in one forward pass, and it keeps the prefix that matches what it would have produced anyway. The output doesn't change, and when the guesses are right you get several tokens per expensive step.

MTP (multi-token prediction) means the guesser is a small extra head trained alongside the model to predict tokens further ahead, so you don't need a separate draft model. With two speculative tokens, each step can give up to three tokens. On the PRO 5000 with the adapter, acceptance was 69–88%, about 2.4–2.8 tokens per step.

Two results surprised me. The first was that draft-head precision matters. My FP8 checkpoint had quantized the MTP head too, and acceptance sat at 1.79 tokens per step; splicing the BF16 head back in raised it to 2.56. The second was that this didn't turn into fleet throughput. At the 32-agent peak with FP8 KV, the fixed head gave 512 tok/s against 521 before. In the NVFP4 runs, single-stream speed with MTP was also lower (127.6 vs 144.2 tok/s with the adapter; 204.1 vs 227.0 on the 5090, where MTP also squeezed the KV pool to 0.63 GiB). I haven't profiled why. My guess is that a busy batch already keeps the GPU occupied, so the draft and verify work only adds cost. So I keep MTP for latency-sensitive, low-concurrency serving and leave it off for agent fleets.

This is the serve shape I use on the 48 GB card:

vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4 \
  --max-model-len 65536 --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 --max-num-batched-tokens 8192 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-lora --max-lora-rank 64 --lora-modules ft=<adapter-dir> \
  --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-auto-tool-choice
# optional, for latency: --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
# 32 GB cards: add --max-num-seqs 32 and cap max_cudagraph_capture_size at 32-64

Context compression

The formula points at the other lever. With a fixed pool, halving mean agent context roughly doubles the number of agents that fit. I haven't measured compaction on my setup yet, so this part is just from reading. AgentDiet, which trims a coding agent's trajectory at inference time, reports 39.9–59.7% fewer input tokens and 21.1–35.9% lower total cost with no loss in task performance. LLMLingua-style token pruning reports up to 20× prompt compression with little accuracy loss, and retrieval that only sends relevant chunks commonly reports 60–80% token reduction. All of those act on the denominator of the formula.

Caveats

Most of these cells are one run on one box. The agent benchmarks ran on vLLM 0.20 with the FP8 checkpoint, and the NVFP4 work on a 0.25 release candidate. I still haven't done quality evals of the 4-bit serve or of NVFP4-Fast, and the cost figure assumes cheap interruptible capacity is available, which sometimes it wasn't.