Ryle Goehausen

Single-user speed vs total throughput: reading inference-chip claims

Ryle · Sep 25, 2026

Every so often another inference accelerator announces a number with "tokens per second" after it, and I've seen those numbers range from a few hundred to tens of millions. Mostly they're measuring different things. A rack serving ten thousand conversations at 100 tokens per second each, and a chip running one conversation at a million tokens per second, would both get described as "a million tokens per second," even though they're very different machines. So I went through the public claims from the fast-inference specialists that were out when I wrote this, and sorted them by what they actually measure. In short, the best backed-up single-stream result I could find is about 17,000 tokens per second on a heavily quantized 8B model, and I couldn't find anything with public evidence that gets close to a million tokens per second for one user.

Five numbers that all get called tokens per second

A "token" is the unit a language model reads and writes, roughly three quarters of an English word. Generating text has two phases. Prefill reads the whole prompt in one parallel pass and builds the model's working state (the key/value cache, or KV cache). Decode then produces output one token at a time, and each new token depends on the one before it. The numbers vendors publish cut across those two phases in different ways:

MetricWhat it countsWhat it tells you
Output tokens/s per user (per stream)Accepted output tokens in one dependent sequence, per second of decodeHow fast a single answer, agent loop or reasoning chain advances
Aggregate output tokens/sThe sum across every sequence running at once on a server, rack or clusterHow many users or jobs the hardware can serve
Output tokens/s per MWAggregate rate divided by system powerThroughput per unit of energy; says nothing about one user's latency
Prefill tokens/sInput tokens processed while building prompt stateHow fast long prompts are ingested, which is a separate job from writing the answer
Time to first token (TTFT)Queueing, scheduling and prefill before the first output token appearsHow responsive the system feels at the start of a request

Most of the confusion comes from some simple arithmetic. Ten thousand streams at 100 tokens per second each adds up to one million aggregate tokens per second, but each of those streams still takes about 100 seconds to write 10,000 tokens. Adding users raises the aggregate and doesn't help any single user, and you can't split a serial chain of reasoning across users.

Prefill numbers are also much bigger than decode numbers on the same hardware, since the whole prompt can be processed in parallel. A big prefill figure matters for long-document workloads, but it doesn't say how fast the answer comes out.

Why one stream is the hard number

Aggregate throughput scales with hardware, since more chips means more conversations. Single-stream speed doesn't scale like that. Each token has to pass through every layer of the model in order, and the next token can't start until this one is chosen. When I built a visualizer for one decode step, the critical path came out to thousands of dependent operations per token with an average parallelism of about 1.5 between operations. The parallelism is all inside each operation. So per-user speed comes down to how short each of those serial steps can be made, including the time to read the weights and the cached history each step needs, and making the hardware wider doesn't shorten that chain.

There are two things I watch for in single-stream claims. One is that a single stream can span many chips. "Per user" describes the sequence, and a vendor might spread one conversation across a whole rack to get the number. The other is speculative decoding, where a small draft model guesses several tokens ahead and the full model checks them in one pass, so more than one token can be accepted per step. That's a real speedup, but the fair count is accepted tokens from the target model, and the vendor should say that speculation was on.

I found it useful to turn rates into time per token. At 17,000 tokens per second, the average token takes about 59 microseconds. At 1,000 tokens per second it's 1 millisecond. A million tokens per second for one stream would mean one microsecond per token, sustained, across the whole model. With speculative blocks that can be an amortized average rather than one full forward pass per microsecond, but that's still what it would take.

Sorting the claims

I split the public figures into three groups: measured single-stream results, aggregate claims (measured or modeled), and targets for hardware that isn't available yet. The rows are grouped by metric, not ranked, because the model, precision, context length and hardware scale are different on every row.

PlatformDisclosed figureWhat it measures
Single stream, measured
Taalas HC1~17,000 output tokens/s per user, Llama 3.1 8B; EE Times saw 15,000+ on the public demoOne stream on a model hardwired into silicon, 3- and 6-bit weights, vendor benchmark at 1k in / 1k out. Vendor calls it a technology demonstrator.
NVIDIA Groq 3 LPX3,431 median output tokens/s at 100K context, Gemma 4 31BOne stream, measurement attributed to Artificial Analysis in NVIDIA's report. Separately 4,767 median and 5,520 P80 on SPEED-Bench coding tasks.
SambaNova SN50Up to 850 tokens/s short context, over 450 long context, MiniMax M2.7One stream, attributed to Artificial Analysis. Four H200 GPUs do prefill, 16 SN50 chips do decode.
Aggregate
d-Matrix Corsair60,000 tokens/s per server at 1 ms/token (Llama 3 8B); 30,000 per rack at 2 ms/token (Llama 3 70B)Aggregate, labeled preliminary in the Nov. 2024 announcement. The stated token latencies imply roughly 1,000 and 500 tokens/s per stream (my arithmetic).
Etched Sohu500,000+ tokens/s, Llama 70B, one eight-chip serverAggregate server throughput, June 2024 company claim. Not a 500K tokens/s conversation.
Euclyd CRAFTWERK CWS 327.68M tokens/s at 125 kW, Llama 4 MaverickMulti-user aggregate, explicitly "modeled performance," Sept. 2025.
Tensordyne Napier363,000 tokens/s per rack, ~3M tokens/s per MW, DeepSeek-R1 FP4, at 210 tokens/s per userSimulated aggregate, reported by EE Times in June 2026. Dev cloud planned for end of 2026, systems Q2 2027.
Targets and ambitions
Cerebras CS-5Up to 10,000 tokens/s per user (Gemma 4 31B, gpt-oss-120b); up to 5,000 per user and 3M tokens/s per MW on multi-trillion-parameter models2027 design targets from the Hot Chips 2026 roadmap, not measured CS-5 results.
Euclyd CRAFTWERK SiP20,000 tokens/s for a single user, Llama 4 MaverickCompany projection reported by EE Times in Sept. 2025, before silicon existed.
Fractile"Thousands of tokens per second to thousands of concurrent users"; "up to 25x faster"Per-user plus concurrency ambition, without public model or configuration details.

Once they're grouped like this, the big numbers make more sense. Every figure in the hundreds of thousands or millions is aggregate throughput or throughput per megawatt, and most of those are modeled or simulated. Tensordyne's EE Times coverage is one of the more useful disclosures because it gives the rack number together with a per-user rate: 363,000 tokens per second per rack at 210 tokens per second per user. That's a throughput machine running lots of conversations at a modest per-user speed, which is a reasonable thing to build.

The measured single-stream numbers sit in a much narrower range, from hundreds to a few thousand on real mid-size models, with Taalas well ahead on a small one. Taalas does it by fixing one model into the chip with very low-precision weights. The vendor acknowledges some quality loss relative to GPU benchmarks, the chip supports LoRA adapters and a configurable context window, and changing the base model means new silicon. It's a strong result, and it shows what extreme specialization can get you. It's also an 8B model at a 1k/1k setting, so it doesn't say much about a frontier reasoning model at long context.

Groq 3 LPX stood out to me for a different reason: 3,431 tokens per second at 100K context on a 31B model is a long-context single-stream number with a third-party measurement attached. Long context is where per-user speed usually collapses, because each token has to read an ever-larger KV cache. The SambaNova result is an example of the hybrid setup several vendors seem to be moving toward, where GPUs do prefill and specialized parts do decode.

I also ran into the opposite problem. I found third-party write-ups describing Etched's 500,000 figure as a batch-one result. The original disclosure describes eight-chip server throughput, and a total that large is normally the sum of many concurrent requests. I also found an article claiming a 17,000 tokens per second single stream on a consumer GPU that still had an editorial "link required before publication" placeholder in the text, and I didn't count it.

Is anyone near a million per user?

Not as far as I can tell. I didn't find anything with public evidence that gets near a million tokens per second for one stream of useful model output. The highest measured single-stream figure is about 17,000, which is roughly 60 times short of that. The highest single-stream projections on the list, Euclyd's 20,000 per SiP and Cerebras's 10,000 per user, are design targets for hardware that hasn't shipped.

There also isn't a meaningful universal maximum unless you set a floor on model size and quality. A tiny model putting out trivial output can run very fast, and that doesn't tell you anything about running a capable model. Comparing Taalas's quantized 8B against a multi-trillion-parameter target, or Euclyd's mixture-of-experts workload against dense Llama 70B, mixes up the model with the machine. Peak FLOPS and memory bandwidth on their own don't determine tokens per second either, and Euclyd's 8 PB/s per package is a memory specification and doesn't by itself establish a token rate.

What I want to see in a claim

When I read one of these announcements now, this is what I look for before I take the headline number seriously:

  • Model and version, tokenizer, precision, and some quality check against a reference.
  • Input and output lengths, and context length.
  • Accepted output tokens per second per stream, at a stated concurrency, with speculation disclosed.
  • Aggregate throughput as a separate number from per-stream speed.
  • TTFT and a latency distribution alongside any best case. A P80 figure describes the 80th percentile and is neither a maximum nor a guarantee.
  • How many chips or racks one stream occupies, sustained duty cycle, and wall-plug power.
  • Whether the number was measured on silicon, measured by a third party, or modeled.

Most announcements give two or three of those. That's fine for marketing, but it means the number answers a narrower question than the headline makes it sound like. For capacity planning, aggregate tokens per second per megawatt is the metric I'd use, and the simulated rack figures are at least aimed at that. For agents and long reasoning chains, where one sequence has to get through many thousands of dependent tokens, per-stream speed is what decides wall-clock time. From what I found, that number is in the hundreds to low thousands on real models, with one specialized 8B demonstrator at 17,000.

Sources