Speed: tokens per second and first-token time

How to read this page

A cell is one prompt length (1K to 128K tokens), one content type (prose, code, reasoning, JSON) and one number of streams (requests in flight at once), measured 3 times; each request generates 1,024 tokens with thinking on, every prompt is unique and every request has its own seed. Decode is the per-stream generation speed, aggregate all generated tokens of a batch per second of wall time, first token the time from sending to the first generated token. With thinking on, the code and JSON cells measure reasoning text about a code or JSON prompt.

K3.25 speed noise: a repeat hours later moved the median by under 2% and single cells by up to 13%; single-cell ratios within ±10% read as level.
K4 speed noise: a ±8% band declared before the run (both engines' drift repeats stayed inside it); single-cell ratios within ±8% read as level.

K3.25 (2026-10-26/27)

CuteAFD v3.0.1: 1,008 measured requests in 132 batches, 0 errors, 0 replies with </think> in the content; vLLM fork v0.9.2: 1,008 measured requests in 132 batches, 0 errors, 0 replies with </think> in the content. Ranges run over the content types; first token is the median over them. Full grid: t/T-SUMMARY.md.

K4 (2026-10-27)

The K4 grid is smaller: 1 and 4 streams at 1K-32K and 1 stream at 128K, set once for every arm of its frozen protocol. Readings per cell use the declared band.

vLLM: 264 measured requests, 0 errors; CuteAFD: 264 measured requests, 0 errors. Prompt-token counts identical on every cell except the 128K code cell (12 CuteAFD requests longer: those prompts hold literal <think> text, which vLLM parses as a special token).

Caveats

• Long prompts with several streams mostly measure scheduling. The vLLM fork interleaves chunked prefill with decode, so its streams' decode overlaps other requests' prefill (a median 40-81% of each decode window) and their first tokens come early; CuteAFD finishes each prefill faster but queues the requests behind it. Streams that decode without another stream's prefill running decode at the same speed on both engines. At 32K and 128K, 71-92% of a batch's wall time passes before its last first token, so the aggregate there mostly reflects CuteAFD's faster long prefill (about 4,400-4,500 against 3,700-3,900 tokens/s).
• Draft policies differ. The vLLM fork drafts 3 tokens per step with a BF16 drafter; CuteAFD drafts adaptively with an FP8 drafter. The vLLM fork accepted 1.53 of 3 drafted tokens per step; CuteAFD exposes no counter.
• Inputs differ in two details. Each request's first line is a nonce, different between the engines on K3.25 (prompt bodies identical); the 128K code prompts hold literal <think> text that the vLLM stack parses as control tokens.

More: t/CAVEATS.md, the drift check drift/RESULTS-DRIFT.md, the K4 per-cell readings t/t-compare.json.

K4 at 8 and 16 streams (follow-up K4f)

The frozen K4 grid stopped at 4 streams; this follow-up ran the cells both engines admit, outside the protocol, with the same prompts and seeds.

Design FOLLOWUPS-K4f.md (declared before any request). The cells the frozen grid dropped that both engines admit, on the protocol's Block T prompts, seeds and nonce salt; one serving launch per arm; no request errors. Prompt-token counts identical except the 128K code prompts on CuteAFD (literal <think> text, as in the protocol).

Machine state. Drift subsets after the vLLM and CuteAFD side cells matched 2026-10-27: decode 0.95-1.03x on both engines; time to first token within 2% except one vLLM cell (1.12x).

Medians over content types and replicates; 1,024 output tokens per request; per-stream decode = tokens per second of one request; aggregate = all tokens finished per second.

Reading (±8% band):

• Throughput under load: level at 1K × 8 and 8K × 8; CuteAFD ahead at 16 streams (1.37-1.44x), at 32K × 8 (1.09x) and 128K × 2 (1.15x). On K3.25, 1K × 8 favoured the vLLM fork by 7-15%; on K4 it is level.
• First token under load: the vLLM fork answers short prompts sooner with many requests in flight (1K: 1.8-1.9x sooner); level at 8K and 128K × 2; the vLLM fork sooner at 32K × 8 (1.33x). Mechanism as on K3.25: the vLLM fork interleaves chunked prefill with decode; CuteAFD's prefill is faster per request but queued requests wait.
• W8A16 + amax matches CuteAFD as shipped at high concurrency: aggregate 0.97-1.00x, first token +4% in every cell.
• 32K × 16 not run: CuteAFD's K4 KV pool (447,488 tokens) does not hold 16 × 33,792.

Files: t/V-side.json, t/C-side.json, t/C-w16amax-side.json, t/{V,C}-f-drift.json, K4f-cells.json.