Method: what was frozen, what ran, in what order

What was measured, in plain words

• Precision (Block Q). Each engine reads fixed texts token by token (teacher-forced), and its next-token probabilities are compared with the BF16 model's on the same positions: P-A, brandonmusic's BF16 teacher logits on 25 texts of 2,048 tokens; P-B, tpurtell's fidelity set, 64 prompts up to 14.4K tokens scored on the model's own replies; Q2, six long human-written documents at 32K-128K tokens, where no BF16 reference exists and the engines are compared with each other.
• Speed (Block T). Prompts of 1K to 128K tokens in four content types, 1 to 16 requests at once (K4: 1 and 4), 1,024 tokens generated per request, thinking on, 3 replicates per cell, a distinct seed and a unique first line per request.
• Long output (Block L). Replies up to 128K tokens with machine checks: hidden tests for code, the reference answer for math, a schema for JSON, the requested length for prose.
• Arms. Each engine as its author ships it, plus a short list of local settings (listen address, request logging, output limit, GPU memory utilization, a launcher guard); the same checkpoint bytes for the experts on both engines.

K3.25: frozen protocol and order

• Frozen. Protocol v2 with amendments A1 and A2, frozen on 2026-10-26 at 17:32 (UTC−7), before any speed (Block T) or long-output (Block L) request.
• Order as run. CuteAFD's precision captures began at 16:32, before the freeze, as amendment A1 declares (the instruments were fixed and no vLLM result existed yet); then the vLLM fork's precision captures (17:47-18:40); then speed and long output on the vLLM fork first, CuteAFD second. The protocol had planned CuteAFD, the vLLM fork, then CuteAFD again as a drift check; the drift repeat that ran instead covers both engines and was declared after the audit, before any of its requests.
• Follow-ups. F1 (CuteAFD's FP8 switches), F2 (the vLLM fork's prefix cache) and F3 (a second vLLM launch for precision) were declared after the precision results were known; they are outside the frozen protocol and reported separately.
• Review. After the runs, a separate reviewer who did not run the measurements re-ran the scorers and checked the claims against the receipts (read-only, CPU only, 2026-10-27): the precision and speed blocks and the vLLM fork's long output. Its corrections are applied. It covers K3.25 only.

K4: frozen protocol and order

• Frozen. A second protocol, frozen on 2026-10-27 at 17:58:09 (UTC−7) after the startup and smoke checks and before any comparison request; generation 1, no amendments. Every driver refuses to send a request unless its own hash and the grid's match the freeze record.
• Order as run. vLLM launch 1, CuteAFD's scoring launch, vLLM launch 2 for precision (the two vLLM launches bracket CuteAFD in time); speed on the vLLM fork, then CuteAFD, then a final vLLM drift subset; long output on the vLLM fork, then CuteAFD.
• Measures. Precision was captured as each engine's top 20 tokens per position and scored with top-20 measures, identically on both arms.
• Follow-ups. F1-K4, F1-K4b, K4c and K4d (CuteAFD settings) were declared after the precision results, each before its own requests, outside the frozen protocol.
• No separate review of any part of the K4 run.

The frozen K4 protocol also included a third engine, not reported here; its memory limits set the concurrency grid and the long-output cap.

Which vLLM fork

"vLLM fork v0.9.2" is tpurtell's v0.9.2 release image (digest ea0430aa…) started with the v0.9.0 repository's start.sh through its IMAGE override (the launcher is otherwise unchanged); the K4 launches set MODEL_PROFILE=k4 in the same launcher.

The protocol files

• protocol/PROTOCOL.md: the K3.25 protocol as frozen (its status lines were written before the freeze and left unchanged, so the file still matches its frozen hash)
• protocol/FROZEN.json, protocol/HASHES.json: the freeze record and which published copies are byte-identical to it
• protocol/FOLLOWUPS.md: follow-ups F1-F3 and the drift check, as declared
• METHOD.md: the K4 method, restated for the two published arms
• f1k4/: the K4 follow-up declarations (FOLLOWUPS-K4*.md) and results

What was scored

The K4 run scored the same panels and positions with top-20 measures. Teacher dataset identity and hashes: q/pa/PROVENANCE.json.

Setup

Hardware. 2× NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (300 W), driver 595.84, Linux 7.0, IOMMU in pass-through mode. Besides the engine under test, the only processes on the GPUs were desktop graphics clients (about 2 GB on GPU 1), the same for both engines.

Block Q additionally needs each engine's scoring path: CuteAFD's bench endpoint (ENABLE_BENCH=on, an API key, FULL_PREFILL_LOGITS=on), and on vLLM a prompt-logprob capture plugin (repro/stage0/capture-vllm). The FP8 projections in Core-FP8 are reproducible from K3.25-v1's BF16 tensors with amax/448 128×128 block scales (99.99998% of E4M3 codes equal): results/2026-10-26-cuteafd-setup-checks. Exact launch files: repro/cuteafd/ and repro/REPRODUCE.md. GPU telemetry from 2026-10-27 01:06 on (after both engines' Block T): both GPUs at the 300 W cap while busy, GPU 0 median 1,890 MHz and 87 °C (telemetry/).

The K4 measurement used the same engines, images, drafter and machine with brandonmusic's GLM-5.3-Flash-tr3-4bpw (brandonmusic/GLM-5.3-Flash-tr3-4bpw @ a5fee929) on both arms; its arms, declared differences and settings are in METHOD.md.

Confounds

What else differs between the two arms, and how each block handles it.

Fixtures are not comparable across model families, and NLL is per tokenizer; every number here is one model family and one tokenizer.

Limits

Two prompt-set sizes are small (Block L n=2 per prompt; Q2 6 documents); teacher-anchored precision stops at 14.4K; decode-shaped scoring not run; CuteAFD scoring cannot run under load; vLLM Block T ran before GPU telemetry began.

Timeline

Chronological, times UTC−7. K3.25 checkpoint first (results/2026-10-26-cuteafd-vs-vllm), then K4.

2026-10-26:

• 14:14-14:47. CuteAFD v3.0.1 cloned at its release tag; local config and the infiniband guard; Core-FP8 checkpoint byte-checked against K3.25-v1 (expert bytes identical) and its FP8 projections checked against K3.25-v1's BF16; source audit; first start, ready in 89 s.
• 15:04. Router route test on CuteAFD (one GPU, about a minute; exploratory).
• 15:56-16:38. Zero-GPU discovery for a higher-fidelity comparison; prompt sources fetched with revisions, licences and hashes; the Stage 0 instruments built and tested on CPU.
• 16:32. CuteAFD Block Q captures begin, before the protocol is frozen (declared in amendment A1: instruments fixed and not tuned, no vLLM precision result yet).
• ~17:15. Amendment A1. Amendment A2: thinking on for every Block T and L request, sampling from the checkpoint's generation config, prompt set and drivers frozen by hash.
• 17:32. Protocol frozen (protocol/FROZEN.json).
• 17:47 to before 18:35. vLLM Block Q captures (capture plugin, then the normal server restored); Block Q scored on CPU by 18:40.
• 17:59. Thinking-off check on vLLM (one request): the same 147-token prompt with thinking on and off; with it off, the reasoning parser returns the generated reasoning as content.
• 18:35-20:35. Block T on vLLM: 1,008 measured requests, 0 errors.
• 20:36-22:39. Block L on vLLM, stages L1 to L3s, 100 requests, 0 errors; analysed.
• 22:39. Block T on CuteAFD started.
• 23:20. Follow-ups F1 (CuteAFD FP8 ablation) and F2 (vLLM prefix cache) declared, after the Block Q results were known; they are labelled as outside the frozen protocol and are reported separately.

2026-10-27:

• 00:29. Block T on CuteAFD finished (started 22:39): 1,008 measured requests, 0 errors.
• ~00:30. F1 amendment A1: a speed subset per F1 arm, declared before any F1 request.
• 00:31-01:40. F1: four CuteAFD scoring launches (KDA projections, head, prefill GEMMs, all three) on panel P-B, each with its speed subset, plus a speed baseline on the unchanged scoring launch.
• 01:04. Separate review of Blocks Q, T and vLLM's Block L by a reviewer who did not run the measurements, read-only and CPU only (AUDIT-2026-10-27.md). Results it established, written at the top of q/RESULTS-Q.md:
  • vLLM's run-to-run variation comes from multi-chunk prefill. Every vLLM capture ran alone; rows inside the first 2,045-token prefill chunk are bit-identical between runs, rows beyond it almost never are.
  • The headline pairs CuteAFD against the mean of the vLLM runs, since one vLLM run is one draw of a varying engine. The 2-8K top-1 difference against the first run does not hold against the second.
  • Length and content are not separated. The 8-16K bucket is block B, long code reading; within block A, the only block present at every length, the KL difference is flat.
  • Q2: no NLL difference larger than about ±0.005 nat at 32K-128K on 6 documents. Per-type intervals rest on 3 documents each; engine-vs-engine disagreement is about 1.8x vLLM's own run-to-run disagreement on matched documents.
  • Trailing context rows are sensitive to any numerical change. They are 30 copies of one unrelated file, and vLLM differs from itself there by up to 1.96 nat; they are not used as a precision probe.
  • C0 control: C1 − C0 −0.0003 [−0.0019, +0.0014].
• ~01:15. Declared after the audit and before any of these requests: the drift subset, F3 (a second vLLM launch for precision) and vLLM draft-acceptance counters; GPU telemetry logged every 5 s.
• 01:42-03:21. Block L on CuteAFD, stages L1 to L3s, 100 requests, every gate passed; analysed together with vLLM's run.
• 03:22-03:35. Drift repeat on CuteAFD (same launch, same seeds and nonces as its Block T cells).
• 03:35-03:51. F3: a fresh vLLM capture launch, P-B and P-A captured once more (run V2).
• 04:03-04:17. Drift repeat on vLLM (fresh normal launch) with acceptance counters; F2 prefix-cache lengths.
• 04:32. Evaluation complete. Every slot of the run done; summary written (SUMMARY.md).
• 10:55-11:14. vLLM JIT caches persisted on this setup (TileLang, Triton, TorchInductor and CUDA, into the cache directory the container already keeps): startup 697 → 382 s; the first start after the change fills the caches (714 s) (results/2026-10-27-vllm-startup-cache).
• 13:03-13:34. Needle pilot on both engines with K3.25, prompts of 256K, 512K and 1M tokens, one request each (exploratory; results/2026-10-27-long-context-pilot).

2026-10-27 (evening), K4 checkpoint (results/2026-10-27-k4-vllm-cuteafd; no separate review of this run):

• 17:34-17:51. K4 startup and smoke checks on both engines (one cold and one warm start each, and CuteAFD's scoring launch).
• 17:58. K4 protocol frozen, before any comparison request (FROZEN.json sha256 368e1700…; METHOD.md).
• 18:03-19:48. K4 Block Q: vLLM launch 1, CuteAFD's scoring launch (every panel twice, byte-identical), vLLM launch 2.
• 19:54-23:25. K4 Block T: vLLM 19:54-20:47 and its drift subset 20:47-20:58; CuteAFD 22:12-22:57 and its drift subset 22:57-23:07; a final vLLM drift subset 23:13-23:25. 264 measured requests per engine, 0 errors.
• Before 23:31. Follow-up F1-K4 (W8A16 prefill) declared, after the Block Q results.
• 23:31-00:32. F1-K4: CuteAFD scoring launch with W8A16 prefill; speed on serving launches 23:50-00:00 (as shipped), 00:02-00:12 (W8A16), 00:14-00:24 (as shipped).

2026-10-28:

• From 00:38. K4 Block L on both engines, vLLM first; analysed afterwards with the frozen K3.25 analyser and the review's loop rule.
• To 12:47. Follow-up F1-K4b (CuteAFD's other FP8 switches) declared before any of its requests; planner dry runs (BF16 KDA projections do not fit a serving launch with K4); the amax and best scale rules captured.
• 13:05-13:10. F1-K4b amendments B1 and B2: drafter-off scoring launches for the BF16 head and KDA arms, gated by a control whose rows were byte-identical to as shipped on all 157,671 scored rows.
• 14:45-15:19. K4c (W8A16 + amax), declared before any of its requests: speed on serving launches 14:45-14:55 (as shipped), 14:57-15:07, 15:09-15:19 (as shipped).
• 17:13-17:47. K4d (W8A16 + best), declared before any of its requests: speed 17:13-17:23, 17:25-17:35, 17:37-17:47.

Status

• Block Q, precision (V1, C1, C0): done
• Block Q per-position scores (P-B, Q2) and tail and position statistics (P-A, P-B, Q2): done
• Block T, throughput: vLLM fork v0.9.2 (V1): done
• Block T, throughput: CuteAFD v3.0.1 (C1): done
• Block L, long output: vLLM fork v0.9.2 (V1): done
• Block L, long output: CuteAFD v3.0.1 (C1): done
• Follow-up F1, CuteAFD FP8 switches: precision at 8-16K and speed (outside the frozen protocol): done
• Follow-up F2, vLLM prefix-cache reuse on repeated prompts (outside the frozen protocol): done
• Follow-up F3, second vLLM launch for precision (outside the frozen protocol): done
• Drift check, repeat subset on both engines: done
• GPU telemetry during the later runs (summary): done
• K4: Startup and smoke checks (Stage 1, before the freeze): done
• K4: Block Q, precision (top-20 measures): P-A, P-B, Q2: done
• K4: Block Q per-position scores and tails (P-A, P-B, Q2): done
• K4: Block T, throughput, with drift repeats: done
• K4: Block L, long output: done
• K4: Follow-ups F1-K4 (W8A16 prefill), F1-K4b (FP8 switches), K4c (W8A16 + amax) and K4d (W8A16 + best), CuteAFD on K4 (outside the frozen protocol): done
• K4: Follow-up K4e: CuteAFD W8A16 + amax under the full Block T grid and Block L (outside the frozen protocol): done
• K4: Q2 document c2: rows where vLLM differs from itself: done

Other runs

Not part of the comparison above. Exploratory runs and checks are labelled as such in their folders and never share a table with the measured results.