Review of the K3.25 analysis (2026-10-27)

The review as published, with suggested follow-up work and local items omitted. Cited files link to this repository or to the engines' public sources at the versions tested; files that are not published are marked. Line numbers refer to the files as reviewed; published copies with placeholders may differ by a few lines.

Reviewer: a separate reviewer who did not run the measurements; read-only. CPU only: no GPU, no requests to :8001, no docker exec. Recomputations ran in docker run --rm --network none containers of local images (v0.9.2 image for numpy; CuteAFD image for tokenizers). The recomputation scripts are not published. Bootstraps I ran are window-cluster percentile bootstraps (5,000 draws, numpy RNG), so their limits differ from the scorer's SplitMix64 ones in the last digit.

1. Summary

The instruments are sound. The scorer port, the capture plumbing, the T driver and the summary script reproduce. The precision difference holds: C1 is further from the reference than vLLM on P-B generated rows, it survives the multiplicity and leave-one-out checks below, and C1's NLL is higher at every context bucket.

Four statements in the write-up go further than the data:

1. Why vLLM varies run to run (cause misattributed). It is not batching or speculation. The data show the variation starts at the second prefill chunk. At 8-16K, vLLM's run-to-run divergence is about two thirds of the cross-engine divergence. The headline magnitudes therefore depend on which vLLM run is used as the baseline.
2. "Located at 8-16K / grows with context" cannot be separated from content. The 8-16K bucket is block B (long code reading). Within block A, the only block present at every length, the KL gap is flat.
3. The FP8 mechanism story leaves out several real differences between the engines' arithmetic. F1a also changes the KDA split path as well as KDA precision.
4. Q2's sub-results rest on 3 clusters and one outlier window.

In Block T, the scheduling explanation holds and can be quantified. Streams that decode without prefill interleaving decode at the same speed on both engines. At 32K/128K the aggregate figure is mostly a prefill measurement.

In Block L, the "terminal loop" flags on code come from the detector counting the answer re-stating code drafted in the reasoning; they are not loops.

2. Findings, ranked

Corrections (a stated result, explanation or number did not match the data)

I1. vLLM's run-to-run variation was attributed to another cause, and its size was understated.

• Claim. RESULTS-Q.md:45-46 says vLLM repeats differ slightly ("P-B KL ±0.0006, Q2 floor KL 0.0047"), "as expected with continuous batching and speculation on".
• Evidence: no batching. Every vLLM capture ran alone. vllm-v092-dumps/runs/v1-r{1,2}/{pa,pb,q2}/capture-manifest.json (raw capture or local log, not published) has steps_with_other_requests = 0 for all 25 + 64 + 18 (r1) and 25 + 64 + 6 (r2) windows. Chunking is identical between the two runs (64/64 windows have the same chunk_lens), and cached_tokens is 0.
• Evidence: where the variation starts. pb3.py (the auditor's recomputation, not published) splits windows by number of prefill steps:

Inside 2-step windows, rows of the first 2,045-token chunk are 100% bit-equal and rows of the second chunk are 2-67% bit-equal (a04, d01, d04, c03). This matches an earlier investigation (local-inference-evals, LEDGER.md (not published)): identical solo prefills diverge from the layer-3 indexer top-k at token 2,051, the first position where sparse selection applies.

• Evidence: size, from the full-vocabulary dumps (ee2.py (the auditor's recomputation, not published), gen rows, 34 windows):

"±0.0006" is the shift of a mean, not the size of the run-to-run noise.

• Impact: the headline depends on which vLLM run is the baseline (pb1.out (the auditor's recomputation, not published)):

The 2-8K top-1 "CuteAFD better" interval (RESULTS-Q.md:30) excludes zero only because of which vLLM run was picked.

I2. The claim of identical inputs does not hold for two Block T details.

• Nonces differ between engines. block_t.py:20 says requests "pair across engines". The nonce salt is label:time_ns (block_t.py:213), so 0 of 1,008 paired requests share a nonce: every prompt differs in its first line between engines. Seeds still pair, but at temperature 1.0 with two different samplers they do not pair the outputs either.
• 128K code prompts differ by 24 tokens. All 24 C1 requests at 128K code report 131,096 / 131,095 prompt tokens against the expected 131,072 / 131,071; vLLM matches the expectation. Those prompts (vLLM source) contain literal <think> once and </think> 11 times (checked in stage0/prompts/T/code/128k/p*.json (raw capture or local log, not published)). vLLM's chat path tokenizes them as special tokens; CuteAFD tokenizes them as text. So A2's "thinking on gives both engines the same prompt" (protocol/PROTOCOL.md (cited as PROTOCOL-DRAFT.md:168-172)) does not hold for those cells: vLLM's model sees 11 end-of-thinking control tokens inside the user turn.
• Impact. Negligible for throughput. It is a stack difference: control-token strings in user text are parsed as control tokens on vLLM.

Narrowed (a statement went beyond the evidence)

U1. "The gap is located at 8-16K and grows with context" is confounded with content (block B).

• The bucket is block B. All 12 block-B windows are in 8-16K, and none are elsewhere. They supply 6,144 of the bucket's 7,522 gen rows; the other 6 windows are block A (pb1.out (the auditor's recomputation, not published)). The bucket effect is therefore inseparable from "long code reading" content.
• Block A shows no growth with context. A is the only block present in all three buckets (pb4.py (the auditor's recomputation, not published), C1 − mean(V)):

Difference-in-differences for KL (8-16K minus <8K): +0.0006 [−0.0030, +0.0044]. So "consistent with a gap that grows with context" (RESULTS-Q.md:51-53) is not supported for KL. A context trend is suggested only in block A's NLL and top-1, on 6 windows.

• "About 0 below 8K" holds for KL only. NLL differs at every bucket: 0-2K +0.0069 [+0.0031, +0.0110] and 2-8K +0.0073 [+0.0015, +0.0128] (pair JSON gen.by_bucket). Block E, at 0-2K/2-8K, has KL −0.0000 but NLL +0.011 [+0.006, +0.016]. A summary reading "~0 below 8K" needs "in KL".
• What does hold.
  • Leave-one-window-out at 8-16K: dKL stays between +0.0037 (drop b08) and +0.0051 (drop b04).
  • 14 of 18 windows are positive.
  • Bonferroni over about 15 bucket×metric looks (99.67% intervals): KL +0.0044 [+0.0003, +0.0082], top-1 −1.04 [−1.67, −0.29], NLL +0.0266 [+0.0186, +0.0325]. All survive, KL narrowly.
• Bucket choice. Buckets and blocks were pre-declared reporting dimensions (protocol/PROTOCOL.md (cited as PROTOCOL-DRAFT.md:53-54)). Headlining the bucket rather than the block was a post-hoc framing, and F1 inherits it (declared after Q results; labelled as a follow-up, which is correct).
• Clustering. The 12 B windows have distinct prompts (prompt_sha256) but similar start offsets (10,790-12,877). Whether they share a source repository or session is not recorded in windows.json; if they do, the effective cluster count is lower. Unverified.

U2. The mechanism inference ("FP8 projections, head and KDA") omits several known differences, and F1a is not a clean ablation.

• What is right. The protocol's arithmetic table (protocol/PROTOCOL.md (cited as PROTOCOL-DRAFT.md:33-40)) matches the resolved launch. In run-family.sh:2003-2006 and :2061-2064, auto gives row128 KDA and an FP8 head on every layout. The F1 dry-runs confirm it: f1a and f1b raise weight bytes and shrink the pool (local/logs/dry-run-f1a.log (raw capture or local log, not published) vs dry-run-score.log).
  • Note: docs/models/glm5_flash.md:161-166 says the two-GPU head split defaults to BF16 KDA and head. That doc and the code differ on this point.
• Omitted differences between the engines' arithmetic or geometry (found in source and logs, not measured):
  • DSA indexer. CuteAFD keeps BF16 key/gate rows (keys, forced on a head split; dry-run-score.log:1). vLLM uses FP8 index-K with the B12x kpool top-k (prior audit phase1-indexer-selection.md:31 (internal working file, not published); VLLM_USE_B12X_KPOOL_INDEXER=1). This difference runs in CuteAFD's favour for precision. The indexer engages only past about 2K tokens, which is exactly where any context-dependent gap would appear. Selection and tie handling differ too (same earlier investigation).
  • Attention placement. CuteAFD uses context: MLA split by token across both GPUs, with a cross-GPU partial-softmax merge (dry-run-score.log:61). vLLM uses split:25: whole MLA layers owned by one GPU, unsharded (start.sh:41-45).
  • KDA split path. On CuteAFD, --kda-output-shard --kda-prefill-expanded (run-family.sh:2012-2017).
  • Prefill geometry. vLLM uses 2,045/2,045/262 chunks aligned to 4,352 (b08.steps.jsonl (raw capture or local log, not published)), and the scored rows straddle chunk boundaries. CuteAFD prefills context in 4,096-row lanes, then scores the 512 rows through the scoring path.
  • Logit precision. vLLM's logits are BF16 before log-softmax (logits_dtype: torch.bfloat16 in every steps.jsonl). Its head output is rounded too; it is not exact BF16-head arithmetic.
  • Prefill-shaped scoring measures CuteAFD's W8A8 prefill GEMMs. Decode and verify rows in serving do not use W8A8 activations; prompt KV written during prefill does carry the effect.
• New evidence, a cheap mechanism clue. CuteAFD's distributions are systematically flatter (ee2.py (the auditor's recomputation, not published), teachH.py (the auditor's recomputation, not published), gen rows):

The teacher bound is top-1024 plus the tail as one bucket. Excess entropy over the teacher is +17% for C1 and +5% for V1. This explains why the P-B NLL gap (+0.0168) is about 6x the KL gap: the panel text was generated greedily (windows.json recipe.temperature = 0), so NLL rewards sharpness. It also explains why C1 has the lower NLL on surprising tokens (trailing rows, Q2 c2-064k). Systematic flattening points more toward a scale or bias effect (head, norms, softmax inputs) than toward zero-mean rounding noise. This is inferred, not tested.

• F1a is confounded. GLM5_FLASH_KDA_FP8=off also removes --kda-output-shard --kda-prefill-expanded (run-family.sh:2012-2017: added only when kda_fp8 != off). It reverts to the half-K BF16 partials path, whose documented cost is about KL +0.0008 and a top-1 loss (run-family.sh:1981-1986). The discovery notes this (discovery-cuteafd-fidelity.md:496 (internal working file, not published)), but FOLLOWUPS.md:13 describes F1a as a pure precision switch. A null or adverse F1a result would be uninterpretable.
• The F1 reading rule is ambiguous (FOLLOWUPS.md:22-23). "Moves … toward V1 by more than the paired interval": interval width, or lower bound? It is also anchored to the single run V1. See I1.

U3. Q2: "no NLL difference at 32K-128K" is acceptable only as a bounded statement; the prose/code sub-claims are not supported.

• Too few clusters. Q2 has 6 documents (3 prose, 3 code), each at 3 lengths (q2-C1-vs-V1.json: documents, cluster: doc_id). A percentile bootstrap over 3 clusters has 10 distinct resamples. The type-level intervals at RESULTS-Q.md:36-37 (prose +0.0019 [+0.0007, +0.0032]; code −0.0029 [−0.0078, −0.0002]) are artefacts of that.
• t-intervals on the 6 document means (C1 − V1). Recomputed from per_window:

Neither sub-claim excludes zero.

• One window dominates the code result. In q2-code-c2-064k, C1 − V1 NLL is −0.0334 and engine KL is 0.0946; the other 17 windows have KL 0.005-0.017 and NLL −0.0023 to +0.0078. It has 74 rows with KL > 1 (others: 0-3), including rows at 25.6 nats, with errors on both sides: row 3330 has C NLL 12.6 / V 0.0, row 754 has C 0.0 / V 8.4. There is no vLLM repeat for document c2, so it cannot be attributed to either engine.
• "3x vLLM's own run-to-run disagreement" (RESULTS-Q.md:38-39). The floor comes from 2 documents (pg28988, c0) and the cross-engine figure from 6. On the same 2 documents the cross-engine KL is 0.0086 against a floor of 0.0047, which is 1.8x. Without c2-064k the 18-window mean drops from 0.0141 to 0.0094.
• Power. Q2 can exclude an NLL penalty above about 0.004-0.005 nats on human text at 32-128K. That is informative next to P-B's +0.027, so a bounded sentence is justified. "Does not differ" without the bound is not.

U4. Trailing context rows: "a property of the panel's context rows" is right; the implied reference-side cause is not.

• All 30 trailing spans are the same filler. Every span begins File: rust/crates/cuteafd-api/src/openai/probe.rs (ctx4.py (the auditor's recomputation, not published)). It is an unrelated file inserted as the observation after the model's tool call, for example a call to read_file ledger/dates.py. The 12,032 rows are 30 near-copies of one text, so they are effectively one item. The tokenization is canonical: re-encoding gives identical ids in 30 of 30.
• The rows are hypersensitive. Cross-engine KL on rows 20 or more positions into the trailing span is 0.72 / 3.14 / 2.53 nats on a03 / d03 / a10 (ee.py (the auditor's recomputation, not published)), against 0.008-0.020 on the same windows' gen rows. vLLM against itself is 1.96 nats on a10, a multi-chunk window. Both engines are confident (entropy about 1 nat) but disagree.
• What follows. Any arithmetic perturbation produces nat-scale KL on these rows. Both engines being about 1.7 nats from the teacher, with NLL below the teacher's, fits "three perturbed copies of a fragile computation". It does not show a reference-side error, which A1 (protocol/PROTOCOL.md (cited as PROTOCOL-DRAFT.md:162-164)) set up as the question.

U5. Block L: the code "terminal loop" flags count restated code, not loops (6 of 8 L3 and 1 of 1 L3s).

• Evidence (l1.py (the auditor's recomputation, not published)). Re-tokenizing every L3/L3s code reply:
  • 100% of the repeated 50-grams in the final 4,096-token window have their earlier occurrence in the reasoning span.
  • Self-repetition within the content is 0.000-0.001.
  • 37-88% of content 50-grams are copied from the reasoning.
  • exact_period_m08 is None for every flagged reply: no periodic structure.
• What the detector measures here. The onset sits about 1K tokens before the end of the reasoning (for example 42,234 of 43,783). The answer re-states code drafted in the reasoning; this is not looping. 7 of 8 L3 replies pass all tests (217 tests).

Added checks (checks or confounds not covered at first)

M1. No drift control or in-run telemetry for Block T.

• Run order. The protocol's CVC drift control (protocol/PROTOCOL.md (cited as PROTOCOL-DRAFT.md:19)) has no declared subset. T ran V then C (V1 18:35-20:35, C1 22:39-00:29), with no repeat of either.
• Telemetry. There is only a pre-run snapshot: temperature, power limit, memory (t/gpu-before-*.txt (raw capture or local log, not published): 66/63 °C before V1, 63/61 °C before C1). There are no clocks, power draw or throttle reasons during the runs. On 300 W Max-Q cards, 4 hours apart, this is a real gap.
• A partial CuteAFD repeat exists. F1-base (f1/speed-F1-base.json, same seeds, scoring launch, 2 hours later) against C1-r1 gives per-cell ratios of 0.96-1.08 on decode and 0.96-1.05 on aggregate. So single-cell noise is about ±4-8%, and T-SUMMARY ratios between 0.95 and 1.08 cannot be read as differences. T-SUMMARY has no intervals, against principle 7 (protocol/PROTOCOL.md (cited as PROTOCOL-DRAFT.md:20)).
• Desktop on GPU 1. Equal in both runs: 2,045 MiB (V1) and 2,058 MiB (C1) not owned by the engine. Active rendering by a desktop app or the browser during a run cannot be excluded from the logs.

M2. The single-stream speed advantage is plausibly draft policy, and the "code" and "json" labels are misleading.

• Draft policy differs. vLLM uses a fixed DFLASH_TOKENS=3 (start.sh:61) with a BF16 drafter. CuteAFD uses SPECULATOR_DRAFTS=adaptive (run-family.sh:1700-1707) with an FP8 drafter and W8A8 draft linear (:2074-2076). Acceptance was not recorded, although vLLM's /metrics exposes the speculative-decoding counters.
• Speedup pattern. 1.16-1.41x on code, json and reasoning against 1.02-1.06 on prose: consistent with deeper drafting on predictable text.
• The labels. At 1,024 output tokens with thinking on, 0% of code and json requests on either engine left the reasoning (t1.out (the auditor's recomputation, not published): content 0% in every code and json cell). So "code" and "json" mean "reasoning about a code or json prompt".
• Comparable content otherwise. Characters per token agree within about 5%. Reasoning stop rates are 33-58% on both engines.

M3. "As shipped" has undocumented deltas; the full list of launch differences follows.

• Undocumented on the vLLM side.
  • GPU_MEMORY_UTILIZATION=0.95 (repro/k4/vllm/env.sh (cited as the K3.25 recipe, which is not published; the K4 copy carries the same settings)) instead of the launcher default 0.97 (start.sh:104, fp8 profile).
  • The launcher is the v0.9.0 repo's start.sh (git log: 9f0c617 release: v0.9.0) run with the v0.9.2 image. Whether v0.9.2's own launcher defaults differ, for example DFLASH_TOKENS, is unverified here.
• Differences beyond the documented deltas (from the configs and logs read):

M4. Results are not read against the panel's own paired bar.

• The P-B manifest sets paired_bar: kl_margin 0.005, top1_margin 0.005.
• tpurtell's one-sided compare (bar.py (the auditor's recomputation, not published), the port of fidelity.rs:269-333):

• So C1 is not shown within his bar on top-1 against vLLM, while KL is within it, narrowly. RESULTS-Q.md:20-21 reports only the absolute bar.

M5. The panel's text provenance is not stated.

• Provenance. The P-B generated rows are greedy (temperature 0) outputs of CuteAFD v1.0.0 serving official FP8 with a BF16 head, kda-fp8 off, fp8-prefill none (windows.json: generation_server, recipe).
• Bias. Any engine-family bias would favour CuteAFD, so it runs against the finding (good to state).
• What NLL measures. On greedy text, NLL measures sharpness (see U2).

M6. Only prefill-shaped scoring was done.

• His CLI has --score-path decode (discovery-cuteafd-fidelity.md:23,454 (internal working file, not published)). It was not run.

3. What holds up (with how I checked)

Scorer and capture

• Our port of tpurtell's P-B rule matches his CLI (0 differences on 32,768 positions; claimed, not re-run). P-A reproduces the v0.8.0 result 0.0493912 (stage0/scorers/selfcheck/pa.log). Unit tests 27/27 (tests/results.json).
• vLLM's dumps match its API log-probabilities: max |Δ| 5.7e-6 across all captures (capture-manifest.json (raw capture or local log, not published), chosen_check_server).
• The vLLM capture launch equals the served launch apart from the plugin mounts and env (stage0/capture-vllm/launch-compare.txt).
• CuteAFD's serving and scoring dry-run plans are identical (diff dry-run-1.log dry-run-score.log).

Numbers in RESULTS-Q.md

• Every headline, bucket, block and C0 number I traced matches the logs and pair JSONs:
  • P-A: 0.0458 / 0.0476 / +0.0017.
  • P-B: 0.0457 / 0.0484 / +0.0027, NLL +0.0168, top-1 −0.51; bucket table; block B +0.0047 / +0.029 / −1.1.
  • Bar values 0.0484 / 93.33 / 96.31 / 99.48, and his FP8 baseline 0.02164 / 95.77.
  • Q2 overall and per-type values, after negation.

The precision gap exists

• Survives Bonferroni and leave-one-window-out (U1).
• C1 NLL is higher than V1 in 12 of 12 block-B windows.
• C0 tracks C1, so load-quantized vs official FP8 projections do not matter: C1 − C0 −0.0003 [−0.0019, +0.0014].

Seeds and determinism

• T: 1,008 distinct seeds of 1,008 per engine, the same seed list on both engines.
• L: 32 of 32 per stage, recorded per request.
• Q (vLLM): 64 distinct seeds of 64, irrelevant for teacher-forced rows.
• CuteAFD repeats are byte-identical.

Freeze

• Protocol, drivers, scorers and manifest hashes match FROZEN.json today (sha256 prefixes re-computed). FROZEN.json itself is c4815666….
• vLLM captures started at 18:00, after the 17:32 freeze. The A1 changes predate any vLLM result.

The thinking-off confound is real

• The template has no enable_thinking. It always ends <|assistant|><think> (chat_template.jinja:259-260).
• vLLM renders 147 prompt tokens for both flag values (q/thinking-check-vllm.json).
• The check is one request with unrecorded max_tokens and does not close </think>. The template and token count are the decisive evidence.

Block T arithmetic and the scheduling explanation

• t_summary.py regenerates T-SUMMARY.md byte-identically.
• aggregate_tok_s = sum of completion tokens / (last token − first send), applied identically to both engines.
• The scheduling explanation is supported (t2.py (the auditor's recomputation, not published)). vLLM decode windows overlap other streams' prefill by a median 40-81%, CuteAFD's by 0-51%. Streams whose window overlaps less than 5% decode at the same speed on both engines:

So the 1.5-4x is prefill-first vs interleaved scheduling. It trades against vLLM's earlier first tokens, which is why vLLM's TTFT median is lower at c ≥ 4.

• At 32K/128K, 71-92% of batch wall time precedes the last first token. The +6-11% aggregate there is mostly CuteAFD's +8-14% batch prefill rate.
• vLLM prefix cache "0 hits" is confirmed: primer and 3/3 repeats show cached_tokens 0 and TTFT 2.10 s, equal to uncached 8K. CuteAFD shows 8,192 cached and 0.02 s.
  • A candidate mechanism for F2 (inferred): GLM53_DFLASH_BOUNDARY_LOOKUP=0 "drops one block per group" (start.sh:64-67) under --mamba-cache-mode align (start.sh:465). An 8,192-token prompt has one full 4,352 block. F2's lengths of 8,704 and 17,408 test exactly this.

Block L harness

• The code sandbox is properly confined: --network none --cpus 2 --memory 2g --pids-limit 256, non-root, read-only source, 300 s kill (checkers/code_check.py:14-17,73-85).
• The JSON checker's rule is fixed and strict.
• Block L V1 ran with 0 errors at every stage.

GPU time used by this audit: none.