Precision: how close to the BF16 model?

"Reading" follows each run's declared rule: K3.25, the 95% interval excludes 0; K4, it also has to exceed the vLLM fork's launch-to-launch difference on the same rows. Band: the largest difference between any two vLLM runs' means on the same rows. CuteAFD repeats are byte-identical, so CuteAFD needs no band.

Teacher panel (P-A): every position of 25 texts of 2,048 tokens

brandonmusic's BF16 teacher logits, the full vocabulary. The K3.25 KL20 row is K3.25's own captures scored with K4's top-20 measure, so the two checkpoints can be compared on the same quantity. The panel is drawn from the corpus whose calibration windows K4's producer published (document-disjoint from them), so it can favour K4 against K3.25; read checkpoint differences beside P-B (Three checkpoints).

Fidelity set (P-B): the model's own replies, prompts up to 14.4K tokens

tpurtell's cuteafd-fidelity set, generated positions only (his rule). K3.25 is scored with his coarse KL (top 1,024 tokens plus the rest), K4 with its top-20 truncation, which keeps 98-99% of the coarse value on these rows.

The K3.25 8-16K figure (+0.0054) is against the mean of all three vLLM runs. Against other references it reads +0.0044 (the first vLLM run alone), +0.0050 (the first two runs, the reference follow-up F1 used) and +0.0052 (K4's top-20 measure); the audit also reports +0.0057 against the second run alone.

Long human-written text (Q2, 32K-128K tokens): no BF16 reference

Six documents (three books, three source files) at 32K, 64K and 128K tokens; the last 4,096 positions of each. With no BF16 reference this compares how surprised each engine is by the text (NLL), not closeness to BF16. K3.25 had a vLLM repeat on 2 of the 6 documents; K4 ran two full vLLM launches on all six, so its noise floor covers every document.

One code window, q2-code-c2-064k, dominates the code result on both checkpoints; on K4 the vLLM fork differs from its own second launch there on 192 rows by more than 1 nat (Research). Without that window K4 reads +0.0012 and −0.0008 against the two vLLM launches.

Notes

• Why the vLLM fork is averaged over runs. Its prefill is not bit-reproducible beyond the first 2,045-token chunk, so one vLLM run is one draw; CuteAFD's repeats are byte-identical. CuteAFD is paired against the mean of three vLLM runs from two launches (K3.25) or two launches (K4).
• Length or content. The 8-16K bucket is block B of the set, long code reading; within block A, the only block at every length, the K3.25 KL difference is flat. Which of the two drives the difference is not separated.
• Projection source inside CuteAFD (K3.25). CuteAFD on Core-FP8 with the official FP8 projections (C1) against CuteAFD on K3.25-v1 with FP8 projections made at load (C0): C1 − C0 −0.0003 [−0.0019, +0.0014], not resolved.
• Where the panel text came from. P-B's replies were written greedily by an earlier CuteAFD release serving the official FP8 model; any bias from that favours CuteAFD, against the observed difference.
• Top-20 measures. KL20 reads about 0.8 of full-vocabulary KL on P-A; KL_pb20 keeps 98-99% of coarse KL on P-B generated positions; every K3.25 difference kept its sign and significance under the truncation.
• Only the prefill path was scored. Tokens generated during decode use another path; its precision was not measured.

Deeper: tails, position and the c2 document on Research; the full write-ups: RESULTS-Q.md (K3.25), the K4 run README, RESULTS-F1.md.