Research: tails, position and other deep dives

Tails and position (K3.25)

Most positions sit far below the mean: the median position's KL on P-A is 0.0032 (vLLM fork) and 0.0035 (CuteAFD) against means of 0.0458 and 0.0476; the highest-KL 5% of positions carry 57% and 57% of the total. p99 on P-B generated positions: vLLM fork 0.567-0.609 over its three runs, CuteAFD 0.620. On P-A, positions with a short history carry more KL on both engines (history 1-256 tokens: 0.114 and 0.115; 1,025-2,047 tokens: 0.032 and 0.035), so a panel's mean depends on which positions it scores.

Tails and position (K4, top-20 measures)

Computed by tools/kl_tails.py from the per-position files in q/rows/ (K3.25) and q/rows/ (K4); python3 tools/kl_tails.py --check reproduces them.

K3.25 by context bucket, with the follow-up F1 arms

Panel P-B, generated rows, against the mean of the three vLLM runs (two launches; pb-vs-vllm-mean3.json). The 8-16K bucket is block B, long code reading; whether length or content drives the difference is not separated here. F1c and F1d are follow-up F1 arms of CuteAFD, outside the frozen protocol. History: the tokens before each scored position. Tails (p99, p99.9) and KL by position: q/tails/.

K4 precision in full, and K4 against K3.25 on the same measure

Q2 without the window q2-code-c2-064k (reported separately by rule): CuteAFD − vLLM launch 1 +0.0012 [-0.0007, +0.0030], CuteAFD − vLLM launch 2 -0.0008 [-0.0086, +0.0071]. On that window vLLM differs from its own second launch on 192 rows by more than 1 nat (c2/). Determinism: CuteAFD's two passes were byte-identical on every window (row hashes); vLLM's two launches returned identical values on 20 of 64 P-B windows and on none of the 25 P-A windows, though their mean KL20 there differs by less than 1e-7.

Where the long-code difference may come from

• CuteAFD's distributions are flatter at 8-16K (K3.25): predictive entropy 0.580 against the vLLM fork's 0.523, higher in 18 of 18 windows (read from the full-vocabulary dumps by the audit). This is why its NLL difference is larger than its KL difference on this greedily generated text.
• Candidates, not inferred: the FP8 projections, head and KDA projections; the sparse-attention indexer's key format (CuteAFD BF16, the vLLM fork FP8); attention placement (CuteAFD splits each attention layer across both GPUs, the vLLM fork gives whole layers to one GPU); prefill geometry; the vLLM fork's BF16 logits. The settings follow-ups switch the ones CuteAFD exposes; on K4 what remains with every switch off sits in parts with no BF16 option.

Q2 document c2 on K4: a window where the vLLM fork differs from itself

CPU-only reading of the K4 Block Q captures (top-20 rows), arms V-r1 and V-r2 (two vLLM launches) and C-r1 (CuteAFD). Document c2 is a concatenation of vLLM's own Python source. Its three Q2 windows are prefixes of one token stream (c2-032k = tokens 0-32K, c2-064k = 0-64K, c2-128k = 0-128K), and each scores its last 4,096 rows.

KL20n per window (the frozen Q2 scorer's engine-against-engine form) on c2-064k: vLLM against its second launch 0.094, vLLM against CuteAFD 0.148 and 0.168 (one value per vLLM launch). On the other 17 windows the same pairs read 0.003-0.039 (vLLM against vLLM) and 0.005-0.066 (vLLM against CuteAFD).

What the rows show:

1. On c2-064k the two vLLM launches disagree with each other on 192 rows by more than 1 nat. vLLM's run-to-run variation alone produces nat-scale changes there; CuteAFD's count against either launch (292, 298) is of the same order. The K3.25 run had no vLLM repeat of c2, and its audit recorded that this window "cannot be attributed to either engine". On K4 the window is sensitive for vLLM against itself as well.
2. The sensitivity is specific to the document and the position. It is small at 32K, largest at 61-65K and smaller again at 127-131K of the same stream; the c0, c1 and prose windows show 0-9 such rows between the vLLM launches. The text is ordinary code throughout (no long lines, runs, binary or non-ASCII stretches, scanned in 4K-token blocks).
3. Many of the volatile rows are confident ones. Of the 192 vLLM-against-vLLM rows on c2-064k, 38 have one launch giving the actual token more than 0.9 probability; 20 are top-1/top-2 swaps between the launches. The window's NLL (1.01) is more than twice its top-20 entropy (0.46): the model is confident there and often does not predict the actual token.
4. Long-range exact copying accounts for part of it. Rows whose next token completes a 12-gram that also occurs more than 2,048 tokens earlier are about three times as volatile between the vLLM launches on c2-064k (14% against 4.4%); c1-128k has 287 such rows and none of them is volatile.

Open question (not tested): whether near-duplicate code earlier in the stream, and which earlier variant the sparse-attention indexer selects, accounts for the confident flips. Answering it needs per-layer indexer selections from two launches.

The protocol reports q2-code-c2-064k separately and also gives the Q2 numbers without it (../q/q2-top20.json, without_outlier). Files: c2-summary.json (the table and items 2-3, from c2_summary.py), c2-rows.json (per window and pair: mean ΔNLL, rows above 1 nat by quarter of the window, the largest ΔNLL rows with their positions), copy-rows.json (item 4).

Follow-up designs

• FOLLOWUPS.md: K3.25: F1 (FP8 switches), F2 (prefix cache), F3 (second vLLM launch), drift check
• FOLLOWUPS-K4.md: K4: W8A16 prefill
• FOLLOWUPS-K4b.md: K4: the other FP8 switches, K4c (W8A16 + amax), K4d (W8A16 + best)
• RESULTS-F1.md: K3.25 F1 results in full
• RESULTS-F1K4b.md: K4 settings results in full