Settings: what each CuteAFD setting buys
CuteAFD has switches that trade a little speed, or GPU memory, for precision. On K3.25, W8A16 prefill closes about 59% of the long-code difference from the vLLM fork for about 4% longer time to first token; with all three switches no difference is resolved, at about 4% slower decode. On K4 on two RTX PRO 6000s, W8A16 prefill with the amax scale rule closes about two thirds for about 4% longer first token, with decode level; the BF16 options do not fit there. Every setting here was measured in a follow-up outside the frozen protocols.
Every setting tested
Long code: P-B 8-16K, CuteAFD − vLLM fork (K3.25 coarse KL; K4 KL_pb20, top-20), 95% interval. Change against as shipped: the same positions, CuteAFD against itself (bit-deterministic, so no band). First token and decode: against an as-shipped launch measured beside it. Settings marked "no" do not fit a serving launch on two RTX PRO 6000s with K4; their precision comes from scoring launches.
| Setting | Config lines (added to the serving config) | Serves on 2x RTX? | Long code (8-16K): CuteAFD − vLLM fork | Change against as shipped (paired) | Teacher panel (P-A): CuteAFD − vLLM fork | First token against as shipped | Decode against as shipped | Source |
|---|---|---|---|---|---|---|---|---|
| K3.25: as shipped | yes | +0.0054 [+0.0034, +0.0074] | not scored | Block Q | ||||
| BF16 KDA projections | GLM5_FLASH_KDA_FP8=off | yes | +0.0028 [+0.0007, +0.0052] against the mean of the first two vLLM runs only (as shipped reads +0.0050 there) | −0.0022 [−0.0047, +0.0005] | not scored | −0.9% (−2.0% to +0.3%) | 0.92x median | F1 |
| BF16 output head | GLM5_FLASH_FP8_HEAD=off | yes | +0.0048 [+0.0028, +0.0069] against the mean of the first two vLLM runs only (as shipped reads +0.0050 there) | −0.0003 [−0.0004, −0.0001] | not scored | +0.4% (+0.1% to +0.7%) | 0.94x median | F1 |
| W8A16 prefill | GLM5_FLASH_FP8_PREFILL=off | yes | +0.0022 [+0.0005, +0.0041] | −0.0032 [−0.0049, −0.0013] | not scored | +4.1% (+3.8% to +4.5%) | 0.99x median | F1 |
| W8A16 prefill + BF16 KDA projections + BF16 output head | GLM5_FLASH_KDA_FP8=offGLM5_FLASH_FP8_HEAD=offGLM5_FLASH_FP8_PREFILL=off | yes | +0.0016 [−0.0005, +0.0033] | −0.0038 [−0.0063, −0.0013] | not scored | +2.8% (+1.5% to +3.6%) | 0.96x median, worst cell 0.86x | F1 |
| K4: as shipped | MAX_CONTEXT_TOKENS=262144 | yes | +0.0090 [+0.0062, +0.0122] | +0.0027 [+0.0012, +0.0047] | Block Q | |||
| W8A16 prefill | GLM5_FLASH_FP8_PREFILL=off | yes | +0.0072 [+0.0048, +0.0096] | −0.0019 [−0.0039, −0.0002] | +0.0009 [−0.0014, +0.0028] | +3.5% (+3.4% to +3.6%) | level (0.96-1.08x per cell) | F1-K4 |
| amax scale rule | FP8_SCALES=amax | yes | +0.0068 [+0.0046, +0.0090] | −0.0023 [−0.0054, +0.0008] | +0.0025 [+0.0003, +0.0050] | not measured (stored scale values only) | F1-K4b | |
| best scale rule | FP8_SCALES=best | yes | +0.0080 [+0.0066, +0.0096] | −0.0010 [−0.0039, +0.0015] | +0.0009 [−0.0012, +0.0025] | not measured (stored scale values only) | F1-K4b | |
| W8A16 prefill + amax scale rule | GLM5_FLASH_FP8_PREFILL=offFP8_SCALES=amax | yes | +0.0030 [+0.0011, +0.0049] | −0.0061 [−0.0095, −0.0029] | +0.0006 [−0.0016, +0.0027] | +4.2% (+4.0% to +4.2%) | level (0.93-1.09x per cell) | K4c |
| W8A16 prefill + best scale rule | GLM5_FLASH_FP8_PREFILL=offFP8_SCALES=best | yes | +0.0047 [+0.0030, +0.0065] | −0.0043 [−0.0068, −0.0023] | +0.0009 [−0.0005, +0.0023] | +4.1% (+4.1% to +4.6%) | level (0.96-1.04x per cell) | K4d |
| BF16 output head | GLM5_FLASH_FP8_HEAD=off | no | +0.0087 [+0.0059, +0.0119] | −0.0003 [−0.0004, −0.0002] | +0.0025 [+0.0010, +0.0045] | not measured (does not fit) | F1-K4b | |
| BF16 KDA projections | GLM5_FLASH_KDA_FP8=off | no | +0.0055 [+0.0026, +0.0084] | −0.0035 [−0.0065, −0.0007] | +0.0028 [+0.0015, +0.0045] | not measured (does not fit) | F1-K4b | |
| Every switch (best scale rule, BF16 KDA and head, W8A16 prefill) | FP8_SCALES=bestGLM5_FLASH_KDA_FP8=offGLM5_FLASH_FP8_HEAD=offGLM5_FLASH_FP8_PREFILL=off | no | +0.0041 [+0.0027, +0.0054] | −0.0050 [−0.0078, −0.0021] | +0.0012 [−0.0008, +0.0030] | not measured (does not fit) | F1-K4b | |
| Control: drafter off (validation, not a setting) | SPECULATOR=off | scoring only | +0.0090 [+0.0061, +0.0121] | 0 (rows byte-identical) | +0.0027 [+0.0012, +0.0048] | F1-K4b |
The share of the difference a setting closes is (as shipped − setting) ÷ as shipped against the vLLM fork: K3.25 W8A16 59%; K4 W8A16 21%, W8A16 + amax 67% (about two thirds), W8A16 + best 48%. Sources: RESULTS-F1.md, RESULTS-F1K4.md, RESULTS-F1K4b.md (K4b, K4c, K4d) and their summary JSON.
Settings map
What this shows. Each triangle is one CuteAFD setting: across, its first-token cost against as shipped; up and down, how far it still is from the vLLM fork on long code reading, here against the mean of its first two runs (follow-up F1's reference).
How to use it. Points low and to the left close the most for the least first-token time; decode cost is in each label, because on K3.25 it is not level for every switch.
Table view: the exact numbers
| Setting | First token against as shipped | Decode against as shipped | Remaining difference from the vLLM fork, 8-16K |
|---|---|---|---|
| as shipped | 0 (reference) | 1.00x (reference) | +0.0050 [+0.0031, +0.0071] |
| BF16 KDA | −0.9% | 0.92x | +0.0028 [+0.0007, +0.0052] |
| BF16 head | +0.4% | 0.94x | +0.0048 [+0.0028, +0.0069] |
| W8A16 prefill | +4.1% | 0.99x | +0.0019 [+0.00004, +0.0039] |
| all three | +2.8% | 0.96x | +0.0013 [−0.0008, +0.0031] |
Here the vLLM reference is the mean of its first two runs, as follow-up F1 publishes it (f1/F1-RESULTS.json); the settings table above gives the three-run figures. Speed: each arm ÷ the F1 base launch over the 8 cells of the speed subset; no interval is published for the speed ratios.
What this shows. The same map for K4: filled triangles serve on two RTX PRO 6000s, hollow ones need more GPU memory; scale rules and settings that do not fit have no measured speed and sit in the right-hand column.
How to use it. On two RTX GPUs, read the filled triangles: the lowest is the closest-to-vLLM setting that serves, and its position across is its first-token cost.
Table view: the exact numbers
| Setting | First token against as shipped (range over cells) | Decode against as shipped | Remaining difference from the vLLM fork, 8-16K | Serves on 2x RTX |
|---|---|---|---|---|
| as shipped | 0 (reference) | 1.00x (reference) | +0.0090 [+0.0062, +0.0122] | yes |
| W8A16 | +3.5% (+3.4% to +3.6%) | 0.96-1.08x per cell | +0.0072 [+0.0048, +0.0096] | yes |
| W8A16 + amax rule | +4.2% (+4.0% to +4.2%) | 0.93-1.09x per cell | +0.0030 [+0.0011, +0.0049] | yes |
| W8A16 + best rule | +4.1% (+4.1% to +4.6%) | 0.96-1.04x per cell | +0.0047 [+0.0030, +0.0065] | yes |
| amax rule | not measured | not measured | +0.0068 [+0.0046, +0.0090] | yes |
| best rule | not measured | not measured | +0.0080 [+0.0066, +0.0096] | yes |
| BF16 head | not measured | not measured | +0.0087 [+0.0059, +0.0119] | no |
| BF16 KDA | not measured | not measured | +0.0055 [+0.0026, +0.0084] | no |
| every switch | not measured | not measured | +0.0041 [+0.0027, +0.0054] | no |
Speed: per content cell (1K and 32K prompts, 1 and 4 streams, code and prose; 8 requests each), the median per request divided by the mean of the as-shipped launches run before and after the setting. Decode stayed inside the base-to-base band for every measured setting. No interval is published for the speed ratios.
W8A16 prefill + amax scale rule as a deployment (follow-up K4e)
CuteAFD with W8A16 prefill and the amax scale rule on K4, through the full speed grid and the four long-output stages (declared before any request: FOLLOWUPS-K4e.md). The as-shipped drift subsets before and after stayed within the ±8% band of the main run's drift cells, so the columns compare directly. Per cell: the median over every measured request. Descriptive: one run; read speed against the K4 noise band.
| Cell (prompt × streams, content) | Decode tok/s: vLLM fork | CuteAFD as shipped | W8A16 + amax | First token s: vLLM fork | CuteAFD as shipped | W8A16 + amax |
|---|---|---|---|---|---|---|
| 1K × 1, code | 145.7 | 174.8 | 179.2 | 0.27 | 0.42 | 0.44 |
| 1K × 1, json | 130.0 | 145.3 | 149.7 | 0.27 | 0.42 | 0.43 |
| 1K × 1, prose | 113.2 | 126.6 | 125.1 | 0.26 | 0.42 | 0.43 |
| 1K × 1, reasoning | 164.1 | 219.3 | 225.1 | 0.27 | 0.42 | 0.44 |
| 1K × 4, code | 78.4 | 89.9 | 94.0 | 0.83 | 1.26 | 1.31 |
| 1K × 4, json | 72.6 | 83.9 | 83.6 | 0.90 | 1.25 | 1.29 |
| 1K × 4, prose | 59.5 | 70.6 | 67.3 | 0.83 | 1.26 | 1.30 |
| 1K × 4, reasoning | 92.7 | 113.7 | 112.6 | 1.01 | 1.26 | 1.30 |
| 8K × 1, code | 146.7 | 178.8 | 168.6 | 2.20 | 1.83 | 1.90 |
| 8K × 1, json | 130.1 | 153.0 | 147.9 | 2.20 | 1.83 | 1.91 |
| 8K × 1, prose | 110.5 | 115.9 | 118.9 | 2.20 | 1.84 | 1.91 |
| 8K × 1, reasoning | 159.6 | 220.4 | 212.9 | 2.20 | 1.84 | 1.91 |
| 8K × 4, code | 68.4 | 75.6 | 74.8 | 5.71 | 5.30 | 5.52 |
| 8K × 4, json | 60.7 | 69.0 | 68.8 | 5.72 | 5.30 | 5.52 |
| 8K × 4, prose | 54.7 | 62.7 | 60.6 | 5.73 | 5.30 | 5.53 |
| 8K × 4, reasoning | 72.1 | 83.1 | 97.3 | 5.73 | 5.32 | 5.54 |
| 32K × 1, code | 142.4 | 169.7 | 174.6 | 9.05 | 7.37 | 7.67 |
| 32K × 1, prose | 108.5 | 121.4 | 123.1 | 9.04 | 7.38 | 7.69 |
| 32K × 4, code | 47.3 | 75.0 | 75.2 | 24.00 | 27.55 | 28.69 |
| 32K × 4, prose | 41.0 | 63.1 | 59.0 | 24.05 | 27.57 | 28.71 |
| 128K × 1, code | 144.6 | 171.8 | 176.0 | 36.30 | 30.08 | 31.19 |
| 128K × 1, prose | 113.2 | 119.9 | 117.2 | 36.42 | 30.09 | 31.20 |
| Stage | Content | Requests | vLLM fork: finished / at cap, median tokens, check | CuteAFD as shipped: the same | W8A16 + amax: the same | W8A16 + amax: median decode tok/s |
|---|---|---|---|---|---|---|
| L1 | prose | 8 | 0 / 8, 8,192, words/required 0.00 | 0 / 8, 8,192, words/required 0.00 | 0 / 8, 8,192, words/required 0.00 | 80.6 |
| L1 | code | 8 | 0 / 8, 8,192, all tests pass 0/8 | 0 / 8, 8,192, all tests pass 0/8 | 0 / 8, 8,192, all tests pass 0/8 | 113.0 |
| L1 | reasoning | 8 | 8 / 0, 536, correct 8/8 | 8 / 0, 631, correct 8/8 | 8 / 0, 611, correct 8/8 | 185.3 |
| L1 | json | 8 | 0 / 8, 8,192, valid 0/8 | 0 / 8, 8,192, valid 0/8 | 0 / 8, 8,192, valid 0/8 | 119.7 |
| L2 | prose | 8 | 2 / 6, 32,768, words/required 0.97 | 2 / 6, 32,768, words/required 0.86 | 4 / 4, 32,350, words/required 0.99 | 84.4 |
| L2 | code | 8 | 0 / 8, 32,768, all tests pass 0/8 | 5 / 3, 31,018, all tests pass 5/8 | 2 / 6, 32,768, all tests pass 2/8 | 120.7 |
| L2 | reasoning | 8 | 8 / 0, 599, correct 8/8 | 8 / 0, 624, correct 8/8 | 8 / 0, 642, correct 8/8 | 181.6 |
| L2 | json | 8 | 2 / 6, 32,768, valid 2/8 | 2 / 6, 32,768, valid 2/8 | 3 / 5, 32,768, valid 3/8 | 144.1 |
| L3 | prose | 8 | 7 / 1, 32,764, words/required 0.96 | 8 / 0, 34,842, words/required 0.96 | 8 / 0, 35,193, words/required 1.01 | 79.8 |
| L3 | code | 8 | 8 / 0, 51,343, all tests pass 8/8 | 8 / 0, 39,378, all tests pass 8/8 | 8 / 0, 37,537, all tests pass 8/8 | 123.4 |
| L3 | reasoning | 8 | 8 / 0, 706, correct 8/8 | 8 / 0, 558, correct 8/8 | 8 / 0, 544, correct 8/8 | 180.0 |
| L3 | json | 8 | 6 / 2, 66,812, valid 5/8 | 7 / 1, 55,715, valid 3/8 | 7 / 1, 67,902, valid 6/8 | 162.0 |
| L3s | prose | 1 | 1 / 0, 29,505, words/required 0.81 | 1 / 0, 33,344, words/required 0.98 | 1 / 0, 26,290, words/required 0.78 | 125.2 |
| L3s | code | 1 | 1 / 0, 55,168, all tests pass 0/1 | 1 / 0, 64,335, all tests pass 1/1 | 1 / 0, 57,465, all tests pass 1/1 | 162.0 |
| L3s | reasoning | 1 | 1 / 0, 639, correct 1/1 | 1 / 0, 638, correct 1/1 | 1 / 0, 672, correct 1/1 | 258.2 |
| L3s | json | 1 | 1 / 0, 15,788, valid 1/1 | 1 / 0, 8,596, valid 1/1 | 1 / 0, 13,638, valid 1/1 | 199.4 |