CuteAFD v4.0.0 against tpurtell's vLLM fork
CuteAFD v4.0.0 gives the same precision as v3.0.1. Against tpurtell's vLLM fork it is faster or level in every speed cell, holds 1.2-1.4x the context, starts in under 100 seconds, and stays slightly further from BF16 on both checkpoints. No reply looped on either engine. Setup: two RTX PRO 6000 only, no DGX Spark expert nodes.
What was tested
| Engine | CuteAFD v4.0.0, coordinator image sha256:e257bc61… |
| Layout | GPU only: two RTX PRO 6000 Blackwell Max-Q (300 W), experts local on the same two GPUs. No DGX Spark expert nodes, no RDMA network. CuteAFD's disaggregated layouts (attention on RTX, experts on Sparks) were not run |
| CuteAFD v4.1.0 | Released after this run; read in source, not run. For this GPU-only layout on RTX PRO 6000 we found no change to the GLM-5.3-Flash compute path: its one kernel-path change moves an expert-input rotation only on GB10 (Spark) builds, and the author states both placements give identical results. The other changes are a renamed launcher setting and paths for Sparks and other models. No difference is expected for these results; none was measured |
Precision
CuteAFD sits slightly further from BF16 than the vLLM fork on both checkpoints, on both panels; every gap is resolved and every gap is a small fraction of the gap between checkpoints (Three checkpoints).
CuteAFD v4.0.0 minus the vLLM fork, same checkpoint, same rows; right of the line = CuteAFD further
| vLLM fork | CuteAFD | Difference | 95% interval | ||
|---|---|---|---|---|---|
| K3.25 | |||||
| KL, P-A full vocabulary | 0.0458 | 0.0476 | +0.0017 | [+0.0006, +0.0028] | resolved |
| KL, P-A top-20 | 0.0365 | 0.0380 | +0.0015 | [+0.0006, +0.0025] | resolved |
| KL, P-B generated, top-20 | 0.0444 | 0.0477 | +0.0036 | [+0.0020, +0.0051] | resolved |
| Top-1 agreement, P-A | 93.50% | 93.15% | −0.35 pts | [−0.54, −0.16] | resolved |
| K4 | |||||
| KL, P-A full vocabulary | 0.0246 | 0.0281 | +0.0035 | [+0.0016, +0.0059] | resolved |
| KL, P-A top-20 | 0.0192 | 0.0219 | +0.0027 | [+0.0012, +0.0047] | resolved |
| KL, P-B generated, top-20 | 0.0315 | 0.0358 | +0.0045 | [+0.0024, +0.0068] | resolved |
| Top-1 agreement, P-A | 95.28% | 94.73% | −0.55 pts | [−0.72, −0.37] | resolved |
vLLM fork values are receipts from other days on the same panel, plan and teacher (rows asserted identical). Top-20 measures (P-A KL20, P-B KL_pb20) are not comparable in level with full-vocabulary KL, or across panels.
Speed
Ratio CuteAFD / vLLM fork, same checkpoint; noise band ±8% counts as level; bar scale 0.8-1.8x, line at 1.0
| K3.25 | K4 | |
|---|---|---|
| Speed cells, aggregateof 22 | 19 faster · 3 level · 0 slower | 22 faster · 0 level · 0 slower |
| Speed cells, clean decodeof 22 | 15 faster · 7 level · 0 slower | 19 faster · 3 level · 0 slower |
| Decode, one stream | 1.06-1.46x | 1.07-1.42x |
| Aggregate, 32K-128K input | 1.40-1.63x | 1.38-1.66x |
| Time to first token at 128KCuteAFD vs vLLM fork | 19.8 s vs 34.9 s | 20.0 s vs 36.3 s |
Largest gains on reasoning prompts (~1.4x) and long inputs. 0 errors in every cell.
Memory and startup
Serving launches, two RTX PRO 6000 Max-Q
| K3.25 | K4 | |
|---|---|---|
| KV pool, vLLM forktokens | 4,762,898 | 1,377,179 |
| KV pool, CuteAFD v4.0.0tokens | 5,799,4241.22x | 1,931,2641.40x |
| Startup, vLLM forkcold / warm | 348 / 349 s | 292 / 313 s |
| Startup, CuteAFD v4.0.0cold / warm | 91 / 86 s | 86 / 81 s |
K4 on CuteAFD v3.0.1 held 447,488 tokens; v4.0.0 holds 4.3x that.
Long outputs
32 replies per cap (8 code, 8 JSON, 8 prose, 8 reasoning); one run each, r2 = repeat with new seeds; descriptive
| K4 vLLM (10-27) | K4 CuteAFD v4 r1 | K4 CuteAFD v4 r2 | K4-Core-K6 v4 r1 | K4-Core-K6 v4 r2 | |
|---|---|---|---|---|---|
| Finished within 8K | 8 | 8 | 8 | 8 | 8 |
| Finished within 32K | 12 | 17 | 18 | 11 | 13 |
| Finished within 130K | 29 | 31 | 31 | 31 | 30 |
| Code: all tests pass, 130Kof 8 | 8 | 8 | 8 | 7 | 8 |
| Code reply length, mediantokens, 130K cap | 51,343 | 39,377 | 35,407 | 46,417 | 41,913 |
| …of which thinking, median | 47.5K | 34.8K | 30.6K | 42.4K | 37.9K |
| JSON valid, 130Kof 8 | 5 | 5 | 5 | 7 | 6 |
| Replies that looped | 0 | 0 | 0 | 0 | 0 |
Small caps measure thinking length: at 8K, 139 of 144 capped replies were still thinking. Answers are about the same size on every arm; the extra length is thinking. Most JSON failures are one prompt asking for 1,030 records, which does not fit after long thinking. The vLLM fork column is the 2026-10-27 run (same requests); engines do not share a sampler, so columns are patterns across runs, not paired differences.
v4.0.0 against v3.0.1
Same day, same checkpoints
| Result | |
|---|---|
| Panel P-A, K3.25 and K4 | bit-identical |
| Panel P-B | within ±0.00005, not resolved |
| Long-output text | code, prose and reasoning replies identical (75/75); JSON replies differ (25/25) |
| Why the JSON replies differ | v4 cuts prompt processing into different pieces (one piece up to 4,096 rows instead of two lanes); a few kernels change arithmetic at fixed row counts. Confirmed: v3.0.1 run with one lane matches v4 bit for bit. Size about 1e-5 KL. |
Multi-turn tool calling
tool-eval-bench, 88 scenarios, both engines on the same prompts: 160-163 (vLLM fork) and 163 (CuteAFD) of 176 on K3.25, 158-163 and 158 on K4; the same two scenarios fail everywhere; CuteAFD's turns came in 2.7-3.1 s against 4.8-5.3 s at eight scenarios at once. Details, per-scenario splits and the benchmark's own rubric: Tool calling.
Not covered
- K4-Core-K6 on the vLLM fork: v0.9.2 does not run it. K4-Core-K6 results are on the checkpoints page.
- CuteAFD with DGX Spark expert nodes, or any layout other than two RTX PRO 6000 with local experts.
- CuteAFD v4.1.0 as a run (source read only; see What was tested).
- Full-vocabulary P-B: the fidelity set ships top-k rows only.
- Multi-turn tool calling is on its own page: Tool calling. No soak test (hours or days of continuous serving) was run.
Numbers: SUMMARY.json.