CuteAFD and tpurtell's vLLM fork on two RTX PRO 6000s

Two inference engines by the same author, tpurtell (wrldsuksgo2mars on Hugging Face), on the same model and machine:

CuteAFD (tpurtell/cuteafd), his own runtime, built for attention/FFN disaggregation (AFD): attention and the expert (feed-forward) layers run as separate workers, so experts can sit on other machines; here, on two GPUs, they stay local.
His vLLM fork (tpurtell/glm-5.3-flash-ext3-2x-rtx), his vLLM-based engine for this model on two RTX GPUs.

The model is GLM-5.3-Flash on two RTX PRO 6000 Blackwell Max-Q GPUs at 300 W, in three quantized checkpoints: K3.25 (tpurtell's mixed 3.25-bit EXL3 experts), K4 (brandonmusic's 4-bit EXL3 experts, Local Inference Lab) and K4-Core-K6 (tpurtell's 4-bit experts with a 6-bit core; CuteAFD only). The questions: how close each engine stays to the original BF16 model, how fast it is, and how to set it up.

Results as of 2026-10-31; versions tested: CuteAFD v4.0.0 and v3.0.1, vLLM fork v0.9.2. Measured by Michael M (UrbanAstroLA).

Findings

Latest: CuteAFD v4.0.0 (2026-10-30/31). Two RTX PRO 6000 only, experts local, no DGX Spark expert nodes. CuteAFD v4.1.0, released after the run, was read in source, not run: no change to this GPU-only setup's compute path was found.

v4.0.0 against the vLLM fork: faster or level in every speed cell measured (1 and 4 streams), first token level or sooner in every cell, KV pool 1.2x (K3.25) and 1.4x (K4); slightly further from BF16 (full-vocabulary teacher panel +0.0017 and +0.0035, both resolved). Same precision as v3.0.1 (teacher panel bit-identical). No reply looped on either engine. CuteAFD v4
Three checkpoints: BF16's top token is missed 68 (K3.25), 53 (K4) and 53 (K4-Core-K6) times per 1,000 tokens; where BF16 is at least 90% sure, 0.61, 0.10 and 0.18 (teacher panel). K4 and K4-Core-K6 differ by calibration text (legal for K4; maths and function calling for K4-Core-K6) and neither is closer on both panels. Three checkpoints
Multi-turn tool calling: tool-eval-bench, 88 scenarios: 158-163 of 176 on every arm, scores overlap between engines, CuteAFD turns about 1.7x sooner at 8 scenarios at once. A functional check, not a ranking. Tool calling
Thinking length and the 6-bit core (investigation, 2026-11-02/04): K4-Core-K6 thought longer than K4 on CuteAFD in one run (code 1.43x, read after the fact). Rerun with new seeds under a frozen protocol: 1.15x [0.97, 1.32], code 1.27x, inconclusive. A derived checkpoint with K4-Core-K6's experts and its BF16 core separates the parts: experts 0.99x [0.93, 1.05], core 1.16x [0.98, 1.36]. On fixed reasoning traces neither measurably changes when the model stops, on CuteAFD (bit-identical across launches) or on the vLLM fork with a local loader modification (not a tpurtell release). Whether the core changes the loop rate is open. Thinking length vLLM part

CuteAFD v3.0.1 (2026-10-26/28): the findings below; where v4.0.0 changes a result, the line says so.

Precision: the quantization matters more than the engine. On K3.25, KL from the BF16 model (teacher panel) is 0.0458 on the vLLM fork and 0.0476 on CuteAFD. The largest engine difference is on long code (8-16K-token prompts): CuteAFD +0.0054 further on K3.25, +0.0090 on K4 (top-20 measure). Precision
Speed: CuteAFD generates faster; under concurrency the vLLM fork's first tokens come sooner. One stream: 114-168 tokens/s on the vLLM fork, 120-235 on CuteAFD (K3.25). First token, one request: 1K prompt 0.25 s and 0.41 s, 128K prompt 35.1 s and 29.6 s. With 4-16 requests at once the vLLM fork's first tokens come sooner in most cells, while tokens per second are level or ahead on CuteAFD except at 1K × 8 (v4.0.0, measured at 1 and 4 streams: CuteAFD's first token is level or sooner in every cell). Speed
A CuteAFD setting trades about 4% first-token time for precision. W8A16 prefill closes about 59% of the long-code difference on K3.25 (8K prose prompt 1.808 → 1.878 s); W8A16 prefill with the amax scale rule closes about two thirds on K4 (32K code prompt 7.38 → 7.69 s), decode level. Settings
Long tasks: the same at the longest cap; CuteAFD finished more code at 32K. At the 128K cap code passed every test in 7/8 (K3.25) and 8/8 (K4) on both engines, math 8/8. At the 32K cap CuteAFD finished 4/8 (K3.25) and 5/8 (K4) code tasks, the vLLM fork 0/8 and 0/8, consistent with CuteAFD's shorter code replies (0.74x and 0.77x the vLLM fork's median length at 128K); descriptive, n = 8. Needles in prompts of 256K to 1M tokens (K3.25): all found on both (exploratory); at 1M the first token took 270 s on the vLLM fork and 254 s on CuteAFD. Long tasks
Practical. Ready to serve in 382 s on the vLLM fork (JIT caches kept) and 89 s on CuteAFD. A repeated 8K prompt: 2.10 s on the vLLM fork, 0.017 s from CuteAFD's prefix cache. Only CuteAFD honours "thinking off". K4 KV cache: 1,377,179 tokens on the vLLM fork, 447,488 on CuteAFD v3.0.1 (1,931,264 on v4.0.0). Running it

Does the precision difference matter?

Distance from the BF16 model (lower is closer to BF16)

What this shows. Each panel is one measure on one set of texts, with its own axis; compare bars within a panel.

vLLM forkCuteAFDCuteAFD with a settingofficial FP8 model

How to use it. Grey is tpurtell's official FP8 model: the 3.25-bit quantization adds most of the distance; the blue-orange gap is the engine difference.

Table view: the exact numbers

Panel and measure	Row	KL from BF16
K3.25 · BF16 teacher panel (P-A) · full-vocabulary KL	vLLM fork	0.0458
	CuteAFD	0.0476
K3.25 · tpurtell's fidelity set (P-B), all generated positions · coarse KL	Official FP8 model (tpurtell's baseline)	0.0216
	vLLM fork	0.0451
	CuteAFD	0.0484
	CuteAFD, W8A16 prefill	0.0468
K3.25 · the same set, 8-16K long code · coarse KL	vLLM fork	0.0504
	CuteAFD	0.0557
	CuteAFD, W8A16 prefill	0.0525
K3.25 and K4 · BF16 teacher panel (P-A; can favour K4) · top-20 KL (KL20)	K3.25, vLLM fork	0.0365
	K3.25, CuteAFD	0.0380
	K4, vLLM fork	0.0192
	K4, CuteAFD	0.0219

The official FP8 baseline is tpurtell's published value for his set, quoted in q/RESULTS-Q.md. Top-20 KL (KL20) reads about 0.8 of full-vocabulary KL on the same rows.

KL is how far an engine's next-token probabilities are from the BF16 model's on the same text (0 is identical, lower is closer; units: nats).

The quantization's own distance dwarfs the engine difference. On tpurtell's set his official FP8 model scores 0.0216, the 3.25-bit checkpoint 0.0451 (vLLM fork) and 0.0484 (CuteAFD): the engine difference is 7% of the vLLM fork's distance (11% on long code). K4 is closer than K3.25 on both panels: 0.53-0.58 of K3.25's top-20 distance on the teacher panel, 0.69-0.75 on tpurtell's set. The teacher panel is drawn from the corpus K4's producer published, so it can favour K4 (Three checkpoints).
tpurtell's bar counts a paired change below 0.005 KL or 0.5 top-1 points as within it (defined for changes to one engine; applied here by analogy). K3.25 overall is inside on KL (+0.0033, interval to +0.0049), not on top-1 (−0.46 points, to −0.73); long code is not shown inside on either checkpoint (+0.0054, +0.0090); K4 with W8A16 prefill + amax reads +0.0030 (to +0.0049).
On tasks, code and math checks at the 128K cap and the needle checks came out the same on both engines.
Not measured: the distance from BF16 beyond 14.4K tokens (no reference exists there) and on the decode path (scoring runs prompts through prefill).

Pick by workload

What the results show on this machine and these checkpoints; not a general ranking.

Workload	The data shows	Key numbers (vLLM fork first, CuteAFD second)	Evidence
Long code (8-16K), closest to BF16	vLLM fork closer to BF16	KL at 8-16K: 0.0504 vs 0.0557 (K3.25); 0.0334 vs 0.0424 (K4, top-20)	Precision
CuteAFD on K3.25, closer to BF16	W8A16 prefill closes about 59% of the difference	difference +0.0054 → +0.0022; first token +3.8% to +4.5% per cell	Settings
CuteAFD on K4, two RTX GPUs	W8A16 prefill + amax scale rule closes about two thirds; the BF16 options do not fit	difference +0.0090 → +0.0030; first token +4.0% to +4.2% per cell	Settings
One request at a time	CuteAFD generates faster	114-168 vs 120-235 tok/s	Speed
Long prompts, one at a time	CuteAFD's first token sooner	128K: 35.1 s vs 29.6 s	Speed
Short prompts, many at once	v3.0.1: vLLM fork's first token sooner; tokens/s higher on the vLLM fork at 1K × 8, level at 1K × 16. v4.0.0 at 4 streams: CuteAFD's first token level or sooner, tokens/s higher (8 and 16 streams not rerun)	1K × 8: 1.44 s vs 2.76 s, 345-454 vs 321-410 tok/s; 1K × 16: 2.38 s vs 4.31 s, 378-508 vs 397-516 tok/s	Speed
Many long prompts at once	vLLM fork's first tokens sooner; CuteAFD's tokens/s higher	32K × 16: 72 vs 106 s, 100-105 vs 107-112 tok/s	Speed
Repeated long prefix	CuteAFD reuses it	8K repeated: 2.10 vs 0.017 s	Running it
Multi-turn tool calling (agents)	both pass near the test's ceiling; scores overlap; CuteAFD's turns come sooner	tool-eval-bench, 88 scenarios: 160-163 vs 163 of 176 (K3.25), 158-163 vs 158 (K4); median turn 4.8-5.3 vs 2.7-3.1 s at 8 at once	Tool calling
Long replies (to 128K)	same code and math at the 128K cap; CuteAFD finished more code at 32K	128K code: 7/8 vs 7/8; 32K code finished: 0/8 vs 4/8 (K3.25)	Long tasks
Switching models often	CuteAFD starts faster	382 vs 89 s	Running it
Many long contexts at once (K4)	v3.0.1: vLLM fork holds about 3x the KV cache; v4.0.0: CuteAFD holds 1.4x	1,377,179 vs 447,488 tokens (v3.0.1); 1,377,179 vs 1,931,264 (v4.0.0)	Running it
K4 on CuteAFD, two RTX GPUs	context up to 262,144 tokens (MAX_CONTEXT_TOKENS); the 256K-1M needle checks ran on K3.25	262,144 tokens	Running it

Settings measured

The configurations measured here, as files you can copy; with each, what it measured. Measured, not a recommendation. The exact files are in repro/; checkpoints download as in Reproduce, step 1.

vLLM fork v0.9.2, both checkpoints. Startup 697 → 382 s with the JIT caches kept (K3.25).

# the launcher: tpurtell's vLLM fork repository at tag v0.9.0; its start.sh runs the v0.9.2 image pinned below
git clone https://github.com/tpurtell/glm-5.3-flash-ext3-2x-rtx && cd glm-5.3-flash-ext3-2x-rtx && git checkout v0.9.0
export IMAGE=ghcr.io/tpurtell/glm-5.3-flash-exl3-4bpw-2x-rtx@sha256:ea0430aa772e8cedfef1cdf80c30f4962ba43bb0991ac329950067ba1375f3e2
export MODEL_PROFILE=k325                # k4 for the K4 checkpoint
export GPU_MEMORY_UTILIZATION=0.95       # the default 0.97 does not start beside a desktop session
export PORT=127.0.0.1:8001               # the default publishes port 8001 on every interface
# keep the JIT caches inside the cache directory the container already keeps between starts
export DEBUG_ENV="TILELANG_CACHE_DIR=/root/.cache/tilelang
  TRITON_CACHE_DIR=/root/.cache/triton
  TORCHINDUCTOR_CACHE_DIR=/root/.cache/torchinductor
  CUDA_CACHE_PATH=/root/.cache/nv/ComputeCache"
./start.sh

CuteAFD v3.0.1: the launcher. On a host without /dev/infiniband, apply the guard in repro/cuteafd/patches/ first.

git clone https://github.com/tpurtell/cuteafd && cd cuteafd && git checkout v3.0.1
mkdir -p local                           # save the config below as local/<name>.config, then:
./run.sh --config local/<name>.config --restart

CuteAFD on K3.25, as shipped (glm53-flash-exl3-k3.25-core-fp8-2rtx.config; ADDR, MAX_OUTPUT_TOKENS and USAGE are the local settings: the launcher defaults are 0.0.0.0 under host networking, a 4,096-token output limit, and request logging on).

MODEL_ID=wrldsuksgo2mars/GLM-5.3-Flash-EXL3-K3.25-Core-FP8
MODEL_REVISION=d31ebc921f1f098b4a56d37aaccb3cdef4096184
GLM5_FLASH_FP8_MODEL_ID=off
SPECULATOR=dflash2
SPECULATOR_MODEL_ID=incoai/GLM-5.3-Flash-DFlash2
SPECULATOR_MODEL_REVISION=bf582e4eacc1810f76656d1811693ff6c6737d2a
POOL_TOKENS=auto
EXPERT_BACKEND=local
SPARK_COUNT=0
RTX_GPUS=2
COORDINATOR_GPU=0
ADDR=127.0.0.1:8001
MAX_OUTPUT_TOKENS=393216
USAGE=off
COORDINATOR_DOCKER_INFERENCE=ghcr.io/tpurtell/cuteafd-coordinator:v3.0.1@sha256:392db7c0bdd39ec5aacda800f9a3ada70342721a2357ad1ec09ad0ef5981459a

With the measured setting, W8A16 prefill: append the line below. Measured: long-code difference from the vLLM fork +0.0054 → +0.0022, first token +3.8% to +4.5% per cell, decode 0.99x (scoring launches, follow-up F1).

GLM5_FLASH_FP8_PREFILL=off

CuteAFD on K4, as shipped (glm53-flash-exl3-k4-2rtx.config; MAX_CONTEXT_TOKENS=262144 is needed on two RTX GPUs, where the planner refuses K4 at its default). Measured: KV pool 447,488 tokens, 16 requests at once.

MODEL_ID=brandonmusic/GLM-5.3-Flash-tr3-4bpw
MODEL_REVISION=a5fee929cf4888b1824323e33e8a19b60129e025
GLM5_FLASH_FP8_MODEL_ID=off
SPECULATOR=dflash2
SPECULATOR_MODEL_ID=incoai/GLM-5.3-Flash-DFlash2
SPECULATOR_MODEL_REVISION=bf582e4eacc1810f76656d1811693ff6c6737d2a
POOL_TOKENS=auto
MAX_CONTEXT_TOKENS=262144
VISION=off
EXPERT_BACKEND=local
SPARK_COUNT=0
RTX_GPUS=2
COORDINATOR_GPU=0
ADDR=127.0.0.1:8001
MAX_OUTPUT_TOKENS=393216
USAGE=off
COORDINATOR_DOCKER_INFERENCE=ghcr.io/tpurtell/cuteafd-coordinator:v3.0.1@sha256:392db7c0bdd39ec5aacda800f9a3ada70342721a2357ad1ec09ad0ef5981459a

With the measured setting, W8A16 prefill + amax scale rule: append the two lines below. Measured: long-code difference +0.0090 → +0.0030, teacher panel no longer resolved, first token +4.0% to +4.2% per cell, decode level, the same memory.

GLM5_FLASH_FP8_PREFILL=off
FP8_SCALES=amax

Speed in numbers

	K3.25: vLLM fork	K3.25: CuteAFD	K4: vLLM fork	K4: CuteAFD
Decode, 1 stream, tok/s	114-168	120-235	108-164	116-220
Decode per stream, 4 streams (1K-8K), tok/s	59-99	61-117	55-93	63-114
First token, 1K	0.25 s	0.41 s	0.27 s	0.42 s
First token, 8K	2.11 s	1.80 s	2.20 s	1.84 s
First token, 32K	8.70 s	7.26 s	9.05 s	7.38 s
First token, 128K	35.1 s	29.6 s	36.4 s	30.1 s
Startup to the ready line	382 s (697 s without JIT caches)	89 s	257 s warm, 307 s cold	86 s warm, 76 s cold
KV cache, tokens	4,707,515	4,726,784	1,377,179	447,488

Decode: range over content types and 1K-128K prompts; first token: median over content types, one request at a time; 1,024 tokens per request. Startup and KV cache: Running it. Every speed cell: Speed.

How sure are we

K3.25: frozen protocol. After the runs, a separate reviewer who did not run the measurements re-ran the scorers and checked the claims against the receipts; its corrections are applied (Review). It covered the precision and speed blocks and the vLLM fork's long output; it did not cover CuteAFD's long output, the follow-ups F1-F3 (it saw their designs and the F1 baseline speed run), the drift repeats, the startup measurement or the needle pilot. CuteAFD is paired with three vLLM runs from two launches. Thin: 8 long replies per content type, six long documents. K3.25 speed noise: a repeat hours later moved the median by under 2% and single cells by up to 13%; single-cell ratios within ±10% read as level.
K4: frozen protocol; no separate review of any part of it, its settings follow-ups included. CuteAFD is paired with two vLLM launches. K4 speed noise: a ±8% band declared before the run (both engines' drift repeats stayed inside it); single-cell ratios within ±8% read as level.
One machine: two RTX PRO 6000 Blackwell Max-Q cards at 300 W, with a desktop session using about 2 GB on GPU 1; speed on other power limits or cards was not measured.

Terms

AFD: attention/FFN disaggregation: attention and the expert layers run as separate workers.
BF16 model: the original, unquantized model; the precision reference.
K3.25, K4, K4-Core-K6: the three quantized checkpoints: 3.25-bit, 4-bit, and 4-bit experts with a 6-bit core (EXL3).
FP8: 8-bit floating point, CuteAFD's format for the other weights.
KL: distance between next-token distributions, in nats; 0 is identical.
Coarse KL, KL20, KL_pb20, KL20n: KL over the reference's top 1,024 tokens plus the rest (tpurtell's rule); over each engine's top 20 (K4, teacher panel); the top-20 form of tpurtell's rule (K4, fidelity set); top 20 shared by two engines (K4, long texts).
Top-1: how often the most likely next token matches the reference's.
P-A, P-B, Q2: the text sets: BF16 teacher texts (2K tokens), tpurtell's fidelity set (to 14.4K), six long documents.
Long code (8-16K): P-B's longest prompts, all code reading.
95% interval, resolved: a difference's likely range; resolved if it excludes zero.
Level: a speed ratio inside the noise band: not a difference.
Prefill, decode: reading the prompt; generating tokens.
First token: time until the first generated token.
Stream: one request in flight.
W8A8, W8A16: CuteAFD prefill with 8-bit or 16-bit activations.
Scale rule: how CuteAFD makes FP8 scales from BF16 weights at load: pow2 (powers of two, the default), amax (from the largest value), best (the launcher's third rule).
KDA, head: linear-attention projections; the final vocabulary layer.
Drafter: DFlash2, a small model that proposes tokens for the main model to check (speculative decoding).
Serving, scoring launch: the engine as users run it; the same engine with the endpoints that return per-position probabilities for precision scoring.
Block Q, T, L: the measurement blocks: precision, speed (throughput), long output.
Run labels: K3.25: V1 and V1r (vLLM fork, one launch), V2 (a second launch), C1 (CuteAFD), C0 (CuteAFD control on K3.25-v1); K4: V-r1 and V-r2 (two vLLM launches), C-r1 (CuteAFD).
Prefix cache: reuse of a repeated prompt's processed state.
JIT caches: kernels compiled at startup, kept between starts.
Thinking: the model's reasoning before its answer; always on here.
tool-eval-bench: an open benchmark of 88 multi-turn tool-calling scenarios, each graded pass (2 points), partial (1) or fail (0); 176 points in all. Its own rubric is on the Tool calling page.

Pages

Checkpoints (K3.25, K4, K4-Core-K6) · CuteAFD v4 (the v4.0.0 rerun) · Tool calling (multi-turn tool use, both engines) · Thinking length (investigation: the 6-bit core, the experts and when the model stops) · Precision (closeness to BF16) · Settings (every CuteAFD setting) · Speed (every speed cell) · Long tasks (long replies, long prompts) · Running it (operator notes) · Method (protocols and timeline) · Audit (the K3.25 review) · Research (deep dives) · Reproduce (every step)