Running it: what an operator needs to know

Launch configurations

The configurations measured are on the home page as copyable blocks (Settings measured); the exact files used are in repro/cuteafd/configs (K3.25), repro/k4 (K4) and described in Reproduce. Scoring launches for precision add CuteAFD's bench endpoint (ENABLE_BENCH=on, an API key file, FULL_PREFILL_LOGITS=on) or, on the vLLM fork, a prompt-logprob capture plugin; serving does not need them.

Each item is the behaviour as measured or as read in source, with how it is known. None is a claim about output quality.

Capacity

K3.25 leaves room for about 4.7 million tokens of KV cache on both engines; K4 for far fewer (1.38 million on the vLLM fork, 447,488 on CuteAFD). K3.25's weights take about 69 GiB per GPU on the vLLM fork (its startup log); K4's weight size is not in the published files. The smaller K4 pools are consistent with its 4-bit experts taking more memory than K3.25's 3.25-bit ones, but that was not measured, and why CuteAFD's K4 pool is smaller than the vLLM fork's was not examined beyond the planner's lines below. These are v3.0.1 pools. On CuteAFD v4.0.0 the serving pools are 5,799,424 (K3.25), 1,931,264 (K4) and 2,240,512 (K4-Core-K6) tokens, against the vLLM fork's 4,762,898 and 1,377,179 in the same run (results/2026-10-30-cuteafd-v4).

CuteAFD on K4 needs one local setting on two RTX GPUs: at its defaults the planner refuses K4 (pool 260,608 tokens, below its 1,048,576-token context floor); MAX_CONTEXT_TOKENS=262144 gives a 447,488-token pool beside the desktop's context on GPU 1 (the dry run predicted 605,696). Its BF16 KDA and BF16 head options do not fit a K4 serving launch. With that setting CuteAFD serves K4 with prompts up to 262,144 tokens; the needle checks at 256K-1M tokens ran on K3.25.

Startup

Startup time is the cost of switching models. "Ready" here is the engine's ready line in its server log: for K3.25 counted from container start (measured from the server logs: results/2026-10-27-vllm-startup-cache). CuteAFD v3.0.1 is ready in 89 s: its kernels are compiled ahead of time and shipped in the image (PROGRAMS.json), so there is no startup JIT; its CPU planner computes the memory layout up front, so there is no profiling pass; and the experts load through a parallel shared read in 30.6 s. On this setup, the container keeps ~/.cache between starts but not the TileLang, Triton, TorchInductor and CUDA kernel caches, so the vLLM fork v0.9.2 compiled 49 TileLang kernels at every start and was ready in 697 s (11.6 min). Persisting those caches (pointing TILELANG_CACHE_DIR, TRITON_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR and CUDA_CACHE_PATH into the cache directory the container already keeps) brings it to 382 s (6.4 min, −45%):

The first start after the change fills the caches and takes 714 s; later starts are warm. What remains is mostly weight loading (about 4 minutes including loader overhead). Other projects keep these caches between starts the same way (blackwell-llm-docker#73, which reports about 3.2 min of TileLang compiling per restart; sglang#39364; vLLM's Docker guide mounts the Hugging Face cache and ~/.cache/vllm).

On K4, counted from the start command (Stage 1 of the K4 run; one cold and one warm start per engine, descriptive). The vLLM fork's 257 s warm start is not directly comparable with K3.25's 382 s: a different checkpoint, measured in a different session (K3.25: the JIT-cache measurement of 2026-10-27 morning; K4: that evening, with the same cache overlay); what differs between the two starts was not broken down. CuteAFD's K4 cold and warm starts (76 s and 86 s) are one start each, so their 10 s difference is not read as an effect.

Prefix cache

• vLLM prefix-cache reuse is in whole 4,352-token blocks, without the last two blocks of a prompt (measured, F2): repeated prompts of 1K to 8,704 tokens got 0 cached tokens; a 17,408-token prompt got 4,352, then 8,704. The server log shows the MLA cache groups at block size 4,352 in align mode. The v0.9.0 opt-in boundary lookup (GLM53_DFLASH_BOUNDARY_LOOKUP, default 0) was not tested.
• CuteAFD restored a repeated 8K prompt from its prefix cache in 0.017 s (Block T cached cell; the vLLM fork took 2.10 s, uncached). Its serving launch keeps 20 entries. The K4 scoring launch turns the prefix cache off (PREFIX_CACHE_ENTRIES=0) to fit the full prefill-logits buffer beside the desktop's context; scoring probes never read the prefix cache, so no scored value changes.

Thinking off and control tokens

• Thinking off: the two stacks render different prompts (prompt measured with one request; parser behaviour read in source). The checkpoint's chat template, byte-identical on both stacks, has no thinking switch and always ends with <|assistant|><think>. CuteAFD renders enable_thinking: false as an empty <think></think> (one more prompt token). vLLM renders the same prompt with the flag on or off (147 prompt tokens both times), and with it off its reasoning parser returns the generated reasoning as content. Thinking off would therefore give the two engines different prompts and different generated content, so every generation request in Blocks T and L runs with thinking on.
• Control-token strings in user text (measured in Block T and the K4 smoke checks). The 128K code prompts contain literal <think> and </think> strings; the vLLM stack parses them as control tokens, CuteAFD keeps them as text, so those prompts are 24 tokens longer on CuteAFD.
• Multi-turn reasoning history (rendered offline with each template; prompt tokens measured on both engines, results/2026-11-02-tool-calling). Clients that send an assistant turn's reasoning back in the history (tool-calling agents do) get different prompts by default: K4's chat template drops reasoning from earlier user turns, the template the vLLM fork serves keeps it, and CuteAFD renders each checkpoint's own template. With chat_template_kwargs: {"clear_thinking": false} all three templates (K4, the vLLM fork's, K4-Core-K6) render byte-identical text, and both engines reported the same prompt tokens on every checkpoint.

Launcher defaults that matter

• CuteAFD (read in the launcher): output limit 4,096 tokens, which clamps longer requests (the runs here set MAX_OUTPUT_TOKENS=393216; with it, a 12,000-token request ran to its max_tokens, measured); request logging on unless USAGE=off; listen address 0.0.0.0 under host networking unless ADDR is set; the coordinator launch passes --device=/dev/infiniband, which Docker requires to exist (the runs here pass it only when it exists, in the same form as the repository's wip.sh: repro/cuteafd/patches/). Its swiglu_limit is a compiled constant (10, this checkpoint's declared value); the value in config.json is parsed but not compared with it.
• vLLM fork: the runs use the v0.9.2 release image with the v0.9.0 repository's start.sh (its IMAGE override; the launcher is otherwise unchanged). Its GPU memory utilization defaults to 0.97, which does not start beside a desktop session on GPU 1 (the runs use 0.95), and its port is published as PORT (default 8001, on every interface; the runs use 127.0.0.1:8001). The K4 launches also carried the stack's local environment overlay unchanged: the JIT caches above and a host-memory stopgap that calls malloc_trim in the workers periodically (noted in repro/k4/vllm/env.sh).

Determinism and drafting

• vLLM's multi-chunk prefill is not run-to-run deterministic (measured, Block Q captures, each run alone on the server). Rows inside the first 2,045-token prefill chunk are bit-identical between runs; rows beyond it almost never are. CuteAFD's repeats are byte-identical. CuteAFD is therefore compared with the mean of three vLLM runs from two launches (K3.25) or of two launches (K4).
• Same seed, same prompt (K4 smoke check, 3 prompts): CuteAFD returned identical text 3 of 3 times, the vLLM fork 1 of 3; the cause was not examined. Repeats in these runs always use distinct seeds.
• Draft acceptance (measured on vLLM, drift subset): 1.53 of 3 drafted tokens accepted per verify step (51%). CuteAFD drafts adaptively and exposes no acceptance counter on its API.

Other engine observations

• CuteAFD converts K4's BF16 projections to FP8 at load (read in source before the K4 freeze): attention, dense and shared projections in 128×128 blocks with pow2 scales, head and KDA projections per row. On K3.25 those projections came from the official FP8 checkpoint (Core-FP8); the vLLM fork serves them in BF16.
• CuteAFD's router logits differ slightly between its two GEMM routes (measured on one GPU, exploratory): across the 160/161-row switch 0.36% of logits are bit-equal, the largest difference is 8.2e-5, and the top-8 expert set differs in 2 of 64,000 rows; within each route the logits do not depend on the row count, and decode steps stay on one route. This is numerical variation between routes, the same kind as observed on the vLLM engine: results/2026-10-26-e1-router-route.
• Earlier vLLM-engine items D1-D7 in CuteAFD (read in source, not measured): the source audit maps the seven items from earlier evaluation (five of them addressed in tpurtell's v0.9.2) onto CuteAFD. For four, CuteAFD covers the same case in its own code; three do not arise in its design: results/2026-10-26-cuteafd-setup-checks.