Long tasks: long replies and very long prompts

At the longest output cap the two engines did the same on code and math: code passed every hidden test in 7 of 8 replies on both with K3.25 and 8 of 8 with K4, math 8 of 8. At the 32K cap CuteAFD finished more code tasks (0 of 8 on the vLLM fork against 4 on CuteAFD with K3.25, 0 against 5 with K4), consistent with its shorter code replies (0.74x and 0.77x the vLLM fork's median length at the last cap). Descriptive: n = 8 per content type and stage.

Needles in very long prompts (exploratory)

Exploratory: one request per prompt and engine, not under a frozen protocol. tpurtell's two needle designs from his vLLM fork, a secret code near the start and six records spread from 5% to 99% of the prompt, at 256K, 512K and 1M tokens, on both engines with K3.25. Every reply finished on its own and found every needle on both engines.

PromptPrompt tokensFirst token: vLLM forkCuteAFDResult: vLLM forkCuteAFD
single-256k262,15662.5 s58.0 sfoundfound
six-256k262,15663.2 s58.5 s6/6 in order6/6 in order
single-512k524,300132.2 s123.2 sfoundfound
six-512k524,300129.8 s122.5 s6/6 in order6/6 in order
single-1m1,000,012270.5 s255.4 sfoundfound
six-1m1,000,012269.7 s254.2 s6/6 in order6/6 in order

Receipts without reply text: results/2026-10-27-long-context-pilot. It shows that both engines accept and answer 1M-token prompts; it does not rank them.

Long output: replies up to 128K tokens

Four task types with machine checks: multi-file Python projects with hidden tests, MATH-500 level-5 problems, 10,000-12,000-word essays, and product catalogues as JSON. Each stage raises the output cap; L3s repeats one prompt per type one request at a time. K3.25 ran 8 requests in flight with caps of 8K, 32K and 128K tokens; K4 ran 2 in flight with caps of 8,192, 32,768 and 130,048, because K4's grid was common to its protocol's arms (Method). Math replies were short on both: median 566 (vLLM fork) and 634 (CuteAFD) tokens at the last cap on K3.25.

K3.25: long-output checks and reply length, both engines (more passing is better; descriptive, small n)

What this shows. Top: how many of the n replies passed their check, per stage and content type (8 requests in flight). Bottom: CuteAFD's median reply length divided by the vLLM fork's.

vLLM forkCuteAFDvLLM fork (length 1.0)
Passed, out of n
0%25%50%75%100%← fewer passmore pass →L1, 8K capL2, 32K capL3, 128K capL3s, 128K cap, 1 request at a timeL1 code, all tests pass: vLLM fork 0/8, CuteAFD 0/8 (descriptive, n=8)code: all tests passvLLM 0/8 · CuteAFD 0/8L1 reasoning, correct: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)reasoning: correctvLLM 8/8 · CuteAFD 8/8L1 prose, finished: vLLM fork 0/8, CuteAFD 0/8 (descriptive, n=8)prose: finishedvLLM 0/8 · CuteAFD 0/8L1 JSON, valid: vLLM fork 0/8, CuteAFD 0/8 (descriptive, n=8)JSON: validvLLM 0/8 · CuteAFD 0/8L2 code, all tests pass: vLLM fork 0/8, CuteAFD 4/8 (descriptive, n=8)code: all tests passvLLM 0/8 · CuteAFD 4/8L2 reasoning, correct: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)reasoning: correctvLLM 8/8 · CuteAFD 8/8L2 prose, finished: vLLM fork 1/8, CuteAFD 0/8 (descriptive, n=8)prose: finishedvLLM 1/8 · CuteAFD 0/8L2 JSON, valid: vLLM fork 2/8, CuteAFD 2/8 (descriptive, n=8)JSON: validvLLM 2/8 · CuteAFD 2/8L3 code, all tests pass: vLLM fork 7/8, CuteAFD 7/8 (descriptive, n=8)code: all tests passvLLM 7/8 · CuteAFD 7/8L3 reasoning, correct: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)reasoning: correctvLLM 8/8 · CuteAFD 8/8L3 prose, finished: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)prose: finishedvLLM 8/8 · CuteAFD 8/8L3 JSON, valid: vLLM fork 6/8, CuteAFD 4/8 (descriptive, n=8)JSON: validvLLM 6/8 · CuteAFD 4/8L3s code, all tests pass: vLLM fork 1/1, CuteAFD 1/1 (descriptive, n=1)code: all tests passvLLM 1/1 · CuteAFD 1/1L3s reasoning, correct: vLLM fork 1/1, CuteAFD 1/1 (descriptive, n=1)reasoning: correctvLLM 1/1 · CuteAFD 1/1L3s prose, finished: vLLM fork 1/1, CuteAFD 1/1 (descriptive, n=1)prose: finishedvLLM 1/1 · CuteAFD 1/1L3s JSON, valid: vLLM fork 1/1, CuteAFD 1/1 (descriptive, n=1)JSON: validvLLM 1/1 · CuteAFD 1/1
Median reply length, CuteAFD ÷ vLLM fork
0.67x0.8x0.9x1.1x1x← CuteAFD shorterlonger →vLLM forkL1, 8K capL2, 32K capL3, 128K capL3s, 128K cap, 1 request at a timeL1 code: median reply CuteAFD 8,192 vs vLLM fork 8,192 tokens = 1.00x; every reply on both engines reached the capcode1.00x (both at cap)L1 reasoning: median reply CuteAFD 596 vs vLLM fork 626 tokens = 0.95xreasoning0.95x (596 vs 626)L1 prose: median reply CuteAFD 8,192 vs vLLM fork 8,192 tokens = 1.00x; every reply on both engines reached the capprose1.00x (both at cap)L1 JSON: median reply CuteAFD 8,192 vs vLLM fork 8,192 tokens = 1.00x; every reply on both engines reached the capJSON1.00x (both at cap)L2 code: median reply CuteAFD 32,692 vs vLLM fork 32,768 tokens = 1.00xcode1.00x (32,692 vs 32,768)L2 reasoning: median reply CuteAFD 614 vs vLLM fork 802 tokens = 0.77xreasoning0.77x (614 vs 802)L2 prose: median reply CuteAFD 32,768 vs vLLM fork 32,768 tokens = 1.00xprose1.00x (32,768 vs 32,768)L2 JSON: median reply CuteAFD 32,768 vs vLLM fork 32,768 tokens = 1.00xJSON1.00x (32,768 vs 32,768)L3 code: median reply CuteAFD 38,190 vs vLLM fork 51,298 tokens = 0.74xcode0.74x (38,190 vs 51,298)L3 reasoning: median reply CuteAFD 634 vs vLLM fork 566 tokens = 1.12xreasoning1.12x (634 vs 566)L3 prose: median reply CuteAFD 39,344 vs vLLM fork 48,832 tokens = 0.81xprose0.81x (39,344 vs 48,832)L3 JSON: median reply CuteAFD 67,381 vs vLLM fork 74,589 tokens = 0.90xJSON0.90x (67,381 vs 74,589)L3s code: median reply CuteAFD 56,847 vs vLLM fork 85,441 tokens = 0.67xcode0.67x (56,847 vs 85,441)L3s reasoning: median reply CuteAFD 629 vs vLLM fork 675 tokens = 0.93xreasoning0.93x (629 vs 675)L3s prose: median reply CuteAFD 33,168 vs vLLM fork 30,382 tokens = 1.09xprose1.09x (33,168 vs 30,382)L3s JSON: median reply CuteAFD 12,367 vs vLLM fork 17,151 tokens = 0.72xJSON0.72x (12,367 vs 17,151)

How to use it. Read the long-cap rows (L3, L3s): where the two marks sit together the engines did the same; with n = 8, one or two replies apart is not evidence of a difference.

Table view: the exact numbers
StageContentn per enginevLLM: passedCuteAFD: passedvLLM: finished / at capCuteAFD: finished / at capvLLM median tokensCuteAFD median tokensLength ratio
L1code80/80/80 / 80 / 88,1928,1921.00x
L1reasoning88/88/88 / 08 / 06265960.95x
L1prose80/80/80 / 80 / 88,1928,1921.00x
L1JSON80/80/80 / 80 / 88,1928,1921.00x
L2code80/84/80 / 84 / 432,76832,6921.00x
L2reasoning88/88/88 / 08 / 08026140.77x
L2prose81/80/81 / 70 / 832,76832,7681.00x
L2JSON82/82/82 / 62 / 632,76832,7681.00x
L3code87/87/88 / 08 / 051,29838,1900.74x
L3reasoning88/88/88 / 08 / 05666341.12x
L3prose88/88/88 / 08 / 048,83239,3440.81x
L3JSON86/84/86 / 27 / 174,58967,3810.90x
L3scode11/11/11 / 01 / 085,44156,8470.67x
L3sreasoning11/11/11 / 01 / 06756290.93x
L3sprose11/11/11 / 01 / 030,38233,1681.09x
L3sJSON11/11/11 / 01 / 017,15112,3670.72x

Passed: code, every hidden test passes; reasoning, the correct MATH-500 answer; prose, the reply finished on its own; JSON, schema, count and ids valid. Descriptive: 4 prompts per content type, 2 seeds each (n = 8; 1 at L3s). No interval is published for these counts or for the length ratio. The code, JSON and prose tasks were sized for thinking off: at the first two caps, and for the 1,030-record JSON task at the last, the reasoning leaves too little room for the answer, so those cells measure finishing against the cap rather than task completion.

K4: long-output checks and reply length, both engines (more passing is better; descriptive, small n)

What this shows. Top: how many of the n replies passed their check, per stage and content type (2 requests in flight). Bottom: CuteAFD's median reply length divided by the vLLM fork's.

vLLM forkCuteAFDvLLM fork (length 1.0)
Passed, out of n
0%25%50%75%100%← fewer passmore pass →L1, 8,192 capL2, 32,768 capL3, 130,048 capL3s, 130,048 cap, 1 request at a timeL1 code, all tests pass: vLLM fork 0/8, CuteAFD 0/8 (descriptive, n=8)code: all tests passvLLM 0/8 · CuteAFD 0/8L1 reasoning, correct: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)reasoning: correctvLLM 8/8 · CuteAFD 8/8L1 prose, finished: vLLM fork 0/8, CuteAFD 0/8 (descriptive, n=8)prose: finishedvLLM 0/8 · CuteAFD 0/8L1 JSON, valid: vLLM fork 0/8, CuteAFD 0/8 (descriptive, n=8)JSON: validvLLM 0/8 · CuteAFD 0/8L2 code, all tests pass: vLLM fork 0/8, CuteAFD 5/8 (descriptive, n=8)code: all tests passvLLM 0/8 · CuteAFD 5/8L2 reasoning, correct: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)reasoning: correctvLLM 8/8 · CuteAFD 8/8L2 prose, finished: vLLM fork 2/8, CuteAFD 2/8 (descriptive, n=8)prose: finishedvLLM 2/8 · CuteAFD 2/8L2 JSON, valid: vLLM fork 2/8, CuteAFD 2/8 (descriptive, n=8)JSON: validvLLM 2/8 · CuteAFD 2/8L3 code, all tests pass: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)code: all tests passvLLM 8/8 · CuteAFD 8/8L3 reasoning, correct: vLLM fork 8/8, CuteAFD 8/8 (descriptive, n=8)reasoning: correctvLLM 8/8 · CuteAFD 8/8L3 prose, finished: vLLM fork 7/8, CuteAFD 8/8 (descriptive, n=8)prose: finishedvLLM 7/8 · CuteAFD 8/8L3 JSON, valid: vLLM fork 5/8, CuteAFD 3/8 (descriptive, n=8)JSON: validvLLM 5/8 · CuteAFD 3/8L3s code, all tests pass: vLLM fork 0/1, CuteAFD 1/1 (descriptive, n=1)code: all tests passvLLM 0/1 · CuteAFD 1/1L3s reasoning, correct: vLLM fork 1/1, CuteAFD 1/1 (descriptive, n=1)reasoning: correctvLLM 1/1 · CuteAFD 1/1L3s prose, finished: vLLM fork 1/1, CuteAFD 1/1 (descriptive, n=1)prose: finishedvLLM 1/1 · CuteAFD 1/1L3s JSON, valid: vLLM fork 1/1, CuteAFD 1/1 (descriptive, n=1)JSON: validvLLM 1/1 · CuteAFD 1/1
Median reply length, CuteAFD ÷ vLLM fork
0.67x0.8x0.9x1.1x1x← CuteAFD shorterlonger →vLLM forkL1, 8,192 capL2, 32,768 capL3, 130,048 capL3s, 130,048 cap, 1 request at a timeL1 code: median reply CuteAFD 8,192 vs vLLM fork 8,192 tokens = 1.00x; every reply on both engines reached the capcode1.00x (both at cap)L1 reasoning: median reply CuteAFD 631 vs vLLM fork 536 tokens = 1.18xreasoning1.18x (631 vs 536)L1 prose: median reply CuteAFD 8,192 vs vLLM fork 8,192 tokens = 1.00x; every reply on both engines reached the capprose1.00x (both at cap)L1 JSON: median reply CuteAFD 8,192 vs vLLM fork 8,192 tokens = 1.00x; every reply on both engines reached the capJSON1.00x (both at cap)L2 code: median reply CuteAFD 31,018 vs vLLM fork 32,768 tokens = 0.95xcode0.95x (31,018 vs 32,768)L2 reasoning: median reply CuteAFD 624 vs vLLM fork 599 tokens = 1.04xreasoning1.04x (624 vs 599)L2 prose: median reply CuteAFD 32,768 vs vLLM fork 32,768 tokens = 1.00xprose1.00x (32,768 vs 32,768)L2 JSON: median reply CuteAFD 32,768 vs vLLM fork 32,768 tokens = 1.00xJSON1.00x (32,768 vs 32,768)L3 code: median reply CuteAFD 39,378 vs vLLM fork 51,343 tokens = 0.77xcode0.77x (39,378 vs 51,343)L3 reasoning: median reply CuteAFD 558 vs vLLM fork 706 tokens = 0.79xreasoning0.79x (558 vs 706)L3 prose: median reply CuteAFD 34,842 vs vLLM fork 32,764 tokens = 1.06xprose1.06x (34,842 vs 32,764)L3 JSON: median reply CuteAFD 55,715 vs vLLM fork 66,812 tokens = 0.83xJSON0.83x (55,715 vs 66,812)L3s code: median reply CuteAFD 64,335 vs vLLM fork 55,168 tokens = 1.17xcode1.17x (64,335 vs 55,168)L3s reasoning: median reply CuteAFD 638 vs vLLM fork 639 tokens = 1.00xreasoning1.00x (638 vs 639)L3s prose: median reply CuteAFD 33,344 vs vLLM fork 29,505 tokens = 1.13xprose1.13x (33,344 vs 29,505)L3s JSON: median reply CuteAFD 8,596 vs vLLM fork 15,788 tokens = 0.54xJSON0.54x (8,596 vs 15,788)

How to use it. Read the long-cap rows (L3, L3s): where the two marks sit together the engines did the same; with n = 8, one or two replies apart is not evidence of a difference.

Table view: the exact numbers
StageContentn per enginevLLM: passedCuteAFD: passedvLLM: finished / at capCuteAFD: finished / at capvLLM median tokensCuteAFD median tokensLength ratio
L1code80/80/80 / 80 / 88,1928,1921.00x
L1reasoning88/88/88 / 08 / 05366311.18x
L1prose80/80/80 / 80 / 88,1928,1921.00x
L1JSON80/80/80 / 80 / 88,1928,1921.00x
L2code80/85/80 / 85 / 332,76831,0180.95x
L2reasoning88/88/88 / 08 / 05996241.04x
L2prose82/82/82 / 62 / 632,76832,7681.00x
L2JSON82/82/82 / 62 / 632,76832,7681.00x
L3code88/88/88 / 08 / 051,34339,3780.77x
L3reasoning88/88/88 / 08 / 07065580.79x
L3prose87/88/87 / 18 / 032,76434,8421.06x
L3JSON85/83/86 / 27 / 166,81255,7150.83x
L3scode10/11/11 / 01 / 055,16864,3351.17x
L3sreasoning11/11/11 / 01 / 06396381.00x
L3sprose11/11/11 / 01 / 029,50533,3441.13x
L3sJSON11/11/11 / 01 / 015,7888,5960.54x

Passed: code, every hidden test passes; reasoning, the correct MATH-500 answer; prose, the reply finished on its own; JSON, schema, count and ids valid. Descriptive: 4 prompts per content type, 2 seeds each (n = 8; 1 at L3s). No interval is published for these counts or for the length ratio. The code, JSON and prose tasks were sized for thinking off: at the first two caps, and for the 1,030-record JSON task at the last, the reasoning leaves too little room for the answer, so those cells measure finishing against the cap rather than task completion.

Per stage and content: l/SUMMARY.md (K3.25), the Block L table in the K4 run README.