Datasetpublic

witcheer/rtx-5090-benchmarks

RTX 5090 LLM Benchmarks Speed and quality benchmarks for quantized LLMs on NVIDIA RTX 5090 32GB, measured with llm-bench-rig. Quality Benchmarks Generative evaluation through llama-server chat completions. Replicates standard benchmark methodology using custom evaluators — no lm-evaluation-harness dependency. Results are split by reasoning mode: comparing a thinking-on (reasoning) model's quality against a thinking-off model is apples-to-oranges, so the two groups… See the full description on the dataset page: https://huggingface.co/datasets/witcheer/rtx-5090-benchmarks.

sourceHugging Facemitupdated 2d agoView on Hugging Face

1likes1.2kdownloads

Dataset Card

RTX 5090 LLM Benchmarks

Speed and quality benchmarks for quantized LLMs on NVIDIA RTX 5090 32GB, measured with llm-bench-rig.

Quality Benchmarks

Generative evaluation through llama-server chat completions. Replicates standard benchmark methodology using custom evaluators — no lm-evaluation-harness dependency.

Results are split by reasoning mode: comparing a thinking-on (reasoning) model's quality against a thinking-off model is apples-to-oranges, so the two groups are ranked separately. q_avg is the mean of the five tasks.

Thinking OFF (non-reasoning · direct answer)

Model	Params	Quant	MMLU	ARC-C	HellaSwag	GSM8K	HumanEval	q_avg
Gemma 4 31B-it	30.70B	Q6_K	87.8	97.6	92.0	97.5	96.3	94.2
Qwopus3.6-27B-Coder	27.32B	Q5KM	87.5	96.8	95.2	97.5	93.3	94.1
Qwen3.6-27B²	26.90B	Q6_K	87.9	96.9	95.4	97.3	93.3	94.2
Qwen3.6-27B²	26.90B	NVFP4³	87.6	96.4	95.4	97.5	93.3	94.1
Qwen3.8-Flash-Next¹⁶	125B (10B active)	UD-Q2KXL	88.5	97.6	95.4	95.9	92.1	93.9
Qwopus3.6-27B-Coder-Compat	27.32B	Q6_K	87.9	96.7	95.3	97.8	90.9	93.7
Qwable-5-27B-Coder	26.90B	Q6_K	87.9	97.1	95.5	97.0	90.9	93.7
Qwen3.8-27B	27.32B	Q6_K¹⁰	85.3	96.7	94.3	97.5	94.5	93.7
Qwen3.8-27B	27.32B	Q8_0¹⁰	85.2	96.8	94.4	97.4	94.5	93.7
Qwen3.8-27B	27.32B	BF16¹²	85.3	96.8	94.3	97.4	93.9	93.5
Qwen3.8-27B	27.32B	UD-Q4KXL¹⁰	85.1	96.6	94.4	97.3	93.9	93.5
Qwen3.6-35B-A3B	34.66B	UD-Q4KM	85.0	95.7	93.3	96.7	95.7	93.3
Qwen3.8-27B	27.32B	Q4KM¹⁰	85.0	96.8	94.3	97.1	92.7	93.2
Qwen3.6-27B	26.90B	NVFP4-GGUF⁴	87.0	96.7	94.9	97.1	90.2	93.2
Qwen3.6-27B²	26.90B	AWQ-int4⁵	87.5	96.4	95.2	97.3	89.0	93.1
Qwen3.8-27B	27.32B	UD-IQ3_XXS¹⁰	84.3	96.3	93.8	96.8	92.1	92.7
Qwen3.8-27B	27.32B	NVFP4¹¹	84.3	96.9	94.3	97.1	89.6	92.5
Qwen3.6-35B-A3B	34.66B	Escha-W2 2-bit⁷	84.2	95.4	92.7	96.6	93.3	92.4
Ling-3.0-flash⁶	127.49B	Q3KM	84.0	96.3	93.2	93.5	92.1	91.8
Qwen3-Coder-Next	79.67B	UD-Q2KXL	83.7	96.0	89.3	96.0	93.3	91.7
Ling-3.0-flash⁶	127.49B	IQ3_XXS	83.3	96.1	92.2	93.0	93.3	91.6
Qwen3.8-27B	27.32B	UD-IQ2_M¹⁰	83.7	94.9	93.7	96.7	88.4	91.5
Qwen3.8-27B	27.32B	UD-IQ2_XXS¹⁰	82.0	95.7	92.3	95.1	89.0	90.8
Ling-3.0-flash⁶	127.49B	IQ2_M	82.3	95.6	91.9	92.3	89.0	90.2
Ornith 1.5 35B-A3B¹³	35.51B	Q4KM	82.2	94.1	91.0	91.7	87.8	89.4
Ornith 1.5 35B-A3B¹³	35.51B	NVFP4	81.8	94.9	90.9	92.3	86.6	89.3
Gemma 4 12B-it	11.91B	Q6_K	78.9	94.0	81.6	96.4	87.2	87.6
gpt-oss-20b	20.91B	Q4KM	78.6	94.6	74.5	94.8	94.5	87.4
Nemotron-3.5-Lightning	31.58B	Q5KM⁹	78.0	92.7	82.1	86.9	81.7	84.3
Nemotron-3.5-Lightning	31.58B	NVFP4⁸	77.7	92.6	81.4	87.0	81.7	84.1
Nemotron-3.5-Lightning	31.58B	IQ3_XXS⁹	78.0	92.4	81.1	86.4	82.3	84.0
Nemotron-3.5-Lightning	31.58B	IQ2_M⁹	77.2	92.1	80.3	87.3	82.9	83.9
Nemotron-3.5-Lightning	31.58B	Q4KM	77.9	92.2	80.6	85.6	82.3	83.7
Ornith 1.5 9B¹⁴	8.95B	Q6_K	78.2	93.4	86.5	86.2	73.2	83.5
Nemotron-3.5-Lightning	31.58B	IQ4_XS⁹	77.9	92.6	82.2	84.7	79.9	83.5
Nemotron-3.5-Lightning	31.58B	Q3KM⁹	77.1	91.7	81.3	86.4	79.3	83.2
Nemotron-3.5-Lightning	31.58B	IQ2_XXS⁹	76.8	92.2	75.7	84.9	81.7	82.3
Nemotron-3-Nano	31.58B	UD-Q4KXL	74.5	89.9	75.6	90.5	80.5	82.2
Nemotron-Cascade-2	31.58B	Q4KM	74.4	91.5	75.7	87.1	79.3	81.6
Qwopus3.8-27B-Flash¹⁵	27.32B	Q6_K	84.5	96.7	94.5	95.8	26.8	79.7
North-Mini-Code-1.0†	30.48B	Q6_K	73.3	60.2	70.8	95.8	86.6	77.4

† North-Mini-Code-1.0 is a reasoning model, run think-OFF for board parity. ARC-Challenge is reasoning-gated: 60.2 think-OFF to ~95 think-ON (+35), which deflates its q_avg. See the report.

² Re-banked 2026-07-15 under the pinned harness (rig ec00ff0, llama-server b9653); the re-bank moved HumanEval by one passing problem (92.7 → 93.3) and nothing else beyond 0.05. ³ unsloth compressed-tensors NVFP4 (23.4GB, 303 modules kept high-precision, lmhead dequanted), served via vLLM 0.21.0 on the native cutlass sm120 FP4 path, measured 2026-07-15 under the same pin. Protocol and per-suite deltas: report addendum. ⁴ NVFP4-GGUF (s-batman MTP GGUF, 14.6GB, flat quant), llama.cpp b9365 BLACKWELL_NATIVE_FP4, June numbers, pre-pin: NVFP4 vs Q6_K. Rows ³ and ⁴ share a format name and nothing else; the 0.9 qavg spread between them is recipe coverage (which modules stay high-precision), not the FP4 format. ⁵ QuantTrio true AWQ (4-bit g128 gemm, 8 modules ignored — flat recipe, 21.9GB), served via vLLM 0.21.0 awqmarlin, measured 2026-07-16 under the same pin. sglang 0.5.14 cannot serve this artifact (hybrid-GDN dtype wall: degenerate output at bf16, crash at fp16) — report. The recipe-coverage rule now holds cross-format: flat recipes cluster (⁴ 93.2, ⁵ 93.1) while the protected recipe (³) holds the baseline.

⁶ 127.5B-total / 5.1B-active hybrid MoE (bailingmoe3), run 2026-08-08/09 on the open llama.cpp PR #26608 branch (head 0266ebca6) — upstream support had not merged at run time. All 512 routed experts in system RAM (--n-cpu-moe 99): Q3KM 46.0 tok/s tg128 at 3.8GB VRAM peak (report). Three-quant ladder measured 2026-08-09 under the identical harness: IQ3XXS 39.6 tok/s ([report](reports/ling-3-0-flash-iq3-xxs.md)), IQ2M 45.6 tok/s (report) — decode speed is non-monotonic across the ladder (IQ3 dequant cost), prefill scales with file size. Companion: `--n-cpu-moe` offload-curve sweep (peak 66 tok/s at 27.5GiB VRAM; the optimum sits ~4GiB below the 32GB ceiling).

⁷ EschaLabs eschamoe 2-bit (12.3GB: 2/3-bit experts, int8 dense), served via the vendor's closed-source escha-sglang runtime (their published 5090 recipe, THINK=0) — the only non-open serving stack on this board, flagged accordingly. Same five-task harness over the server's OpenAI-compatible /v1; 285.7 tok/s single-stream decode measured (vendor claim 283, confirmed). Same base model as the UD-Q4KM row two lines up: the 2-bit build gives up 0.9 q_avg (concentrated in HumanEval, −2.4) for a 40% smaller file that also fits 16GB cards. Report.

⁸ Official NVIDIA NVFP4 release build (21.6GB), served via vLLM 0.25.1 on the native cutlass sm120 FP4 path, measured 2026-08-11 same-day with the Q4KM row below it. First board pair where the vendor's own FP4 recipe is compared against a community Q4 GGUF of the same model on identical harness: quality is a wash (+0.4 qavg, all per-task deltas ≤1.4), and NVFP4 decodes in the same speed class or slightly ahead (410.5 tok/s chat-server median vs 377.4 tg128 — different conventions, see the report for both legs and the sm_120 reproduction traps).

⁹ Six-rung GGUF quant-tax ladder (bartowski imatrix repo), measured 2026-08-12/13, llama.cpp b10338 worktree, all rungs fully VRAM-resident. The tax is nearly zero above the floor: Q5KM to IQ2M spans 0.4 qavg while decode climbs 349 → 399 tok/s and VRAM drops 25.2 → 17.7 GiB. Only IQ2XXS pays (−1.7 qavg vs IQ2M), and the entire drop is HellaSwag. IQ2M strictly dominates Q3KM and IQ4XS (smaller, higher qavg, faster). Ladder GGUFs embed the MTP drafter head (inert here). Ladder report.

¹⁰ Seven-rung GGUF quant-tax ladder (unsloth day-0 repo, UD = dynamic imatrix recipe), measured 2026-08-14/15, llama.cpp b9653, all rungs fully VRAM-resident. No cliff anywhere: Q80 to UD-IQ2XXS spans 2.9 qavg over a 3.4x file-size range, Q6K ties Q80 to the second decimal (93.66 both), and UD-IQ3XXS holds 92.7 from an 11.1GB file (12.8 GiB peak — 16GB-card territory) at 96 tok/s. MMLU pays most of the tax; the IQ2 rungs pinch HumanEval first. The tightness is short-context only: on long-context retrieve-and-use (15 tasks per depth at 16k/32k/64k, measured 2026-08-16) Q6K is perfect at every depth while UD-IQ3XXS holds 73/67/80 — the 1.0 q_avg gap becomes ~27 points under depth load. And it is a benchmark-ceiling artifact too: on GPQA-diamond (standing second-tier metric since 2026-08-17) the same seven rungs spread 11.1 points where the board spreads 2.9 — the 4-bit band ties the top at 49-50.5 while the IQ2 rungs fall to 39-43. Ladder report + both addenda.

¹¹ NVFP4 checkpoint (unsloth day-0 cut, 22.6GB) served by vLLM 0.25.1 on sm120, measured 2026-08-17 — the only non-llama.cpp Qwen3.8 row; quality same-harness over HTTP, cross-stack for speed. Q6K-sized but lands below Q4KM, HumanEval pays nearly the whole tax (89.6 vs 94.5). The lane's draw is the shipped MTP speculative head, which llama.cpp cannot run yet: 1.72–1.81x decode at every prompt depth tested (69 → 114–125 tok/s effective, held at 32k). NVFP4 report.

¹² Full-precision BF16 reference (split GGUF, 54.7GB — 1.7x the card), quality-only via llama.cpp partial offload (-ngl 40), banked 2026-08-17→20 across three overnight windows. Lands mid-ladder at 93.5 (ties UD-Q4KXL, 0.1 under the Q6K/Q80 pair) with GPQA-diamond 48.5 inside the 4-bit band's noise band — the ladder's 4-bit-and-up rungs were already at the full-precision ceiling, and the IQ2 floor's loss is now measured against a true reference. No speed rows by design (partial-offload decode is not comparable to resident rungs). BF16 addendum.

¹³ Ornith 1.5 35B-A3B (MoE, ~3B active), a reasoning model run think-off for board parity, measured 2026-08-21. Q4KM is the vendor's first-party GGUF (20.2GB, llama.cpp b9653, 21.4 GiB resident, 303 tok/s tg128); NVFP4 (23.4GB) served via vLLM 0.25.1 on sm120 at ~300 tok/s chat-server decode — the two stacks land within 0.1 qavg. The board understates the model twice over: think-off parity, and GPQA-diamond 52.0 think-off / 81.8 think-on (16k budget, 2026-08-22) — level with the dense Qwen3.8-27B rungs in both regimes (50.5 / 79.3-80.8) at 3.8x their decode speed. Shipped MTP speculative head measures negative on vLLM 0.25.1 (245-254 vs ~300 base). Report.

¹⁴ Ornith 1.5 9B (dense hybrid-attention), vendor first-party Q6_K (6.85 GiB, llama.cpp b9653, 188 tok/s tg128), think-off for board parity, measured 2026-08-22→24. GPQA-diamond 41.9 think-off / 74.2 think-on — but the think-on number carries a measured failure mode: 25 of 198 chains (12.6%) never terminate under greedy decoding, at 16k AND 32k budgets, and the vendor's own BF16 GGUF still fails to terminate 22 of the 25 at 32k — the wall is the model under this regime, not the quant (the 35B sibling truncates 14 under the identical recipe). On chains that finish, accuracy is 85%. The 35B-A3B sibling dominates the 9B on every axis on this card, including decode speed (303 vs 188 tok/s). Report.

¹⁵ Qwopus3.8-27B-Flash (Jackrong's fine-tune of Qwen3.8-27B, Q6K with embedded MTP head, 22.4GB), measured 2026-09-07 on llama.cpp b10371 against the banked Q6K base, same harness, same day. Four of five tasks sit within the base's error band; the q_avg is HumanEval alone: 26.8 vs 94.5, because the tune emits exactly one leading space on every code line in 150 of 164 answers, so nesting depth is destroyed in the output and no whitespace repair recovers it (48/164 at best). The model card discloses the indentation issue; this is its size. Think-on GPQA-diamond (16k budget, paired with a same-day base re-run that reproduced the 2026-08-19 result to the item): 70.7 vs 79.3, with the tune spending 4,229 completion tokens per correct answer against 7,740 (2.08x on the 125 items both solve), so the card's efficiency claim is real at about 2x and costs 8.6 points. The card's MTP acceptance advantage does not reproduce think-off (weighted 0.762 vs 0.768, 109 to 145 tok/s for both). Report.

¹⁶ Qwen3.8-Flash-Next (125B total / 10B active MoE, 512 experts, qwen4exp hybrid linear attention), unsloth UD-Q2KXL 78.9GB, does not fit the card: run with the routed experts of the first N layers in system RAM (--n-cpu-moe N, 59 GB RAM), llama.cpp b10853 (fresh master worktree; the arch merged 2026-08-27), measured 2026-09-08/09. Quality legs at N=28 (23.0 GiB): highest MMLU on the board (88.5) and GPQA-diamond 54.0 think-off from a 2-bit cut; GSM8K and HumanEval pay the quant tax. Speed at the ladder optimum N=22 (28.5 GiB): decode 59.7 tok/s tg128, 51.0 at 32k depth, prefill 900 tok/s pp512. The full --n-cpu-moe ladder (44 to 18, nine points) climbs 1 to 2.5 tok/s per 1.8 GiB of experts moved onto the GPU, peaks at N=22, inverts at N=20 (30.3 GiB, 50.2 tok/s) and fails to load at N=18: the optimum leaves ~3.5 GiB free, as on the Ling-3 sweep. Per-card points: 16 GB at N=40 (42 tok/s), 24 GB at N=28 (52.5). On this 59 GB box N=28 sits on the page-cache edge (736k major faults during one depth pass, prefill at depth varying 2.7x between identical runs); N=22 does not. MTP head on the open llama.cpp PR #28243 (at N=28): 1.25x on code and repetitive text, a small loss on prose and chat, because under offload a rejected draft costs most of a plain token. Speed rows are offload-conditioned and not comparable to the resident rows. Report.

Thinking ON (reasoning · extended chain-of-thought)

Model	Params	Quant	MMLU	ARC-C	HellaSwag	GSM8K	HumanEval	q_avg
Qwen3.6-35B-A3B	34.66B	UD-Q6_K	94.7	97.0	87.0	92.0	98.0	93.8
gpt-oss-120B¹	116.83B	MXFP4	89.5	95.0	80.0	97.0	98.0	91.9
Muse-Glimmer-30B²	29.85B	UD-Q4KXL	80.6	93.5	88.8	96.7	95.7	91.1
Qwen3.6-28B-REAP-A3B	28.24B	Q6_K	87.7	95.0	82.0	90.0	94.0	89.7

HumanEval correction (2026-06-04). An earlier harness passed API stop sequences (\ndef, \nclass) that fired mid-reasoning, truncating inline-reasoning models before they emitted code — producing false-low scores (Qwen3-Coder-Next read 10%, not 93%). Every model has since been re-run on the fixed, reasoning-aware harness (no stop sequences, max_tokens=4096, indentation-preserving response handling). A second extraction fix (2026-06-04) makes program assembly format-agnostic — it generates candidate assemblies and keeps whichever one compiles — after Nemotron-3-Nano exposed a case where the model indents only the first body line differently (raw HumanEval read 21%; corrected to 80.5%). Do not cite any HumanEval figure published before this date. Why two tables. Thinking-off rows answer directly; thinking-on rows emit an extended reasoning chain first. The two modes are not comparable on the same axis — including on MCQ/GSM8K — so they are ranked separately. Within a family, turning thinking on trades raw knowledge recall for reasoning depth (compare Qwen3.6-35B-A3B in both tables: MMLU 85.0 → 94.7). ¹ gpt-oss-120B runs via MoE CPU-offload (--n-cpu-moe 20) — it does not fit 32GB VRAM (59GB model); ~30GB VRAM + the rest in system RAM, ~47 tok/s generation. It and the other two thinking-on rows were run on a ~100-item-per-task subset (MMLU 2/subject). ² Muse-Glimmer-30B (Meta, VLM: 28B text decoder + 2B vision encoder; text benches exercise the decoder only) sits in this table because its chat template's reasoning channel has no off switch — reasoning_strength (default high) modulates depth but every off-looking value (low/minimal/none/off) still produces ~60 reasoning tokens before an MC answer letter. Scored with the standard 50% sampling at reasoning_strength: low; runs fully resident (15.4GB VRAM peak, 85.7 tok/s tg128) on llama.cpp master b10349, the Muse Glimmer merge commit (PR #26841, merged day-0). Full report in `reports/muse-glimmer-30b-ud-q4-k-xl.md`.

Sampling. MMLU & HellaSwag use 50% stratified sampling (seed=42); ARC-Challenge, GSM8K, and HumanEval run the full item counts (HumanEval = all 164). Full per-model reports in `reports/`.

Second tier, thinking on (IFEval · MATH-500 · HumanEval+ · MBPP+)

The five-task boards above are short-answer tasks. This table is the harder second tier, run with thinking on at a fixed budget: max_tokens 16384, ctx 24576, greedy, zero-shot, one pass per item, llama.cpp b9653, 1,583 items per model. Scores are percentages; IFEval is prompt-strict. cap is the share of items whose completion hit the 16k budget: a cell at 10% or more (▲) is a floor, not a ceiling, and is never flat-compared with a low-cap cell. ◆ marks a format miss (answers without \boxed{}). Full per-task counts, capped items, parse failures, tokens per correct answer and median completion length: `second_tier.csv`. Report with the reads and limits: thinking on at a 16k budget · chart.

Model	Quant	GGUF	IFEval	MATH-500	HumanEval+	MBPP+	Mean	MATH-500 cap
Gemma 4 31B-it	QAT Q4_0	17.7 GB	91.1	94.0	94.5	81.0	90.1	2.2%
Qwen3.8-27B	Q6_K	22.9 GB	89.7	95.2	92.1	81.8	89.7	1.6%
Qwen3.8-27B	UD-IQ3_XXS	11.9 GB	90.2	94.6	92.7	80.7	89.5	1.2%
Nemotron-3.5-Lightning 30B-A3B	Q4KM	24.5 GB	87.8	90.2	81.1	80.4	84.9	6.2%
Ornith 1.5 35B-A3B	Q4KM	21.7 GB	77.3	88.2	91.5	81.5	84.6	2.2%
Qwen3.6-35B-A3B	UD-Q5KM	26.5 GB	87.2	72.8 ▲	95.1	81.2	84.1	30.2%
Qwen3.6-27B	Q6_K	22.9 GB	88.5	73.2 ▲	94.5	79.4	83.9	27.0%
Qwable-27B	Q4KM	16.5 GB	87.6	71.6 ▲	93.9	72.2 ▲	81.3	28.8%
Ornith 1.5 9B	Q6_K	7.4 GB	69.5	84.6	89.0	77.0	80.0	5.8%
Qwopus3.8-27B-Flash	Q6_K	22.4 GB	83.6	33.6 ◆	74.4	80.7	68.1	3.8%

The top three are a tie: the Wilson 95% half-width on a four-task mean here is 1.6 points. The three ▲ MATH-500 cells spent 12.4k to 12.9k completion tokens per correct answer against 1.8k to 1.9k for the two Qwen3.8 rows; a 32k pass on exactly those three legs is queued and will publish as a separate table. Qwable-27B MBPP+ capped 12.2%. The Qwopus3.8-27B-Flash MATH-500 cell has 325 of 500 answers unboxed at 3.8% capped, a formatting result in the same family as its main-board HumanEval (footnote 15).

Methodology

Benchmark	Dataset	Few-shot	Scoring	Items
MMLU	`cais/mmlu`	5-shot	Letter extraction (A/B/C/D)	14,042
ARC-Challenge	`allenai/ai2_arc`	25-shot	Letter extraction	1,172
HellaSwag	`Rowan/hellaswag`	10-shot	Letter extraction	10,042
GSM8K	`openai/gsm8k`	5-shot CoT	Exact numeric match	1,319
HumanEval	`openai/openai_humaneval`	0-shot	pass@1 (code execution)	164

All benchmarks run at temperature=0. MCQ and GSM8K use max_tokens=2048; HumanEval uses max_tokens=4096 with no stop sequences (reasoning models emit code only after long inline reasoning — premature stops were the bug corrected above). Multiple-choice tasks use generative letter extraction instead of loglikelihood scoring — scores are internally consistent for model comparison but may differ from logprob-based evaluations by 5-15%.

Full per-model reports with MMLU category breakdowns, parse reliability stats, and speed data: `reports/`

Error bars

Every score above is a single greedy pass, so each is a binomial proportion with a sampling interval. `board_ci.csv` publishes a Wilson 95% interval per task per row (from the correct/total counts in each run's detail files) together with the parse-failure count, and a q_avg row whose half-width propagates the five task variances through the mean, treating tasks as independent. Measured across the 47 rows with all five tasks (regenerated 2026-09-10), the median qavg half-width is 0.97 points (the four think-on rows scored on small subsets run 2 to 4 points), and humaneval dominates it in 44 of 47 rows with a median 82% share of the variance, because 164 problems at ~93% pass gives a task half-width of about 4.5 points against under 1 point for the other four. Since 2026-09-10 the file also carries one `gpqathinkon row per think-on GPQA leg (seven rows) with three extra columns, tokenspercorrect, completiontokensmedian, cappedrate`, filled where the leg recorded per-item tokens (legs run since 2026-09-09; the August legs did not record them). The per-budget legs of the [reasoning-budget curve](reports/reasoning-budget-curve.md) live under `results/<slug>-thinkon/budget-<N>/` and are tabulated in that report rather than in this file. Two rows whose qavg differ by less than that half-width are a tie; the tables stay sorted by qavg for continuity, so read neighbouring rows as a band of roughly one point rather than a ranking. Regenerate with `python3 scripts/boardci.py results/`.

Speed Benchmarks

What's measured

—Prompt processing (pp): parallel batched token throughput at context lengths 128, 512, 2048, 4096, 8192, 16384
—Text generation (tg): sequential autoregressive token throughput at 128 tokens
—All models fully GPU-offloaded (ngl=99)

Served (HTTP) lane

The llama-bench numbers above measure the engine on its own. The served lane (scripts/speed_served.py, lib/speed_served.py) measures what a client sees through llama-server's streaming chat-completions endpoint, the path the spec-decode and prompt-cache reports were timed on. Four fixed workloads (prose, code, repetitive, chat; 8 prompts each, lib/workloads.py, the same prompt set those reports used) are sent at temperature 0 with max_tokens=256 after one discarded warm-up request; every request is appended as one json line with the server flag set (mode), cache_prompt, the wall time to the first content chunk, the total wall time, prompt and completion token counts, the server's own timings object when present, and a sha256 of the text so two flag sets can be checked for byte-identical output. Vocabulary: TTFT is the time to the first content token as the reader perceives it (prefill plus queueing); perceived TPS is completion_tokens / (total_s - ttft_s), the rate text arrives at once it starts; total TPS is completion_tokens / total_s, throughput including the wait. Percentiles (p50 and p90 over the 32 requests per pass), not means, are reported, alongside the server's predicted_per_second p50 and, when speculation is on, the acceptance rate sum(draft_n_accepted) / sum(draft_n). A served TPS is not comparable to a tg128 figure from llama-bench: different prompt, different token budget, and the HTTP and tokeniser overhead are inside the served number.

Speed at depth (agent-shaped)

tg128 on an empty context is the number every board quotes and the one an agent loop never sees. Coding harnesses hand the model a 10k to 20k token prompt and then decode against it, so two numbers matter more: how fast the first token arrives on a long prompt, and how much decode speed survives at depth. Both come from the same llama-bench depth sweep (-p 512 -n 128 -d 0,8192,32768, -r 2) that has been a standing leg on every treatment since 2026-08-14 and was backfilled on the resident rows on 2026-09-08/09. `speed_at_depth.csv` carries the raw columns; the table is regenerated with python scripts/depth_table.py table results/.

Model	Quant	tg128	tg128 @32k	held	pp512 @8k	TTFT, 16k prompt (est.)
Nemotron 3.5 Lightning 30B-A3B	Q4KM	377.4	353.1	94%	10,950	~1.5 s
Ornith 1.5 35B-A3B	Q4KM	303.2	263.5	87%	8,336	~2.0 s
Qwen3.8-27B	UD-IQ3_XXS	96.2	86.5	90%	3,720	~4.4 s
Qwen3.6-27B	Q4KM	77.1	71.6	93%	3,640	~4.5 s
Gemma 4 31B-it	Q4_0	76.4	65.4	86%	3,673	~4.5 s
Qwen3.8-27B	Q6_K	63.1	58.3	92%	3,119	~5.3 s
Qwen3.6-27B	Q6_K	61.9	58.2	94%	3,026	~5.4 s
Qwopus3.8-27B-Flash	Q6_K	63.0	58.0	92%	3,025	~5.4 s
Qwen3.8-Flash-Next 125B-A10B (`--n-cpu-moe 22`)	UD-Q2KXL	59.7	51.0	86%	826	29.4 s measured (derived 19.8 s; the derivation is a floor under offload)

How to read it: held is tg128 @32k / tg128, the share of decode speed that survives 32k tokens of context. TTFT for a 16k prompt is an estimate, 16384 / pp512 @8k: prefill throughput measured at 8,192 tokens of depth is the closest measured rate for a mid-size prompt, and the true number is a little worse because prefill slows as the prompt grows (compare pp512 @8k with @32k in the csv). When a treatment measured TTFT on the served lane with a real 16k system prompt, that number replaces the estimate and is shown without the tilde. Rows are llama-bench on fully resident models unless a --n-cpu-moe flag is shown; the offload row's prefill is RAM-bandwidth bound and its estimate is cache-state dependent (see the Flash-Next report). Rows without a depth sweep on disk are absent, not zero; the backfill continues through the idle queue.

What your card can run

`card_fit.md` turns the board into a fit table for 8, 12, 16, 24 and 32 GB cards. Each row's VRAM @16k is the peak sampled on the 5090 during the speed sweep (whose largest prompt is 16,384 tokens, so weights + a 16k KV cache + compute buffers), applied as a budget with 768 MiB of headroom: ✅ resident with a 16k context on the board recipe, 🟡 the file fits but the 16k peak does not (shorter context, q80 KV cache, or a smaller quant), ⬜ the file is larger than the card, 🟠 measured with experts in system RAM (the MoE offload rows, which need the RAM as well). Quality (qavg, GPQA-diamond) sits next to each row so the question "what is the best thing a 16 GB card runs" reads off directly. These are 5090 measurements applied to other cards' capacities, not measurements on those cards: peak bytes are the same on any CUDA card at the same context and flags, decode speed is not. Regenerate with python scripts/card_fit.py results/.

Recipes: which llama-server line for this model

`reports/recipes/` answers the question the board does not: given this model on this card, which flags. One page per model, one recommended line at the top, and a six-set table behind it (base, --flash-attn on, KV q8_0, KV q4_0, --parallel 1, MTP draft head), each set served and measured on the four short workloads and on a 16k-token system prompt cold and warm, with VRAM peak and a 40-item GPQA spot check on the KV-quantised sets. Seven pages in (Qwen3.8-27B Q6K, Gemma 4 31B Q40, Qwen3.6-35B-A3B UD-Q5KM, Nemotron 3.5 Lightning Q4KM, Ornith 1.5 35B Q4KM, Qwopus3.8-27B-Flash Q6K, Flash-Next UD-Q2K_XL under offload): flash-attn changes nothing, --parallel 1 is free, q4 KV costs 13 to 29% of decode at 16k depth on all seven, q8 KV is a memory lever on dense models only, and an MTP head pays 1.5 to 2.3x on a 60 to 76 tok/s base but is 0.6 to 0.8x on a 364 tok/s one.

Speed data schema

Column	Description
`model`	Model name
`architecture`	Dense or MoE (with active param count)
`params_b`	Total parameters in billions
`quant`	Quantization method
`size_gib`	File size in GiB
`engine`	Inference engine (llama.cpp or vLLM)
`backend`	Compute backend (CUDA)
`gpu`	GPU model
`vram_gb`	VRAM in GB
`test`	Benchmark test (pp128, pp512, ..., tg128)
`tokens_per_sec`	Throughput in tokens/second
`stddev`	Standard deviation
`date`	Benchmark date

Key findings

MoE (3B active) vs Dense (27B) on same-family Qwen3.6 models:

—Prompt processing: 2.4x faster across all context lengths
—Text generation: 3.5x faster (271 vs 77 t/s)
—Both degrade ~17% at 16K context (attention + VRAM, not parameter count)

Field Reports

One-shot investigations that don't fit the leaderboard format — claim verification, new-architecture probes, and consumer-hardware autopsies, all measured on the same rig. Newest first.

Report	Finding
Speculative decoding and prefix caching on one RTX 5090, Qwen3.8-27B Q6_K: the two free speedups on the model this box now serves · chart	Rerun of the spec-cache study on Qwen3.8-27B Q6K (the served model since 2026-09-13, own MTP head in the unsloth GGUF), same 304-request design, same flag set (llama.cpp b9653, ctx 64k, parallel 1, KV q80, batch 1). MTP n=2 lifts decode from 61 to 109-144 tok/s (1.8x prose and chat, 2.3x code and repetitive) for +9 ms TTFT; n=4 reaches 168-184 on code/repetitive and loses 5-8% on prose/chat; n-gram lookup is 1.00x. Prefix cache: TTFT 1.004 s to 0.069 s on a 2,954-token system prompt (15x), 3.814 s to 0.086 s on 11,570 tokens (44x), decode unchanged. Stacked, a short answer to the 11.5k context goes 4.25 s to 0.30 s. Losslessness: MTP byte-identical to plain greedy decode on 59% of answers; all 13 divergences are the target's own second choice at a near-tie (top-2 gap 0.004 to 0.133 nats, median 0.030). Every cell within noise of the Qwen3.6-27B run a week earlier: the speedups belong to the shape and the serving stack, not the generation.
How much thinking is worth paying for: GPQA-diamond think-on at three completion budgets · chart	Two think-on models, the same 198 items, only `max_tokens` moved (4,096 / 8,192 / 16,384). Qwen3.8-27B Q6K 60.1 / 70.2 / 79.3, Ornith 1.5 35B-A3B Q4K_M 70.2 / 77.3 / 81.8. Every point lost at a smaller budget is a capped item (101 of 101 across the four smaller legs, per-item diff against 16k) and finished items are 94.7 to 96.1% correct at every budget, so the budget removes the hard questions rather than degrading the answers. 43% / 38% of items run past 4k, 34% / 30% past 8k, 22% of both still past 16k. Tokens per correct answer: Qwen 3,972 → 5,636 → 7,740, Ornith 3,248 → 4,732 → 6,996 (Ornith 700 to 900 cheaper at every budget, and 4.6x faster to decode). The 16k legs reproduced the August 16k runs on all 198 items. Limits: two models, one quant each, greedy, curve stops at 16k.
Speculative decoding and prefix caching on one RTX 5090: what the two free speedups buy · chart	The two single-GPU optimisations every inference text names first, measured on the model and flag set this box serves daily (Qwen3.6-27B Q6_K with its embedded MTP drafter, llama.cpp b9653, batch 1, 304 requests, one variable at a time). MTP draft n=2 lifts decode from 61 to 112-144 tok/s (1.8x prose, 1.9x chat, 2.3x code and repetitive) for +10 ms TTFT; n=4 reaches 169-183 on code/repetitive and loses to n=2 on prose/chat, because draft depth only pays above ~0.85 acceptance; n-gram lookup is 1.00x on 256-token answers. Prefix caching cuts TTFT on a re-used 2,954-token system prompt from 1.005 s to 0.067 s (15x) and on 11,570 tokens from 3.835 s to 0.084 s (46x), decode untouched; stacked, a short answer to the long context goes 4.18 s to 0.24 s. Losslessness: MTP output is byte-identical to plain greedy decode on 56% of answers; all 14 divergences examined with top-2 logprobs are flips to the target's own second choice at a near-tie (gap under 0.09 nats), so same quality class, not byte-identical. Limits: one model, one quant, batch 1, 256-token answers.
The benchmark said below-chance. The model was fine: a truncation autopsy · chart	A community benchmark harness (SM12X-LLM-BENCH, full profile, 18h, rc=0, zero infra errors) reported Qwen3.6-27B Q6K at 21.7% on GPQA-diamond — below the 25% random-guessing floor for 4-choice questions. The model was fine: the harness serves thinking models at their default (thinking on) with an 8,192-token completion cap, and on GPQA the model's reasoning chain was still running when the cap hit (ttft p50 184s), so 153/198 items (77%) ended `finishreason=length` with no extractable answer, each scored as wrong. Anchor leg on the same GGUF file, same resident llama-server, same RTX 5090, thinking off so answers fit the budget: 54.5% (108/198, 0 parse failures) — 2.5x the score from one setting. Reads: below-chance accuracy on a competent model is a pipeline bug until proven otherwise; the truncation count was in the harness's own report all along, but the lane still presented as PASS/publishable, so score fields get read and diagnostic fields get skipped (guard proposed upstream as SM12X-LLM-BENCH#3, with #1 long-context detection and #2 cache-inflated prefill from the same run); thinking-mode benchmarks need deliberation-sized budgets, not answer-sized ones. Honest limits: anchor is think-off (lower bound proving artifact, not the model's think-on ceiling); one model, one quant, one lane; the harness's client plumbing was impeccable — the finding is about score presentation.
Kitty's 2-bit KV cache holds long-context retrieval, the axis its paper never measured · chart	Kitty (arXiv 2511.18643) reports near-zero loss at 2-bit KV cache, but measures it only on short-input reasoning tasks (GSM8K, MATH, HumanEval, GPQA, MMLU, AIME), never on long-input retrieval, the axis KV compression exists for. Needle-in-a-haystack, Qwen3-8B, one RTX 5090, f16 KV vs Kitty-Pro 2-bit KV on identical prompts. Calibrate f16 first: it retrieves a single needle at 100% across 8k/16k/32k, every depth. 64k does not fit (a one-shot 64k prefill plus f16 KV OOMs the 32GB card, and Kitty's reference path stores the cache dequantized so it cannot rescue the memory), so the ladder caps at 32k, still 16x Kitty's longest tested input. Single needle: 2-bit ties f16 at 100% in all fifteen cells, including the needle at the very start of a 32k context, but a perfect tie is weak evidence (it cannot tell lossless from too-easy). Harder probe: 8 needles at 8 depths, keyed retrieval, 160 recoveries per leg. Now the task discriminates, f16 itself slips to 97.5% at 32k and 90% at the shallowest depth. Through that, 2-bit KV tracks f16 within noise: 99.4 vs 98.8 overall, about one needle, no depth band degrading. Kitty's near-zero loss extends to long-context multi-fact retrieval up to 32k. Honest limits: 64k is a hardware wall not a pass; f16 is still near ceiling at 98.8%, so a harsher task might separate them; this is the reference fake-quant simulator, quality not a deployed kernel's speed or memory; one model, one retrieval task.
Reasoning is load-bearing for LFM2.5-2.6B on agentic tool-use, and a broken toggle nearly hid it · chart	LFM2.5-2.6B, a small reasoning model, tested on a native tool-calling battery (30 tasks, 3 reps, greedy, one RTX 5090) with reasoning on vs off plus three larger anchors. The first sweep reported a clean null: 96.7% both legs, McNemar p=1.0, not one of 30 tasks moving. It was a measurement artifact. LFM2.5's shipped GGUF chat template hard-forces `<think>` on every assistant turn and ignores enablethinking/reasoning/reasoningeffort, so the off leg still logged ~700 reasoning tokens per task and both legs reasoned. Serving a patched template (--chat-template-file) that pre-closes `<think></think>` when thinking is off dropped the off leg to 0 reasoning tokens across 90 runs and reproduced the on leg exactly, and the result flipped: reasoning on 96.7% to off 70.0%, a 26.7pt fall, McNemar p=0.0078 (8 of 30 tasks flipped, 0 the other way). The drop is worst on the easy tier (single-tool 100 to 60): with no think block the model acts without a plan, making more tool calls (4.16 vs 2.7) and more bad ones (1.48 vs 0.97) at the same output length, so the failures are misfires not truncation. Reasoning roughly doubles the token bill (1403 vs 685 per task) and is worth it on this battery. Honest limits: LFM-on tops all three anchors (Qwen3.5-9B 86.7, Gemma-4-E4B 83.3, Qwen3.5-4B 80.0) but that comparison is underpowered at n=30 (p≥0.0625), so treat it as directional; the significant result is the on/off effect. The probe that gated the run checked only that the on leg reasoned, never that the off leg stopped, a one-sided check that passes a broken A/B, now fixed to check both directions.
NVFP4 on a 5090: real FP4 lands on the Qwen3.6-27B frontier where K-quants do · chart · frontier	NVIDIA's NVFP4 is Blackwell's native 4-bit float; benchmarked as a GGUF rung on the exact Qwen3.6-27B K-quant frontier the quant-tax ladder measured, same harness (MMLU/GSM8K/HumanEval, greedy, think-off), one RTX 5090. It lands on the frontier: composite 92.11 (between Q5 and Q6), with MMLU (86.67) and GSM8K (97.60) identical to Q80 and Q6K, and HumanEval one question under Q8. Clean measurement (zero empty outputs, no grader fallback). Two mechanisms. First, its decode sits on the memory-bound line: 78.6 tok/s from 18.1 GiB served, exactly where the K-quant curve predicts for that footprint (Q5 71.8 at 19.2 GiB, Q4 80.4 at 16.8), so Blackwell's FP4 dequant adds nothing measurable. That is the direct contrast with BTL-3-Compact's AVQ2, which is compute-bound and decodes slower than Q8. Second, it confirms NVFP4 is near-lossless: the NVFP4-vs-Q6 gap is 0.21pt on the 3-suite composite (June's t034 put it near a point on the 5-suite q_avg; the suite and build differ, both say near-lossless rather than cliff). The framing contrast: same rig and base, NVFP4 on the frontier, BTL-3's sub-2.5-bit 13pt below it. Honest limits: the 20.09 GB file carries an inert BF16 MTP draft head, so the single-stream served footprint is 18.1 GiB; the NVFP4-Q8 variant and the vLLM NVFP4-vs-AWQ throughput question were not run; 3-suite composite, one model.
BTL-3-Compact on a 5090: sub-2.5-bit runs, 13 points under plain Qwen3.6-27B · chart	Bad Theory Labs' BTL-3-Compact is a sub-2.5-bit 27B (AVQ2, 8.4 GB) pitched as near-lossless and agentic, built on the exact Qwen3.6-27B this rig already quant-taxed. No prebuilt runtime ships for the card, so the AVQ2 llama.cpp fork was compiled from source for sm120a: it runs, the first independent RTX 5090 validation of the format, with 27B resident in 9.5 GiB (a 27B fits that footprint only if the weights are compressed to it). On the same t090 harness (MMLU/GSM8K/HumanEval, greedy, think-off) it scores a 77.3 composite, 13 points below the K-quant ladder's lowest rung (Q3KM 90.5) at fewer bits, off the honest frontier its own base defines: per-suite 75.4/84.0/72.6 vs the ladder's 83.8/96.4/91.5 at Q3, measurement clean (zero empty outputs, no grader fallback). AVQ2 also breaks the K-quant rule that smaller is faster, decoding at 47 tok/s against Q80's 53 despite a third the bytes, because the mixed-precision unpack is compute-bound. The vendor's marquee numbers (95.12 HumanEval, 88.5 BFCL) are thinking-mode results, and the card discourages thinking as able to "become repetitive or fail to terminate"; a probe reproduced exactly that (most items exhaust a 2-3K token budget inside the reasoning block and never answer), with no published sampling, so the headline lives in a mode the vendor itself flags as unreliable and does not reproduce under greedy on a neutral harness. Honest limits: BTL-3-Compact is a finetune of the base, not the same weights re-quantized, so the gap is model-plus-method rather than the AVQ2 tax; the 92.2% retention claim (Compact vs full-precision BTL-3) and the agentic BFCL axis were not run.
The quant tax on Qwen3.6-27B: near-zero to Q4, one real step at Q3 · chart	The GGUF K-quant ladder the NVFP4 report kept deferring to: Q80 down to Q3KM, same subject, one pin, MMLU/GSM8K/HumanEval scored on the same harness every rung. Composite quality moves 92.3 to 92.0 from Q8 to Q4 (−0.4 pt, inside the noise) while single-stream decode climbs 52.8 to 80.4 tok/s (+52%) and the file shrinks 29.0 to 16.8 GB; Q3KM is the first rung past the noise floor, another +12% decode for −1.4 pt. Two operator reads: Q6K strictly dominates Q8 (identical scores on all three suites, 6 GB smaller, 21% faster), and Q4KM is the sweet spot (80 tok/s, half a 5090's VRAM, within 0.4 pt of Q8). Only MMLU discriminates the top of the ladder (a gentle 86.7 to 84.6 slope Q8 to Q4); GSM8K and HumanEval sit flat until the Q3 step; no imatrix, so this is the low-rung floor. The perplexity column was cut and diagnosed, not hidden: the ladder's 34-chunk `-c 4096` sample gave a non-physical ordering (Q3 below Q8) that the standard `-c 512` stride (274 chunks) collapses to a Q8 = Q3 tie at ~7.05, so it was sampling noise, and at honest sampling wikitext2 barely separates these quants. Composite is a 3-suite mean, not a leaderboard q_avg.
First EAGLE-3 draft head for Hermes-4.3-36B: trained on one 5090, converged at 39% of the epoch, 1.29–1.57x · chart	Hermes-4.3-36B (Seed-OSS architecture) had no speculative-decoding draft head anywhere on Hugging Face; this run trained the first one entirely on a single RTX 5090, using SpecForge's online mode with a 4-bit AWQ teacher (six memory patches, five filed upstream), 54K curated conversations, 54,000 steps across 8 unattended nights (~55 GPU-h, every night rc=0). Release bench (sglang 0.5.14, cuda graphs, 8 fixed prompts/workload, 66 tok/s no-drafter baseline): tree-3-4-8 wins all four workloads at 1.29–1.57x real decode speedup, while the wider tree-5-8-16 accepts deeper (accept-len to 2.22) yet lands slower everywhere — the draft cost eats more than the extra accepted tokens return. Two findings beyond the table: the head converged at step 21,000 (the epoch-end bench reproduces the 39%-of-epoch dry run within noise, so the last five nights of training moved nothing), and the 1.7–2.2x trained-head precedent band is out of reach at batch 1 on one card, where roughly a quarter of the theoretical accept-len gain pays for the draft pass itself. Decision recorded in the report: no epoch 2 on the same data; a v2 changes the data mix, not the step count. Weights + model card follow with the release.
LTX-2.3 audio-video on a 5090: synced sound, decode-bound wall time · chart · clips	First consumer-Blackwell (sm_120) numbers for LTX-2.3, Lightricks' ~19B dual-stream DiT (14B video + 5B audio) that generates video and synchronized audio in one pass. The distilled two-stage pipeline (8+4 steps, fp8-cast, tiled VAE, SDPA) makes a 97-frame ~4s clip with h264 + AAC 48kHz output in ~40s at 768x512 (steady state) and ~50s at 1280x704 on one RTX 5090: 10.8 and 12.3 seconds of compute per second of output video. The surprise is the shape of the cost: 2.5x the pixels adds only ~14% wall time, and true peak VRAM (torch counter) is flat at 24.2GB in both configs, because the DiT accounts for just ~5-12s of the wall while tiled VAE decode + audio decode + mp4/AAC encode dominate. Consequences: render at the higher resolution (the quality jump is nearly free) and expect DiT-side speedups to move at most ~a quarter of wall time at these clip lengths. Honest limits flagged in the report: the first generation at a new shape pays ~+20s of one-time lazy init (kept in the published mean), and the vendor's "5.7x faster than Wan2.2 on a 5090" claim is cited, not re-measured. All ten clips attached. Completes the artifact-first modality arc: music (ACE-Step), image (Z-Image), now video-with-audio, all on one card.
Z-Image-Turbo on a 5090: few-step distillation, measured · chart · samples	First RTX-5090/sm120 numbers for Z-Image-Turbo, an open-weights (Apache-2.0) 6B text-to-image DiT with a Qwen3-4B text encoder, measured out-of-box (bf16, SDPA, no torch.compile, no quant). A 1024px image takes 3.18s (~19/min), a 512px one 0.83s. The "turbo" is distillation from the ~50 denoising steps a normal diffusion model runs down to 8, and compute is exactly linear in the step count (each step is one DiT forward): 1.69s at 4 steps, 3.20s at 8, 6.23s at 16, with 4 steps near-indistinguishable from 8 on portraits. Resolution is the real cost, not memory: 0.83/3.18/9.99/23.36s at 512/1024/1536/2048, super-linear because attention scales with pixel count squared. The whole model fits a 32GB card at every tested size (true per-image peak 20.4/21.7/24.6/28.8GB), so 2048px keeps ~4GB of headroom: a time wall, never a VRAM wall. The classically-hard cases hold at 8 steps, where it renders exact text on a sign ("WITCHEER", letters correct) and draws exactly-N objects on request, with colours and spatial relations landing too (sample grid attached). Measurement note: per-image VRAM comes from torch.cuda.maxmemoryallocated() with resetpeakmemorystats() each iteration, because nvidia-smi over-reports. PyTorch's caching allocator retains its high-water mark and never releases it between generations, flattening a naive VRAM curve to a wrong constant. Speed and footprint only; a GenEval quality pass (its mmcv/mmdet detector needs a from-source sm_120 build) is the natural follow-up. Companion to the ACE-Step music bench, the other few-step distilled generator on this rig.
Nemotron-TwoTower autopsy: a 2.4x speedup that costs a second 30B model · chart	NVIDIA's Nemotron-TwoTower is a diffusion LM adapted from a frozen autoregressive Nemotron-3-Nano-30B that generates 2.42x faster than plain AR at 98.7% quality (arXiv 2606.26493) — clever, and datacenter-only by construction. The speedup comes from running two full 30B backbones co-resident: a frozen context tower plus a trained denoiser that unmasks several tokens per diffusion step. The checkpoint ships both stacks (126GB bf16, ~63B params) and the card requires 2x80GB (~59GB/GPU). The towers can't be shared (the paper's own ablation: tying is "substantially worse") or run sequentially (the denoiser cross-attends to and seeds Mamba state from the live context tower every block-step), so the 2.42x is bought by doubling the model's resident memory. The comparison that matters: on one RTX 5090, speculative decoding already delivers the same multiplier for a draft head in single-digit GB — from this rig's t036 spec-decode run on Gemma-4-26B-A4B (sm120): MTP 2.13x, DFlash 2.19x, EAGLE-3 1.69x, for ~0-2GB of drafter (MTP ships inside the model). Same ~2.2x speedup, ~60x the memory. Two honest caveats: different mechanisms on different models (an architectural argument, not a controlled A/B — TwoTower won't run on the rig), and TwoTower's 2.42x is measured vs plain AR with no speculative-decoding baseline, so against a production AR server already at ~2.2x via spec-decode the marginal win mostly disappears while still costing a second 30B backbone. Doesn't run on consumer hardware at all: 126GB exceeds a 96GB box even fully offloaded, no quant of the custom trustremotecode arch exists, and the mambassm/causal_conv1d kernels are the usual Blackwell build wall. The two HF repos are the same checkpoint (config + shard sha256 identical; -Labs- adds inference.py). Arithmetic autopsy from the published artifact, no 5090 run. Companion to the GLM-5.2 datacenter-only autopsy and the spec-decode three-way.
Making music on a gaming GPU: ACE-Step 1.5 writes a 4-minute song in 1.75s · chart	First RTX-5090/sm_120 numbers for ACE-Step 1.5, an open-weights text-to-music model. One gaming GPU generates a full 4-minute song in 1.75s of compute (2B turbo, DiT-only, bf16, 8-step, batch 1) — level with the model authors' own A100 claim (~1-2s) and ~6x past the RTX 3090 (<10s), at 137x real-time. The higher-quality XL 4B tier costs ~1.65x the time (2.9s, 83x) and ~60% more VRAM (14.8 vs 9.4GB); both fit far inside 32GB, and XL would run on a 16GB card. Real-time factor RISES with length — 81x at 30s to 137x at 4min — because the turbo model's step count is fixed at 8 (distilled from ~50), so a 4-minute track is ~5x the compute of a 30-second one, not 8x. Measured out-of-box with no torch.compile and no quantization: a floor, not a ceiling. Speed only — audio quality is left to the ear (paired 2B-vs-XL samples), a prompt-alignment score the natural follow-up. Companion to the DiffusionGemma AR-vs-diffusion null.
Bias-only steering: nothing moves at bounded budget, and random rewards match correct ones · chart	Bias-Only Reasoning Steering (arXiv 2505.18706, EMNLP 2025) claims RL-training one bias vector per layer (~0.0016% of params, added to mlp.downproj) matches full RL fine-tuning: Qwen2.5-Math-7B MATH500 52.2 to 79.9 (steering even beats full-FT). Their pinned stack is dead on arrival on consumer Blackwell — torch 2.6.0+cu124/vllm 0.8.5 fails its first kernel launch on sm120 — so the recipe was reimplemented from their own configs (RLOO, steering lr 1e-3, qwenmath template, DeepScaleR) at a matched bounded budget (20 steps x 8 prompts x 8 generations, ~1,280 rollouts vs their ~645K), plus the controls neither paper reports: their-own-config LoRA (r4, downproj only), random-reward steering (Spurious-Rewards protocol), and a zero-training 'To'-prefix probe of their companion paper's first-token-substitution mechanism. A five-arm null: base 54.6 MATH500 / 45.0 AMC23 (reproduces their 52.2/45.8 starting point), steering 54.4/45.0, LoRA 53.8/40.0, random-reward steering 54.2/45.0, 'To'-prefix 53.0 — every arm is the base. Correct rewards buy nothing over coin flips at this budget, and the claimed ~10-11pt 'To'-prefix gain lands at -1.6 on the standard template, where the base's generations already open with 'To'. Wall-clock decomposition (identical across arms): rollouts 75%, backward+update 25%, grading under 1% — the '34s vs 52m' headline counts only the optimizer sliver, and the slice that shrinks with trainable-param count is ~none of a step; on 32GB the real bias-only win is memory (full-param 7B RL does not fit at all; ~100K bias params train comfortably). Bounds where the gain is not (early), does not refute their full-recipe endpoint. Steering checkpoints served in stock vLLM via a Qwen2-to-Llama re-badge (mlp_bias=true), fp32-verified logit-identical.
Ornith-1.0-35B's self-written scaffold doesn't survive a different harness · chart	DeepReinforce's Ornith-1.0 (MIT) is an RL coder that co-trains a task-specific agent scaffold INTO the weights; the 35B claims 75.6 SWE-bench Verified (the 82.4 headline is the unrunnable 397B flagship), measured in OpenHands. Held the bugs, harness, quant (Q4KM), and thinking mode (off) fixed and changed only the model: against the exact base it was post-trained from (Qwen3.5-35B-A3B) in the rig's strict native loop, Ornith-35B resolves 5/12 vs the base's 7/12 — a regression, and a strict subset (it recovers nothing the base missed). The two losses (astropy-12907, xarray-3677) are bugs the base solved, lost to tool-call JSON fragility: Ornith emits multi-line bash with unescaped newlines, llama-server's strict parser 500s, and even after the loop is hardened to feed the error back and let it retry (a fix inert for the base, which never 500s), it burns its full 40-step budget producing no patch. The reading: their 75.6 lives in a lenient harness with the model's own scaffold; stripped to a strict neutral loop the self-scaffold model is more fragile than the base it was trained from, so the orchestration didn't travel. An agentic-coding number is a property of the model and the harness, not the model alone. And the rig's own synthetic Agentic Score is worse than blind to it: it ranks Ornith-35B at 98.06, ABOVE the base's 97.5 (#6 on the board), while Ornith resolves fewer real bugs — the synthetic axis inverts the ranking, scoring fluent tool-driving rather than real-bug fixing. Not a refutation of the 75.6 (different harness, temperature, and scaffold); the 397B flagship is datacenter-only and untested. The fourth Qwen-family coding tune to regress on the real anchor — only pi-tune, trained on real agent traces, improved.
Swap the agent harness, not the model: a +1/12 persistence lever · chart	How much of an agentic-coding score is the model and how much is the harness wrapped around it? Held the model fixed (Qwen3.6-27B-Q6_K, one local llama-server on a 5090, think-off, temp 0) and swapped only the agent scaffold, graded on 12 SWE-bench Verified bugs with the official harness. The rig-native tool loop (40-step budget) resolves 8/12; omp v16.1.14 (a deps-free CLI agent, 450s budget, same model and `:8090` endpoint) resolves 9/12 — a strict superset, the lone delta being sphinx-8621. The mechanism is persistence, not reasoning: on the 4 hard bugs the native loop committed no patch (gave up) 3 times, omp once; omp lands patches where native quits, and one of those passed. Both harnesses miss the same 3 bugs (seaborn-3187, requests-1921, pylint-7080) — same model, same ceiling, so the scaffold only moves the give-up rate. Empty-patch rate is the give-up tell, here separating two harnesses on a fixed model. Honest limits: n=12 single seed, so the +1 is inside the noise (the signal is the direction plus the mechanism); the budgets differ by construction (steps vs wall-clock), which is the point — a harness is prompt plus tools plus stopping policy, bundled. The inverse of the Ornith-1.0 claim the rig tests next (RL that bakes the scaffold into training).
Qwen-AgentWorld's zero-fine-tune transfer doesn't reproduce on a 5090 · chart	Qwen-AgentWorld (arXiv 2606.24597) trains a language world model to predict environment transitions and claims the warm-up transfers to agentic tasks with zero agent fine-tuning, +3.4-12.8%. Tested the released LWM-warmed 35B-A3B against its own base Qwen3.5-35B-A3B in a think-OFF/temp-0 controlled A/B on one RTX 5090. The synthetic agentic board is flat (97.5 = 97.5, a saturated axis that hides differences); the real SWE-bench Verified anchor (30 bugs, official harness) goes 14/30 vs the base's 16/30 — a reshuffle rather than a collapse (11 solved by both, 3 AgentWorld-only, 5 base-only), net -2 with more give-ups (13 empty patches vs 10, mean 33/40 steps: it explores more and commits fewer fixes). The claimed +3.4-12.8% transfer lands at 0% on synthetic and -12.5% on real coding. A scoped null: think-OFF to match the base's banked number, so it doesn't refute a think-ON gain (that A/B is the queued falsification leg), and it tests the SWE-coding slice of a seven-domain claim.
FP4 on a consumer 5090: the Blackwell headline loses to plain int4 · chart	FP4 is the Blackwell selling point — benchmarked on one RTX 5090 (sm120) against the quants you'd actually run, Qwen3-14B in vLLM 0.21. Two findings that compound. (1) NVFP4 is the only quant that won't run out of the box: AWQ/FP8 use prebuilt Marlin kernels, but NVFP4 makes FlashInfer JIT-compile native sm120 FP4 cutlass kernels at load — needing ninja on PATH + a real CUDA toolkit + the correct CUDAHOME (the default /usr/local/cuda-13.0 doesn't exist on the box) + a flashinfer-cache clear. (2) Once native FP4 is genuinely running (declared modeloptfp4, not a Marlin dequant fallback), it's still slower than AWQ int4 at every batch: batch-1 decode AWQ 150 vs NVFP4 100 (0.66x) vs FP8 90; batch-32 AWQ 3937 vs NVFP4 3321. The 4-6x FP4 numbers are B200 tensor-core throughput; on consumer sm120 a mature int4-Marlin kernel wins. bf16-14B doesn't fit 32GB (no KV room). (3) The academic real-FP4 path (QuTLASS MXFP4, arXiv 2509.23202), built from source on sm120a (CUTLASS submodule, torch 2.8/cu128, forced -ccbin g++-14 past the GCC-15/CUDA-12.8 guard): its 4x is real at the GEMM — the MXFP4 matmul crosses 4x by batch 128 and peaks ~6x over bf16 on Qwen3-8B — but end-to-end decode runs 3-4x slower than bf16 (20.4 vs 78.6 tok/s at batch 1) and uses ~2x the VRAM, because decode is memory-bound and pays a fixed per-layer rotation+quant tax the tiny matmul can't amortise. Verdict: use AWQ int4 for serving, skip NVFP4 on consumer Blackwell; QuTLASS MXFP4 only pays off for compute-bound high-batch/prefill work, not single-stream serving.
One-Shot EM doesn't reproduce on a 5090: entropy fell, accuracy didn't · chart	One-Shot Entropy Minimization (arXiv 2505.20282) claims +24.7 on Qwen2.5-Math-7B from ONE unlabeled example in ~10 steps, no rewards. The full-param recipe OOMs on 32GB (14GB weights + 14GB bf16 grads), so the consumer-feasible version is LoRA (batch 16). Measured greedy pass@1 with the authors' grader: base reproduces the paper (MATH500 53.4 vs 53.0), but EM adds +2.0 at its peak step then collapses back by step 15, and AMC23 goes −2.5 (claim was +25.8 / +26.2). The keeper is the mechanism: the entropy objective trained fine (mean per-token entropy 0.098→0.035) while accuracy stayed flat — distribution-sharpening, not learning, in the paper's own words. Same base as Spurious Rewards (+21 on MATH500 from random rewards). Honest limits: full-param didn't fit, so this isn't a refutation of the multi-GPU number; the few-shot format control backfired on Qwen-Math's native zero-shot CoT.
Draft-free spec-decode on Qwen3-8B: workload split + dumb-vs-fancy null · chart	Cache-only speculative decoding (no draft model, no extra VRAM) on Qwen3-8B: one greedy loop, three drafters (AR / one-line prompt-lookup / Cacheback's dynamic LRU table) × three workloads. The speedup is workload-shaped, not method-shaped — code 1.45-1.47x, summarize 1.30x, open chat 1.26x, with MAT tracking speedup 1:1, biggest where local agents live and no workload at zero. The null: Cacheback's LRU table ties one-line prompt-lookup (identical on 2/3 workloads, +0.01 MAT on code) — at leader-length 1 they are the same algorithm; Cacheback's real edge is its frozen corpus + tree drafting, not the dynamic table, and the cited "1.86x" is Vicuna-7B + frozen corpus on a 4090, not a modern 8B. Lossless: 46072/46080 tokens byte-identical to greedy; the 8 misses are exact bf16 logit ties (gap 0.000) where greedy itself is non-deterministic, not a decoder bug.
Sovereign TTS head-to-head: 1.7B Apache beats 4B research-license · chart	Qwen3-TTS-1.7B (Apache) vs Fish-S2-Pro (4B, research license) on one RTX 5090, 150 Seed-TTS-eval EN utterances, both bf16 and neither compiled. Round-trip WER is a tie (0.6% each — Fish's sub-1% claim holds, Qwen matches it); SIM-o 0.699 vs 0.625; but RTFx 2.22× vs 0.39× and first-audio latency 1.72s vs 9.74s. Fish-S2-Pro's serving stack assumes SGLang + torch.compile + datacenter cards (its RTF<0.5 is an H200 number) — out-of-the-box on a consumer GPU the small open model is 5.7× faster at the same intelligibility. Size + serving assumptions, not quality. A compiled-Fish rerun is the obvious follow-up.
GLM-5.2 autopsy: clever MLA+DSA, still datacenter-only · chart	The biggest open-weights drop in months (743B MoE, MIT, 1M context), measured by arithmetic from the published config — not served. Credit first: `glm_moe_dsa` = MLA + DeepSeek sparse attention compresses the KV cache ~57× (1M = ~88 GiB vs ~4.9 TiB). But the weights can't fit 96GB addressable at any quant (743B needs 1.03 bits/weight; smallest ~1.58-bit = ~147 GB), the 1M KV alone ≈ the whole machine, and DSA saves compute not memory. ~235 GB to use 1M context — past a single H200. Clever ≠ consumer; the home-lab move is to wait for a GLM-5.2-Air.
Qwen3.6-27B pi-tune: the coding tune that works · chart	A community QLoRA SFT of Qwen3.6-27B on REAL non-thinking agent traces, measured controlled vs its base at matched Q6_K across four legs. Quality (93.3 vs 94.0) and synthetic Agentic Score (98.01 vs 98.61) stay flat — but real SWE-bench Verified resolve goes UP 19 → 20/30 (give-ups 8 → 6) and the MTP drafter holds (2.0-2.4× vs base 1.8-2.2×) where Qwopus-Coder's degraded. The first of three Qwen3.6-27B coding tunes to improve real bug-fixing: across all three the synthetic score is a 2.4pt band while real SWE spans 11-20, so training-data provenance (real traces > synthetic distill), not the "agentic coder" label, is what the anchor sees.
Qwable-5-27B-Coder: real traces, still regresses · chart	A real-trace SFT of Qwen3.6-27B (Claude Fable-5 then Kimi 2.7 Coder agent traces), measured controlled vs its base at matched Q6_K. Quality (93.7 vs 94.0) AND the synthetic Agentic Score (98.61 vs 98.61, identical to the decimal) stay flat — but real SWE-bench Verified resolve drops 19 → 17/30 and give-ups rise 8 → 10. The 4th Qwen3.6-27B coding tune on the anchor: real traces are necessary but not sufficient (pi-tune's terminal/repo/DevOps traces remain the only data that moved real resolve up). The MTP drafter survived the SFT (1.9-2.4×, base range) — so drafter-survival is not capability-preservation.
Qwable-3.6-27b: the distill every cheap eval passes, SWE-bench fails · chart	A dense Qwen3.6-27B + Fable-5-style SFT, measured controlled vs its base at matched Q4KM. Quality (93.4 vs 94.0) AND the synthetic Agentic Score (97.64 vs 98.19) stay flat — but real SWE-bench Verified resolve drops 18 → 11/30 and give-ups rise 7 → 13. Quant ruled out (base Q6 → Q4 = −1 bug). The agentic board would call it neutral; only the reality anchor caught the give-up regression — a 3rd failure mode, and the inverse of the MoE Qwable-v1 (whose synthetic honestly declined).
Qwable-v1: agentic distillation regresses vs base · chart	A Claude-Code/Fable-5-distilled "agentic coder" (Qwen3.6-35B-A3B) measured controlled vs its vanilla base + the Opus-reasoning-distill, same Q5KM. Every post-train step LOWERS the agentic score (99.58 → 97.92 → 96.25), and real SWE-bench Verified resolve drops 19 → 11/30 with give-ups nearly doubling (9 → 16 empty patches). Not a mirage — synthetic fairly predicts real here; the distillation regressed a top-tier base (the vanilla base ties best real resolve on the board).
ECHO maze microcosm · chart	ECHO's "world model for free" env-token loss, isolated in a 10M from-scratch maze transformer (behavior cloning; the only A/B is the loss mask). A clean null under both observation regimes — full-obs and walls-only — every gap inside the seed spread, no growth with maze size. The free aux loss is worth exactly that in pure imitation; the reported gain must live in ECHO's on-policy RL coupling, not the loss as a plug-in. (The paper has no maze — this microcosm is original, verified by grepping the source + repo.)
Spec-decode three-way (Gemma 4 26B-A4B) · chart	MTP vs EAGLE-3 vs DFlash on one RTX 5090 (vLLM 0.21, sm120), including the EAGLE-3 leg nobody publishes. Single-stream near-tie: DFlash 2.19x, MTP 2.13x, EAGLE-3 1.69x — DFlash is feast-or-famine (prose 1.04x, repetitive 4.37x), MTP the steady all-rounder. Six consumer-Blackwell fixes to run it at all (NVFP4 to MARLIN, FLEXATTENTION for the #42068 attention deadlock) plus a self-caught /metrics parser bug. Dense 31B excluded: no clean quant fits 32GB.
Qwopus3.6-27B-Coder · chart	Four legs measured: q_avg 94.1 (#2 thinking-off, beats its base at a smaller quant); "100 tps" MTP verified (96-114 t/s) but the finetuned head accepts worse than the original (1.4-1.6x vs 1.8-2.2x); a perfect 100 Agentic Score — and 57% real SWE-bench resolve, below its own base (63%). Trained on Hermes traces: the in-distribution mirage the reality anchor was built to catch. 67% claim doesn't reproduce.
Qwopus3.6-27B-Coder-Compat: the regressions healed · chart	The "compatibility" re-release of Qwopus-Coder, measured controlled vs base Qwen3.6-27B + the prior Coder at matched Q6K (think-off, temp 0). Quality flat (qavg 93.7). The prior tune's two regressions both heal: the degraded MTP draft head recovers 1.4-1.6x → 1.9-2.3x (back on the base curve), and real SWE-bench Verified resolve returns to base parity — 8/12, the exact same bugs as base, +1 over the prior tune (recovers pytest-6202) with one fewer give-up. Agentic Score 100.0 ties the prior (saturated — the synthetic axis can't separate the two; the anchor can). A compat fix that costs no capability: the coder tune no longer carries a drafter or real-bug penalty.
Keye-VL-2.0-30B autopsy · chart	Five measured walls: "lossless 256K" needs 25.8GB of KV alone; the shipped sparse attention is O(N²)-memory (one 30.65GiB allocation at ~32K, measured); 4-bit quant reaches 4.7% of params; the code's API window is two transformers release candidates wide. Does not run on consumer hardware.
LocateAnything-3B · chart · raw	ScreenSpot-Pro 55.3% measured vs 60.3 claimed (32GB forces extra downscale; accuracy tracks screenshot size). Real fault line: text 63.2% vs icons 42.7%. PBD parallel box decoding verified at 2.07x on the SDPA fallback.
HRM-Text-1B · chart · raw gens	GSM8K 79.5% (claimed 84.5: holds at n=200). Omitting `token_type_ids` — which every standard harness does — silently costs 26 points. The recurrence bill: a 1.2B that decodes like a ~5B (42.9 tok/s bf16, 4x KV cache).
DiffusionGemma vs AR · chart	AR wins at every answer length: diffusion pays a fixed ~3s per 256-token canvas (0.8 effective tok/s on short answers; best case still 2.3x slower). Day-0 public GGUFs were unloadable — convert from source.
Gemma 4 31B QAT + MTP · chart	The MTP draft head lifts decode 76 to 125 tok/s (1.67x). QAT's real value is VRAM, not quality: the Q4 footprint is what fits 128K context plus the draft head on one card.
NVFP4 vs Q6_K · chart	Qwen3.6-27B: NVFP4 trades ~1pt qavg against Q6K.
GRPO on one 5090 · chart	Single-GPU RL: +7.66 GSM8K on a 4B. Train-prompt-to-eval-prompt alignment is the lever.
Embedding retrieval bench · chart	Local embedding models benchmarked for retrieval quality vs speed on the 5090.
Mistral Small 4 speed · chart	Speed profile vs gpt-oss-20b — and a benchmarking trap: reasoning is gated behind `reasoning_effort`, which defaults off.
LFM2.5-VL 1.6B extraction · chart	A 1.6B VL model as a local structured-data extractor.
Nex-N2-mini agentic probe	Adaptive Thinking saves 65% of tokens but costs 13pts task success. Superseded by the dedicated Agentic Score leaderboard (below).

Related Datasets

—witcheer/agentic-score-leaderboard — model-agnostic agentic tool-calling benchmark (7 models, 40 tasks) + the SWE-bench reality anchor
—witcheer/sovereign-asr-bench — local ASR on the 5090: Parakeet-TDT vs Whisper (WER / RTFx / VRAM)

Hardware

Component	Spec
GPU	NVIDIA GeForce RTX 5090 32GB (Blackwell, sm_120a)
CPU	AMD Ryzen 5 9600 (6c/12t)
RAM	64GB DDR5-5600
OS	Ubuntu 26.04 LTS
CUDA	12.8 (patched for glibc 2.41)

Tooling

All benchmarks generated with llm-bench-rig — open-source pipeline for speed and quality benchmarks on GGUF and safetensors models.