x0me/maple-preview-cuda-benchmarks

Maple Preview TQ2_0 CUDA Benchmarks

Reproducibility data for the TQ2_0 CUDA patches in `PascalAI2024/maple-preview-windows-cuda`. This repository contains benchmark data, patch files, hashes, and raw validation evidence. It does not duplicate the Maple model weights.

Result

The fresh local A/B/B/A validation on an RTX 4080 SUPER reproduced the fused-MMQ prompt-processing gain:

The prompt path improved 7.33× by mean. Generation was effectively unchanged (the medians differ by 0.06%); that is expected because the MMQ patch targets batched prompt matmuls, while batch-one generation already uses MMVQ.

Cross-architecture MMQ results

The local row is the 2026-08-09 controlled refresh on driver 610.62 and CUDA 12.8. The two remote rows are the harvested 2026-08-08 Vast runs. Host verification status is explicit because cheap hardware is useful evidence, not a laboratory pedigree.

What changed

The seven-patch series (0001–0008, with experimental patch 0005 reverted):

1. 1.Makes the existing TQ2_0 MMVQ path reachable for batch-one MoE generation.
2. 2.Replaces the scalar ternary dot product with packed signed-byte dp4a work.
3. 3.Enables TQ2_0 CPU-reference coverage in test-backend-ops.
4. 4.Specializes the standalone TQ2_0 dequant path.
5. 5.Adds TQ2_0 MMQ dispatch and template instantiation.
6. 6.Expands TQ2_0's strided packing into natural K-order signed bytes in shared memory.
7. 7.Enables the fused path after the correctness gate.

The final MMQ loader reuses llama.cpp's existing signed-s8 DP4A/Turing-MMA consumer. The important change is removal of the TQ2_0 → FP16 staging → cuBLAS round trip during prompt processing.

Protocol

Local validation uses `reproduce/05-validate-benchmark.ps1`:

1. 1.Refuse to run while another llama-* process is active.
2. 2.Warm both the MMQ and fallback builds; discard those warmups.
3. 3.Run A/B/B/A, where A is MMQ enabled and B is the patch-0007 fallback.
4. 4.Collect five built-in samples for pp512 and tg128 per invocation.
5. 5.Run the full MUL_MAT and MUL_MAT_ID CPU-reference matrices.
6. 6.Record model, binary, patch, source, driver, toolkit, and clock-state evidence.

Benchmark command:

llama-bench -m <MODEL> -ngl 99 -p 512 -n 128 -r 5 -t 8 -o jsonl

Model artifact:

• —Repository: `stamsam/maple-preview-gguf`
• —File: maple-tq2_0.gguf
• —SHA-256: 09d219202562dbd17722dc8e3273527a021182ab7f892c2a06aac459a8f3a090
• —Weights are not included here.

Source pins:

• —Benchmark repository commit: `d161aeb`
• —Runtime fork base: 9ee03eec62d088a117ab916bbe489e7a3872a21f
• —Patch hashes: `evidence/sm89-local-20260809/manifest.json`

Files

• —data/benchmarks.jsonl — one normalized row per hardware/variant result.
• —evidence/sm89-local-20260809/ — controlled A/B/B/A samples, GPU state, hashes, and complete correctness logs.
• —evidence/sm86-vast-20260808/ and evidence/sm120-vast-20260808/ — raw harvested remote results and hashes.
• —patches/ — the exact seven published patch files.
• —reproduce/05-validate-benchmark.ps1 — the local benchmark/correctness harness.

Limitations

• —No A100/sm_80 result is claimed. That is the most useful next architecture because the motivating public test used an A100.
• —Windows WDDM clocks and desktop load affect short runs. The harness warms both builds, publishes all samples, and reports mean, median, spread, minimum, and maximum rather than selecting the best invocation.
• —The Vast machines were explicitly unverified. Exact GPU, driver, CUDA, model, and patch metadata are included so readers can judge the evidence.
• —These measurements validate inference correctness and throughput. They do not validate the model's coding, multilingual, reasoning, or instruction-following quality.
• —Results are specific to this Maple llama.cpp fork and TQ2_0 artifact; they are not a general claim that ternary models always outperform GPUs or Q4 models.

Reuse and citation

Use the JSONL data for comparisons, but cite the exact record IDs and preserve the host-verification and driver fields. For implementation details, issues, or new hardware results, use the `maple-preview-windows-cuda` repository.