CoolFace
Modelpublic

rdtand/Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
6likes100downloads
Model Card

Qwen3.6-27B — PrismaQuant gridbook 5.5-bit (gridbook formats, vLLM)

23.0 GB codebook-quantized Qwen3.6-27B (vision-language, MTP included) serving through stock vLLM with the open gridbook plugin — no fork, no core patches. Weights are stored as product-VQ codebook indices whose codebook values lie exactly on hardware grids (e4m3 / E2M1), so serving decodes into bit-standard FP8/NVFP4 tensors and runs on the same tensor-core GEMM paths a conventional checkpoint uses.

Measured quality (held-out KL vs the BF16 teacher)

Protocol: top-20 prompt_logprobs KL on a 12.3k-token held-out corpus (disjoint from calibration), 8,176 positions, all arms served back-to-back on one box against the same teacher capture — a strict relative comparison.

artifactsizeKL (all)KL (confident half)top-1 (conf)
this artifact (CB ladder, 5.5 bpp body)23.0 GB0.00490.0029599.88%
Qwen3.6-27B-PrismaAURA-5.5bit (NVFP4+FP8)23 GB0.02110.013099.58%
Qwen3.6-27B-PrismaSCOUT 5p3120.6 GB0.03440.024999.04%

At matched size and matched body bitrate, the codebook ladder carries −77% KL versus the strongest conventional (NVFP4+FP8) artifact in this family. Same model, same bytes, same protocol — the delta is the format.

  • —Pre-ship validator: PASS — mean NLL 1.400/tok, p99 NLL 1.807 (thresholds 3.0 / 6.0)
  • —ToolEvalBench (hardmode, seed 1234, sequential): 87 / 100 (129/148) — family anchors on identical protocol: BF16 teacher 86, PrismaSCOUT 85, PrismaAURA-5.5 91; repeat churn band ±2–3

What's inside

sectionformatnotes
504 body LinearsFP8-CB K36–K47 (426 units) + BF16 (69)per-Linear rungs chosen by measured error under a byte budget; K47 (5.88 bpw) ×175, K44 ×109, K40 ×108 — the ⅛-bpw all-integer ladder in actual use
vision tower (110 Linears)NVFP4 weight-only (W4A16)quantized ViT, verified on image inputs
MTP draft blockBF16canon throughput selector: highest-fidelity rung that fits
embeddings + lm_headBF16248k vocab, untied

Full per-Linear assignment ships in allocation/. Codebooks (cb_codebooks.pqcb) are shared per role family and total a few MB.

Serve this model

Plugin`gridbook` — an out-of-tree vLLM quantization plugin. Stock vLLM, no fork, no core patches.
GPUNVIDIA Blackwell, compute capability sm_120 / sm_121. Measured on GB10 / DGX Spark (sm_121). On older GPUs the plugin still loads but runs its Triton fallback kernels — correct, not fast, and not a production serving target.
Memory20.8 GiB weights + KV cache, measured on one 128 GB unified-memory GB10 (Model loading took 20.82 GiB). Before gridbook 0.1.0 the same artifact loaded at 35.86 GiB — dense codebook weights were resident twice (issue #1, fixed in 0.1.0); use 0.1.0 or newer. A 32 GB consumer Blackwell (RTX 5090, sm_120) should now fit weights plus a useful KV budget, and the issue reporter measured 20.54 GiB with 32k context using equivalent patches — but we have not verified 0.1.0 on a 32 GB card ourselves.
vLLMServed here on the vllm-node image used for the 0.1.0 verification; also exercised on 0.25.1 and 0.23.1rc1.dev1060 by an external reporter (issue #1). The plugin surface is small but vLLM internals drift — pin a version you have tested.
ToolchainCUDA toolkit with nvcc on PATH in the serving container — the plugin JIT-builds its kernels on first model load (~30 s, cached). nvcc 13.0 is the tested toolchain.
ParallelismSingle GPU (tp=1). Tensor parallelism is not implemented in the plugin.
bash
pip install gridbook   # JIT-builds kernels for your GPU (capability >= 8.0 floor)
vllm serve rdtand/Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm \
  --max-model-len 32768 --gpu-memory-utilization 0.90

Honest limitations

  • —KL numbers are the relative protocol above, not the full-vocab exact-KL contract; the ranking is protocol-internal and same-session.
  • —ToolEvalBench ordering vs PrismaAURA-5.5 may differ from the KL ordering (task metrics churn ±2–3 points between serving configs).
  • —One box, one seed per benchmark.

Quantized with PrismaQuant (measurement-driven per-Linear format allocation); format spec and kernels: gridbook, Apache 2.0.

Contact: robert.tand@icloud.com