Modelpublic

rdtand/Laguna-S-2.1-prismaquant-gridbook-6bit-vllm

sourceHugging Faceopenmdw-1.1updated 2mo agoView on Hugging Face

8likes77downloads

Model Card

Laguna-S-2.1 — PrismaQuant gridbook 6-bit (256k context on one DGX Spark)

poolside's 117B sparse-MoE coding model served on one 128 GB DGX Spark at 6.0 bits per parameter with the full 256k context window, through stock vLLM with an out-of-tree plugin — no forked runtime, no core patches. Weights are stored in the gridbook codebook formats (FP8-CB / NVFP4-CB: product-VQ over 8-weight vectors whose codebook values live on hardware grids) and decoded inside custom CUDA/Triton kernels at matmul time. Formats + kernels: github.com/RobTand/gridbook. Produced by the PrismaQuant measured-allocation pipeline.

—Base model: poolside/Laguna-S-2.1 (OpenMDW-1.1), 48 layers, 256 routed experts top-10 + 1 shared expert, hybrid sliding(512)/full attention, 256k context.
—Artifact: single safetensors file, 84 GB, 6.009 bpp over quantizable parameters. Allocation chosen on a measured 27-point rate-distortion sweep (knee analysis included in the repo discussion), per-Linear formats solved by knapsack DP over per-(Linear, format) measured costs.
—Measured on-box (DGX Spark GB10, vLLM eager, fp8 KV, plugin 2026-07-23, serve command below): decode 14.9 tok/s, prefill 1,821 tok/s @8k / 1,822 tok/s @63k; 256k context boots; exact needle retrieval at 63k-token depth. Only 12 of 48 layers are full-attention (the rest slide at w=512), so a full 256k request costs ~6 GiB of fp8 KV — the whole model + 256k cache fits the Spark's unified pool. Config tradeoff: the fast-prefill command below (--max-num-batched-tokens 16384, util 0.85) leaves 1.53× full-256k request concurrency; default batching at util 0.87 gives 3.28× concurrency with ~6× slower long-prompt prefill. A DFlash drafter adds nothing on this hardware (37.5% acceptance; measured net-negative to parity) — serve drafter-free.

Format breakdown (measured allocation, not hand-picked)

format	where	params	bytes
FP8CBK48 (6.0 bpw)	60 expert units	72.5B	54.4 GB
FP8CBK44 (5.5 bpw)	22 expert units	26.6B	18.3 GB
FP8CBK40 (5.0 bpw)	10 expert units	12.1B	7.5 GB
FP8CBK38 (4.75 bpw)	2 expert units	2.4B	1.4 GB
FP8CBK47 / NVFP4_CB	27 small Linears	0.3B	0.2 GB
BF16	attention / shared-expert / dense mlp	3.0B	6.0 GB

The experts carry the compression, graded down the fp8-CB ladder by measured sensitivity; the entire per-token dense path (attention, gates, shared expert) rides BF16 — at this budget those 3B params cost only 6 GB and buy maximum fidelity on every high-leverage tensor. This heterogeneous assignment is the PrismaQuant thesis: allocation measured end-to-end beats any uniform format.

Serve this model


Plugin	`gridbook` — an out-of-tree vLLM quantization plugin. Stock vLLM, no fork, no core patches.
GPU	NVIDIA Blackwell, compute capability `sm_120` / `sm_121`. Measured on GB10 / DGX Spark (`sm_121`). On older GPUs the plugin still loads but runs its Triton fallback kernels — correct, not fast, and not a production serving target.
Memory	89.4 GB of weights on disk = 83.2 GiB resident. Needs a ~128 GB unified/VRAM pool: measured on one DGX Spark at `--gpu-memory-utilization 0.85`. Only 12 of 48 layers are full-attention (the rest slide at w=512), so a full 256k request costs ~6 GiB of fp8 KV — model + 256k cache fit the Spark's pool.
vLLM	Version used for these measurements is not recorded on this card.
Toolchain	CUDA toolkit with `nvcc` on `PATH` in the serving container — the plugin JIT-builds its kernels on first model load (~30 s, cached). `nvcc` 13.0 is the tested toolchain.
Parallelism	Single GPU (`tp=1`). Tensor parallelism is not implemented in the plugin.

bash

pip install gridbook
vllm serve rdtand/Laguna-S-2.1-prismaquant-gridbook-6bit-vllm \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 262144 --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.85 --enforce-eager \
  --max-num-batched-tokens 16384

Target hardware: Blackwell (GB10 / sm_120+). The gridbook plugin JIT-builds its kernels on first load.

Honest limitations

—No quality claims. There is no servable BF16 teacher for a 117B model on this hardware, so no KL/perplexity/benchmark deltas are published for this artifact. Gates that were run and passed: load, greedy-decoding coherence, 256k boot, 63k needle retrieval, packing/completeness checks, speed. Treat quality as unvalidated until an independent A/B exists.
—The DFlash speculative-decoding drafter is not yet integrated/validated with this artifact; the numbers above are drafter-free.

Measured A/B vs poolside's NVFP4 release (same box, same protocol)

Identical protocol per arm: fresh boot, slack-gated util clearing the same 8 GiB floor (gridbook: 0.85 + --max-num-batched-tokens 16384, the serve command above; NVFP4: 0.85), vLLM eager, fp8 KV, greedy single-request timing.

	gridbook 6.0 bpp (this repo)	poolside NVFP4 (~4.5 bpp)
weights	84 GB	67 GB
decode	14.9 tok/s	19.1 tok/s
prefill @8k	2,186 tok/s (plugin ≥2026-07-28)	3,603 tok/s
prefill @63k	1,822 tok/s	2,403 tok/s
256k boot	yes	yes
63k needle	exact	exact

(gridbook numbers: plugin >= 2026-07-23 — the MoE prefill rides a CUDA chunk-expander into vLLM's own fused-MoE grouped kernel; serve with --max-num-batched-tokens 16384 and slack-gated util 0.85 as in the command above. Earlier plugin builds prefilled ~6x slower. Full-256k request concurrency at this config is 1.53x; drop --max-num-batched-tokens to trade prefill speed back for KV headroom.)

Reading it honestly: decode is per-byte neutral — the tok/s ratio equals the byte ratio (67/84 = 0.80), so gridbook's decode kernels move bytes exactly as fast as the native NVFP4 path; the deficit is purely that 6.0 bpp stores more bits per parameter than 4.5. Prefill trails by 1.65x at 8k — the remaining gap is the transient expand round-trip vs NVFP4's fused CUTLASS grouped GEMM, a known kernel roadmap item (persistent-N decode-in-mainloop), not a format property. No quality comparison is published for either arm (no servable BF16 teacher at this scale).

Attribution

Quantization by Robert Tand (robert.tand@icloud.com) — PrismaQuant / gridbook. Base model by poolside, served via vLLM.