JasonW2025/Laguna-S-2.1-ModelOpt-NVFP4-W4A4-KVcal-vllm

KV-calibrated variant. Identical weights to Laguna-S-2.1-ModelOpt-NVFP4-W4A4-vllm, plus the k_scale/v_scale tensors that artifact declares but does not ship (a kv_fp8_cast export writes kv_cache_quant_algo: FP8 without calibrating). The 96 scales are transplanted from poolside's own Laguna-S-2.1-NVFP4 — the same base model, so its calibrated amax applies; this is an approximation and is labelled as one (the donor's quantization noise rides along). Verified at serve time as a pair: the base warns Using KV cache scaling factor 1.0 for fp8_e4m3, this variant does not (vLLM v0.25.1). No accuracy delta has been benchmarked.

Laguna-S-2.1 — NVIDIA ModelOpt NVFP4 W4A4, 4.70 bpp

A 64.38 GiB NVFP4 quantization of `poolside/Laguna-S-2.1` (117.6 B params) that serves on a single DGX Spark (GB10, sm_121a).

*Weights and activations are 4-bit* on the MoE experts — not weight-only. That matters on this hardware: it is what lets the fused NVFP4 CUTLASS MoE kernel run at all. A weight-only NVFP4 export has a QuantKey ending xNone, which every fused backend rejects, leaving only MARLIN.

Where the bytes actually are

Measured by reading every tensor's dtype and byte range out of the shard headers, not inferred from config_groups:

Read the 4.70 bpp headline with that table in hand. The experts alone are ~4.50 bpp; the rest of the average is BF16 embeddings, lm_head, shared expert and layer-0 MLP — 1.185 GiB of it in embed_tokens + lm_head for a 100,352-token vocabulary. If you are comparing this against a lower-bpp quantization of the same model, compare the expert format, because that is where 92.4 % of the bytes live and where the two will differ.

config_groups lists only 192 + 47 targets — 48×4 attention projections and 47 expert stacks. Everything else in the table above is untouched BF16.

⚠️ Serving: three things to get right

1. The KV cache is FP8 with calibrated scales — do not strip the declaration

config.json declares kv_cache_scheme: {"dynamic": false, "num_bits": 8, "type": "float"} — static FP8 KV — and this repo does ship the matching tensors: 96 of them, k_scale + v_scale, two per attention layer across all 48 layers, in model-kvscales.safetensors. Serve it as it is. Nothing needs removing.

Check it yourself, and note the /resolve/ in that URL:

curl -sL https://huggingface.co/JasonW2025/Laguna-S-2.1-ModelOpt-NVFP4-W4A4-KVcal-vllm/resolve/main/model.safetensors.index.json | python3 -c "import json,sys; m=json.load(sys.stdin)['weight_map']; print(sum(k.endswith(('k_scale','v_scale')) for k in m), 'KV scale tensors')"
# 96 KV scale tensors

`/raw/` returns a git-lfs pointer, not the index — 133 bytes of oid sha256:…. Grep that for k_scale and you get a confident, wrong 0. Expect 2 per attention layer; a genuine zero next to a kv_cache_scheme declaration is the defect described below.

Where the scales came from, and what that is worth. They are transplanted from poolside's own `Laguna-S-2.1-NVFP4` — the same base model, so its calibrated amax applies to these tensors. It is an approximation and is labelled as one: the donor's own quantization noise rides along. No accuracy delta has been benchmarked. Every number in Performance and Quality below was measured on the byte-identical sibling before these scales existed — that is, with the cache running at scale 1.0. They have not been re-measured here.

Why this variant exists. The sibling `…-W4A4-vllm` declares FP8 KV and ships zero scales, because its ModelOpt recipe used use_constant_amax: true: _get_amax returns the constant 448.0, enable_stats_collection skips the quantizer, no _amax buffer is ever created, and the exporter has nothing to rename into k_scale/v_scale. Numerically that is 448/448 — a hard-coded scale of exactly 1.0, which is also vLLM's own fallback. A declaration without the tensors is not a half-configured feature; it flips the cache to fp8 at 1.0. On this model family we measured roughly 2 points of agentic score against that state at no speed gain — but read it beside The noise floor, stated up front below, where four of fifteen categories move by 1–2 points between two runs of identical bytes. A 2-point gap does not establish an ordering. The case for calibrating is that on any rebuild it is free, not that the gap is proven.

`--kv-cache-dtype auto` changes none of this, in either direction. vLLM reads kv_cache_scheme out of the quantization config before honouring auto and resolves to kv_cache_dtype=fp8_e4m3. The flag cannot remove a declaration, and it cannot invent tensors that were never exported.

One log line is not the evidence you want. This still appears here, and is benign:

WARNING [kv_cache.py:134] Checkpoint does not provide a q scaling factor.
        Setting it to k_scale. This only matters for FP8 Attention backends

It fires because q_scale is absent while k_scale is present — it says nothing about whether K/V were calibrated. The line that actually separates the two artifacts is Using KV cache scaling factor 1.0 for fp8_e4m3: the sibling emits it, this one does not. We checked them as a pair on vLLM v0.25.1.

If you want a BF16 cache regardless, popping kv_cache_scheme from a config overlay will give you one — but on this repo that discards the 96 scales the variant exists to carry. That overlay belongs on the sibling, not here.

2. moe_backend must be a fused CUTLASS one on GB10

Two work, and they are the two you should choose between:

• —`FLASHINFER_CUTLASS` — what the numbers in Performance were measured with.
• —`CUTLASS` (vLLM's own, logged as VLLM_CUTLASS) — also runs on sm_121; we have served this artifact under it at 1,048,576 max-model-len.

Two do not work, and two are actively misleading:

• —FLASHINFER_CUTEDSL and FLASHINFER_TRTLLM fail is_device_capability_family(100), which computes current.to_int() // 10 == capability // 10; GB10 is sm121 → `12`, so it matches family **120**, not 100. Forcing past the guard gets a `getnvccflagslist( supportedmajorversions=[10]) failure in CUTEDSL and a cubin named …_sm100f` in TRTLLM.
• —MARLIN and HUMMING load this checkpoint and silently ignore its 36,288 activation scales. vLLM's nvfp4 oracle nulls a13_scale/a2_scale on those paths and builds nvfp4_w4a16_moe_quant_config — the function name is the proof. Your W4A4 model then runs W4A16 numerics with no warning at all. If you benchmark this artifact under MARLIN you are not benchmarking W4A4.

Check the startup line Using '<X>' NvFp4 MoE backend out of potential backends: [...] and confirm <X> is one of the two fused CUTLASS options.

vllm serve <model> --served-model-name laguna-w4a4 \
  --tensor-parallel-size 1 --dtype bfloat16 \
  --attention-backend FLASHINFER \
  --kernel-config '{"moe_backend":"FLASHINFER_CUTLASS"}' \
  --gpu-memory-utilization 0.85 --max-model-len 32768 --max-num-seqs 4 \
  --kv-cache-dtype auto \
  --no-enable-prefix-caching \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1

Environment: CUTE_DSL_ARCH=sm_121a, VLLM_TRUST_REMOTE_CODE=1. Measured on vllm/vllm-openai:v0.25.1.

3. The reasoning parser is broken upstream (affects every Laguna deployment)

vLLM's PoolsideV1ReasoningParser derives from the DeepSeek-V3 parser, which picks its implementation from chat_template_kwargs["thinking"] / ["enable_thinking"]. Both default to False, so it silently becomes an IdentityReasoningParser whose reasoning_start_str/reasoning_end_str are None — vLLM logs "Auto-initialization of reasoning token IDs failed" and carries on. Chain-of-thought then lands in scored content and a stray </think> appears in ~68 of 69 bench scenarios.

Every score on this card was measured with that defect present, so they are all a few points low. On a sibling artifact, fixing it was worth +3. We have not re-run this one.

Performance

One DGX Spark, GB10 / sm121a, 121.63 GiB unified. `vllm/vllm-openai:v0.25.1`, the serve line above, `FLASHINFERCUTLASS confirmed in the log. **Strictly sequential (concurrency 1)**, temperature 0, prefix caching off, warmups discarded, median of 3. KV cache **1,200,440 tokens** at --max-model-len 32768 --gpu-memory-utilization 0.85`.

Decode: 27.96 tok/s (median; 27.51–28.00 across every row). Note how flat that is — decode on a top-10-of-256 MoE is bandwidth-bound on the active slice, so it barely moves with context at these lengths.

Prefill: ~3 300–3 660 tok/s once the prompt is large enough for compute to dominate. Below ~2 k tokens the "prefill" column is meaningless — TTFT there is fixed overhead, not throughput — so do not read the 727 tok/s row as a prefill rate.

TTFT, with its spread — because the spread is the finding

Warmups discarded per shape, 7 kept samples each:

The two ends are tight — ±1 ms at 81 tokens, ±14 ms at 9 325. The middle is bimodal: at 633 tokens a minority of requests complete in ~316 ms and the rest in ~825 ms. That is not warmup (discarded per shape) and we do not currently have an explanation for it. It is published as a range rather than smoothed into a median, because a single number here would misrepresent what you will actually observe.

Backend choice is worth ~6 % decode and ~25 % prefill

Same weights, same box, same day, only moe_backend changed:

Both are correct W4A4 paths that honour the activation scales; FLASHINFERCUTLASS is faster on this hardware. (The VLLMCUTLASS column was taken at 1 M max-model-len with CUDA graphs on, so it is indicative rather than a controlled one-variable comparison.)

Quality — agentic tool use

69 scenarios covering tool selection, argument construction, multi-step planning, restraint and safety. Temperature 0, concurrency 1, prefix caching off, two runs in one serve session, scenario_count == 69 asserted on every run.

Per category (earned / max):

The noise floor, stated up front

This artifact disagrees with itself. Across its own two runs — identical weights, identical server, same session — four of fifteen categories moved:

That spread is larger than most of the differences between arms in the table above. Treat per-category gaps of 1–2 points as noise unless they repeat.

The reason is not the bench harness. In a controlled study on this stack we found the server is not deterministic at temperature 0: the same request repeated returns different logits, only 20 of 103,752 scored positions are bitwise stable across replicates of one arm, and ~50 % of greedy replicates diverge within the first two tokens.

What we can and cannot claim about W4A4 specifically

We ran a dedicated 4-arm study to test whether 4-bit activations cost anything, using byte-identical weights (109,341 shared tensors verified identical by blake2b) served two ways — once under CUTLASS honouring the activation scales, once under MARLIN silently dropping them.

Every between-arm divergence sat inside the within-arm band. The activation-honouring arm's KL from the activation-dropping arm (0.266) was smaller than its KL from itself (0.308). Paired sign tests on the agentic bench gave p = 0.375 and p = 1.000, and 4 of the 5 discordant scenarios were ones that also flip within a single arm's own two runs.

So: do not read "W4A4 87/88 vs W4A16 81/86" as W4A4 being better. We have refuted that ordering three independent ways. What this artifact is warranted for is the prefill and composition properties in the Performance section, not a quality win.

No BF16 teacher exists

117.6 B parameters in BF16 is 235 GB; a Spark has 128. Nobody — us included — can currently report a KL or perplexity delta against the unquantized model on this hardware. Any claim that this quantization "matches" the original is unsupported, ours included. If you need a fidelity guarantee rather than a task score, measure it on your workload.

Provenance and licensing

• —Base: `poolside/Laguna-S-2.1`, OpenMDW-1.1. LICENSE.md is included here as that license requires, along with the base model's configuration_laguna.py and modeling_laguna.py.
• —Quantizer: NVIDIA TensorRT Model Optimizer (ModelOpt), NVFP4 W4A4 on the routed experts, FP8 W8A8 on attention and the shared expert.
• —Verified before upload: 7/7 shards' declared end equals file size exactly; index resolves both ways; 141 `shared_expert` keys and 0 `shared_mlp` — a sibling copy of this artifact carries the wrong (HY3) names and is quarantined, because serving it leaves 47 shared experts at random init with no error.
• —This variant adds exactly one file — model-kvscales.safetensors, 8,800 B — carrying the 96 transplanted k_scale/v_scale tensors. The index was re-resolved over all 145,537 entries: 96 KV scales, two per attention layer across 48 layers.

Quantized by Jason Wong. No affiliation with poolside or NVIDIA.