JasonW2025/Laguna-S-2.1-ModelOpt-NVFP4-W4A16-vllm

Laguna-S-2.1 — NVIDIA ModelOpt NVFP4 W4A16, 4.66 bpp

A 63.81 GiB NVFP4 weight-only quantization of `poolside/Laguna-S-2.1` (117.6 B params), exported with NVIDIA ModelOpt (20260705.dev19+g67a6b1b4f).

This is the W4A16 sibling of Laguna-S-2.1-ModelOpt-NVFP4-W4A4-vllm. The difference is activation precision, and on GB10 it decides which kernel you get — see Which variant do you want below.

The authoritative per-layer map is hf_quant_config.json in this repo (quant_algo: MIXED_PRECISION), and a full tensor-level dump is in .quant_summary.txt.

Which variant do you want

• —W4A16 (this repo) — weights 4-bit, activations 16-bit. Higher activation fidelity, and it serves anywhere NVFP4 weight-only is supported. On GB10 a weight-only NVFP4 export carries a QuantKey ending xNone, which the fused NVFP4 CUTLASS MoE kernel rejects — it falls back to MARLIN.
• —W4A4 — weights and activations 4-bit on the routed experts, which is what lets the fused NVFP4 MoE kernel run on a DGX Spark.

Pick this repo for fidelity/portability, the W4A4 repo for fused-kernel decode throughput.

Serving (vLLM)

vllm serve JasonW2025/Laguna-S-2.1-ModelOpt-NVFP4-W4A16-vllm \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --no-enable-prefix-caching

configuration_laguna.py / modeling_laguna.py ship in the repo, so --trust-remote-code is required.

Caveats — please read

• —KV scales are declared but not baked. hf_quant_config.json sets kv_cache_quant_algo: FP8, but this export ships no `k_scale`/`v_scale` tensors. vLLM will warn Using KV cache scaling factor 1.0 for fp8_e4m3 and, with --calculate-kv-scales, derive them from the first batch — which is non-deterministic across sessions. For a calibrated KV variant see Laguna-S-2.1-ModelOpt-NVFP4-W4A4-KVcal-vllm.
• —No quality benchmark has been published for this W4A16 arm. The tool-use and agentic numbers quoted on the W4A4 card were measured on that checkpoint and do not transfer here. Treat this repo as unbenchmarked until an arm is run.
• —Built and validated on a DGX Spark (GB10, sm_121a).

License

Inherits the base model's licence (OpenMDW-1.1, poolside/Laguna-S-2.1).