jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality

Laguna-S-2.1-NVFP4-W4A16-Quality

Quality-first mixed-precision ModelOpt quantization of poolside/Laguna-S-2.1, exported directly from Poolside's BF16 checkpoint. This is a serving-oriented checkpoint, not a smallest-possible or single-GPU quant.

The guiding rule was to leave the model's signal path intact and quantize only the routed experts. All attention projections, shared experts, MoE routers, embeddings, dense layer 0, and the LM head remain BF16. In particular, no shared expert in layer 47 was quantized.

Terminal-Bench 2.1 with Poolside's PoolCLI harness is running now. Results will be added when the matched BF16, Poolside NVFP4, and this checkpoint campaigns finish. The KLD results below are complete five-run gates; they are not a substitute for downstream benchmark results.

Precision placement

• —39 routed-expert layers use W4A16 NVFP4.
• —3 routed-expert layers use FP8.
• —5 routed-expert layers remain BF16.
• —Effective routed-expert weight precision: 5.9468 bits.
• —Audited export: 92,469,054,276 bytes, 10 shards, 101,377 tensors.
• —ModelOpt producer: 0.46.0.dev184+g8813b7001.

Placement was selected with KL-guided AutoQuantize over 512 samples from nemotron-post-training-v3, 512 tokens per sample. The chosen placement was then rebuilt directly from BF16 using FP8 scale sweep and static MSE-scaled W4A16 NVFP4. The exact replay recipe is included as quantization_recipe.yaml.

Five-run KLD

KL direction is KL(BF16 reference || candidate). Every row uses the same hash-pinned 2,048-token prompt, 2,047 next-token positions, and 100,352 logits per position. Runs used TP4, BF16 compute, eager execution, seed 0, no prefix cache, and a fresh engine per run. Quant candidates used the same B12X A16 expert path. Lower is better.

Against the pinned Poolside NVFP4 checkpoint, this export reduced mean KLD by 29.97% with BF16 KV and 20.15% with matched dynamic FP8 KV. KLD values are specific to this model and protocol and should not be compared directly to numbers from other model families.

Reference provenance:

• —BF16 weights: poolside/Laguna-S-2.1@a50e85e7e0aae7b0a504d156bd36a616ec9fea38
• —Poolside NVFP4 comparison: poolside/Laguna-S-2.1-NVFP4@216d1f13878dd4e715bc7412848d0f330e95bba6
• —Prompt token IDs SHA-256: 1f4a417c5159d1ac1202e38c0cb4573955d8ceb2037d8e316b7c3344267959d6
• —BF16 reference logits SHA-256: beaf167ad42de75af7a0f33554f37c2672092eed175ccaa76cb726e684bca109
• —Quant config SHA-256: 2102d1290212d218211efbaf0e6b670666a7ea47a11bf255c4980e021899f6af
• —Safetensors index SHA-256: 0a12b7fa0676018cefe4f4c518f60ee6550d94388273bc7e4cfef027604ae68a

The five aggregate JSON files, placement audit, and checkpoint audit are shipped in this repository.

Serving

This checkpoint is intended for Blackwell GPUs and a W4A16-capable vLLM build. Start with BF16 KV when evaluating quality, then test FP8 KV separately for capacity.

vllm serve jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization auto \
  --tensor-parallel-size 2 \
  --kv-cache-dtype bfloat16

The repository includes Poolside's updated tokenizer from BF16 revision 179ee67cf0fff5391c67fe1a392ea849fa6d643f: thinking is enabled by default, reasoning is preserved across turns, </assistant> is special, and the old 32K generation cap is removed.

Status and limitations

• —Terminal-Bench 2.1 is in progress using PoolCLI 1.0.13, four attempts per task, and six-hour agent timeouts.
• —Estonia and LAVD smoke characterization indicated that this release family is sensitive to reasoning behavior; those observations motivated the matched PoolCLI benchmark rather than ad-hoc prompt scoring.
• —This is an early public release. Please report loops, tool-call failures, and runtime details with the exact KV-cache dtype and serving commit.
• —Use is governed by the upstream OpenMDW 1.1 license.

Quantized by Jack Zampolin with the local-inference community's quality-first serving workflow.