NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield

Nanbeige4.2-3B-NVFP4-FP8-LoopShield

Mixed-precision quantization of Nanbeige/Nanbeige4.2-3B (looped transformer, 22 layers × 2 passes/token): FP8-dynamic on attention, `down_proj`, and the first/last 3 layers' MLP; NVFP4 (weight-only, group-16) on middle-layer `gate/up_proj`. 4.5 GB. format: mixed-precision (compressed-tensors).

The nitty gritty

Naive uniform NVFP4 on this looped architecture loses 8 points of GSM8K strict (quantization error compounds across the two loop passes — consistent with LoopQ, arXiv:2605.16343, the only prior looped-LLM PTQ study, which tested INT only; these are, as far as I can tell, the first FP4-family numbers for a looped LLM). The protection recipe that llama.cpp's quant mixes, Unsloth's ablations, and llmcompressor's own non-uniform example all converge on recovered it completely:

The edge-band ratio follows APEX's ablation (~12.5% of depth per side); the tensor priority (down_proj > attention > gate/up) matches llama.cpp's quant mixes, Unsloth's sensitivity ablations, and llmcompressor's own non-uniform example.

Super-weight verification (2026-07-23): this model's super weight (the single most load-bearing scalar, Apple 2411.07191) sits at layers.1.mlp.down_proj.weight[1252, 6883] (largest weight in its tensor, 11.4x p99.99; drives a 26,752-magnitude activation spike, 1,300x the median). This recipe protects both the weight (all-layer FP8 down_proj) and its production path (L1 gate/up in the FP8 edge band) — verified by direct scan, not assumed. One looped-arch novelty from the scan: the spike is pass-asymmetric (26,752 on loop pass 1 of 2; 1,352 on pass 2).

Quality evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV; champion = my FP8-Dynamic quant)

The trade: reasoning at full parity, best-in-family structured-output reliability, small summarization-coverage cost. If you want maximum quality use my FP8-Dynamic; if you want the smallest artifact that keeps reasoning intact on this architecture, use this one.

Speed (RTX 5090, batch-1, vLLM v0.25.1, 64k ctx, fp8 KV)

Decode tok/s, single stream, per-workload best speculative config:

Two workload anchors, not an ISL sweep; decode speed shifts with context length, batch size, and attention backend. The mid-MLP NVFP4 weights serve via the Marlin W4A16 kernel; the FP8 tensors via the FP8 path; the small freeform edge over FP8-Dynamic comes from the lighter weight reads, the small summarize deficit from mixed-kernel overhead under the ngram verify batch.

Serving (vLLM)

Arch not yet upstream (vLLM PR #49433); install the bundled plugin first:

pip install --no-deps ./vllm_plugin
vllm serve <this-repo> --trust-remote-code \
  --max-model-len 65536 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml

Thinking/sampling notes as in the base model: thinking ON by default (chat_template_kwargs.enable_thinking=false to disable), T=0.6/topp=.95/topk=20 defaults ship in generation_config.json, T=1.0 for agentic use.

Download just this artifact:

hf download NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield --local-dir Nanbeige4.2-3B-NVFP4-FP8-LoopShield

Creation (llmcompressor, data-free)

g_fp8 = dict(FP8_DYNAMIC)
g_fp8["targets"] = ["re:.*self_attn\\.q_proj.*", "re:.*self_attn\\.k_proj.*",
                    "re:.*self_attn\\.v_proj.*", "re:.*self_attn\\.o_proj.*",
                    "re:.*down_proj.*"] + \
                   [f"re:model\\.layers\\.{i}\\.mlp\\.(gate|up)_proj.*" for i in (0,1,2,19,20,21)]
g_fp4 = dict(NVFP4A16)
g_fp4["targets"] = [f"re:model\\.layers\\.{i}\\.mlp\\.(gate|up)_proj.*" for i in range(3,19)]
QuantizationModifier(config_groups={"group_0": g_fp8, "group_1": g_fp4}, ignore=["lm_head"])

GPTQ/AWQ were NOT usable on this architecture in llmcompressor 0.12: GPTQ's Hessian inversion fails on all 154 modules (root cause unknown — the double-fire hook was checked and is not the cause); AWQ lacks arch mappings. Reported upstream: #2952, #2953.

Limitations

• —vLLM-only until PR #49433 merges. Bundled modeling_nanbeige.py carries two one-line transformers-5 compat patches (rope key, tied-weights type).
• —W4A4 config-variant possible (same bytes) but NOT recommended: batch-1 decode is faster via the Marlin W4A16 path on consumer Blackwell, and activation quantization is the looped arch's worst failure mode.
• —English-only evals; n=100/46 — directional, not leaderboard-grade.

Which artifact should I choose?

All three serve identically (same plugin, same flags); only the checkpoint differs.

Comparison chain: the columns here use my FP8-Dynamic quant as reference; the FP8 card carries the bf16-original matrix linking the chain back to the unquantized model.

Links & provenance

• —Base model: Nanbeige/Nanbeige4.2-3B
• —Family: FP8-Dynamic · NVFP4-FP8-LoopShield · NVFP4A16 · EAGLE3 draft
• —Arch serving support: vLLM PR #49433 (until merged, the bundled vllm_plugin/ registers it out-of-tree)
• —Upstream tooling reports I filed from this work: llm-compressor#2952 (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · llm-compressor#2953 (AWQ lacks arch mappings)
• —Quantized with llm-compressor 0.12.0 (compressed-tensors format)

Benchmark provenance (ordered: public-harness first, then my closed harnesses)

Public, reproducible (lm-eval-harness local-chat-completions, exact configs in each row's annotation):

1. 1.GSM8K — grade-school math, the reasoning gate (thinking mode, n=100, max_tokens 8192)
2. 2.IFEval — verifiable instruction following (non-thinking, n=250)
3. 3.MMLU-Pro — 10-choice knowledge/reasoning (non-thinking, 25/category)
4. 4.BBH — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask)
5. 5.MultiHop-RAG — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public)

Closed/personal harnesses (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores):

• —Blind-judge summarization — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality?
• —Rewrite taxonomy — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing.
• —JSON parse rate — structured-output emission over the summarization set. Tests: format discipline under quantization.

Citation

@misc{peciukonis2026nanbeige42loopshield,
  author       = {Pe{\v{c}}iukonis, Matas (NullSense)},
  title        = {NVFP4-FP8-LoopShield: loop-aware mixed-precision NVFP4 quantization of Nanbeige4.2-3B},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield},
  note         = {Placement recipe recovering GSM8K 81->89 on a looped/weight-shared LLM; first FP4-family results for the architecture class.}
}