NullSense/Nanbeige4.2-3B-FP8-Dynamic

Nanbeige4.2-3B-FP8-Dynamic

FP8 W8A8 (dynamic per-token activations) quantization of Nanbeige/Nanbeige4.2-3B — the looped transformer (22 layers × num_loops: 2). 4.9 GB (from 7.9), +60% batch-1 decode, no measurable quality loss on any gate I ran. This is my recommended serving artifact for this model.

Why FP8 and not 4-bit for this architecture

The looped architecture executes every layer twice per token, which compounds quantization error twice per token. In my gate battery (below), uniform NVFP4 lost 8 points of GSM8K strict; FP8-dynamic lost nothing — its runtime per-token activation scales adapt to each loop pass's distinct activation distribution. As far as I can tell these are the first published FP4/FP8 quantization results for a looped LLM (the only prior looped-LLM PTQ study, LoopQ arXiv:2605.16343, tested INT formats only).

Quality evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV)

Public harnesses first (lm-eval, reproducible), then my closed harnesses (relative signals within this family). Speed has its own section below.

Judge protocol: blind single-judge, per-article anonymized+shuffled candidates, strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity).

Speed (RTX 5090, batch-1, vLLM v0.25.1, 64k ctx, fp8 KV)

Decode tok/s, single stream, per-workload best speculative config:

These are two workload anchors, not an ISL sweep; decode speed shifts with context length, batch size, and backend. Notes from the serving campaign:

• —The looped arch reads every weight twice per token, so weight-compression speedups roughly double vs a normal 3B (+60% batch-1 here).
• —ngram speculation is workload-dependent: +40-60% on copy-heavy work (summarization/RAG), −15-20% on freeform. The rows above use each workload's best config.
• —Attention backend is context-dependent: FlashInfer (default) wins at 2k+ ctx; VLLM_ATTENTION_BACKEND=TRITON_ATTN lifted spec-off short-form decode a further +13% in my follow-up runs.

Performance under load (GuideLLM, real articles, eagle3 head)

Concurrency sweep of the production config (FP8 + EAGLE-3 draft, TRITON_ATTN, k=3) on real article prompts. Aggregate throughput scales with concurrency; inter-token latency stays essentially flat under load.

Measured with GuideLLM on an RTX 5090, vLLM v0.25.1, real multi-kB article prompts, EAGLE-3 v2 (thinking-aware) head. The server saturates to ~1000 tok/s aggregate at concurrency 16 with per-token latency near-flat (6.2 to 7.6 ms), so it batches well for multi-user / parallel-agent workloads. TTFT p50 is queueing-dominated at these request counts, not a decode signal. Spec acceptance held over the run (154k accepted / 354k drafted, 0.44).

Serving (vLLM)

The architecture is not yet in upstream vLLM (PR #49433 open). Until it merges, install the bundled out-of-tree plugin (vendored from that PR), then serve normally — the checkpoint format is auto-detected:

pip install --no-deps ./vllm_plugin
VLLM_ATTENTION_BACKEND=TRITON_ATTN \
vllm serve <this-repo> --trust-remote-code \
  --max-model-len 65536 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"eagle3","model":"NullSense/Nanbeige4.2-3B-EAGLE3","num_speculative_tokens":3}'

Since 2026-07-23 the recommended speculation is my EAGLE-3 draft head, thinking-aware since the 2026-07-24 retrain: thinking-mode acceptance 0.41 (0.33 before), and +12-41% single-stream decode over the prior head depending on context, in one config (TRITONATTN required with it; FlashInfer + spec drops CUDA graphs to piecewise on SM120). One measured exception: LONG-document summarization (multi-k-char inputs) still favors the ngram config (`{"method":"ngram","numspeculativetokens":8, "promptlookupmax":4,"promptlookup_min":2}`): 231 vs 160 output tok/s on 16 real articles; verbatim copy-spans reward lookup drafting. Pick per dominant workload.

• —Thinking ON by default; pass chat_template_kwargs: {"enable_thinking": false} (+ preserve_thinking: false for chat) to disable. Sampling defaults (T=0.6/topp=0.95/topk=20) ship in generation_config.json; use T=1.0 for agentic.
• —ngram speculation: +40-60% on summarization/RAG, −15-20% on freeform — drop the --speculative-config for freeform-heavy serving.
• —--kv-cache-dtype nvfp4 is SM100-only; use fp8 on consumer Blackwell.

Download just this artifact:

hf download NullSense/Nanbeige4.2-3B-FP8-Dynamic --local-dir Nanbeige4.2-3B-FP8-Dynamic

Creation

llmcompressor QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]), data-free oneshot(). transformers ≥5 users: the bundled modeling_nanbeige.py includes two one-line compat patches vs the original repo (ropescaling `"type"` key, `tiedweightskeys` list→dict) — required for any transformers-5 loading (llmcompressor, finetuning); vLLM serving does not use them.

Limitations

• —vLLM-only until PR #49433 merges (transformers inference works but is unoptimized for the looped arch).
• —English+Chinese model; all my evals are English-only.
• —Evals are n=100 (GSM8K) / n≈44 (judged summarization) — directional, not leaderboard-grade.

Which artifact should I choose?

All three serve identically (same plugin, same flags); only the checkpoint differs. Rule of thumb: this model + 64k fp8-KV context fits in ~7 GB VRAM at FP8, ~6 GB at NVFP4A16; any 8 GB card runs the full family.

Full quality matrix vs the bf16 original

Where bf16 was measured, FP8 shows no regression; that is why FP8 serves as the reference for the rest of the family instead of re-running every gate on bf16. The vendor's published numbers are a different suite entirely (agentic/frontier benches, thinking mode at a 131k token budget) and don't map onto these rows — see Benchmark provenance below.

Links & provenance

• —Base model: Nanbeige/Nanbeige4.2-3B
• —Family: FP8-Dynamic · NVFP4-FP8-LoopShield · NVFP4A16 · EAGLE3 draft
• —Arch serving support: vLLM PR #49433 (until merged, the bundled vllm_plugin/ registers it out-of-tree)
• —Upstream tooling reports I filed from this work: llm-compressor#2952 (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · llm-compressor#2953 (AWQ lacks arch mappings)
• —Quantized with llm-compressor 0.12.0 (compressed-tensors format)

Benchmark provenance (ordered: public-harness first, then my closed harnesses)

Public, reproducible (lm-eval-harness local-chat-completions, exact configs in each row's annotation):

1. 1.GSM8K — grade-school math, the reasoning gate (thinking mode, n=100, max_tokens 8192)
2. 2.GPQA-diamond — graduate-level science QA (gpqadiamondcot_zeroshot, thinking mode, 32k generation budget, full n=198, seed 1234, mean generation ~12.5k tokens, zero budget truncation; flexible-extract because strict-match's answer regex does not fit thinking output; served with the lossless EAGLE-3 head, which does not affect outputs)
3. 3.IFEval — verifiable instruction following (non-thinking, n=250)
4. 4.MMLU-Pro — 10-choice knowledge/reasoning (non-thinking, 25/category)
5. 5.BBH — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask)
6. 6.MultiHop-RAG — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public)

Closed/personal harnesses (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores):

• —Blind-judge summarization — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality?
• —Rewrite taxonomy — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing.
• —JSON parse rate — structured-output emission over the summarization set. Tests: format discipline under quantization.

Citation

@misc{peciukonis2026nanbeige42fp8,
  author       = {Pe{\v{c}}iukonis, Matas (NullSense)},
  title        = {Nanbeige4.2-3B quantization family: loop-aware FP8 and NVFP4 quants of a looped transformer},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic},
  note         = {First published classic-bench (GSM8K/IFEval/MMLU-Pro/BBH) numbers for this model; no measured regression vs bf16 on any gate run.}
}