Modelpublic

srv-sngh/Nanbeige4.2-3B-mlx-nvfp4

sourceHugging Faceapache-2.0updated 22d agoView on Hugging Face

0likes281downloads

Model Card

Nanbeige4.2-3B — MLX NVFP4

Nanbeige4.2-3B quantized to NVFP4 (4-bit float, group size 16) in MLX format, for Krill on Apple Silicon.


Base	`Nanbeige/Nanbeige4.2-3B` (bf16, 8.3 GB)
This build	NVFP4, 4-bit / group 16 (2.26 GB)
Quantized tensors	156 (22 layers x 7, plus `embed_tokens` and `lm_head`)
Context	262,144

Quantized with Krill's native Swift+MLX checkpoint quantizer (krill quantize --mode nvfp4) — no Python, no mlx_lm.convert.

⚠️ Runtime support: Krill only

This build runs in [Krill](https://github.com/srvsngh99/Krill) and nowhere else. That is not a packaging choice — as of this writing no other local inference engine implements the Nanbeige looped architecture:

Runtime	Status
Krill ≥ 0.17.0	✅ native Swift+MLX runtime
llama.cpp / Ollama / LM Studio	❌ `unknown model architecture: 'nanbeige'`
mlx-lm (0.31.3)	❌ no `nanbeige` model module
vLLM / SGLang	❌ upstream ships forks, not mainline support
transformers	✅ but only via `trust_remote_code` on the original bf16 repo — it cannot read these nvfp4-packed tensors

GGUF conversions of Nanbeige 4.2 exist on the Hub, but they do not load: llama.cpp rejects the architecture before it reads any weights. If you are not using Krill, use the original `Nanbeige/Nanbeige4.2-3B` with transformers.

Performance

Measured on Apple Silicon, 512-token prompt, same machine. The baseline is the upstream PyTorch reference (modeling_nanbeige.py, fp16 on MPS) because it is the only other runtime that can serve this model at all — an Ollama comparison is impossible, not merely absent.

	Prefill	Decode	Load	Peak memory
Krill 0.17.0, this nvfp4 build	294.6 tok/s	37.5 tok/s	0.15 s	1.7 GB
PyTorch reference, fp16 (MPS)	204.0 tok/s	9.7 tok/s	10.2 s	~8 GB
speed-up	1.4×	3.9×	68×	~4.7× smaller

For scale, a dense qwen2.5-3b 4-bit in the same Krill build does 508.5 tok/s prefill / 49.8 tok/s decode at 1.5 GB. Nanbeige decodes ~25% slower at a comparable footprint — that is the loop being paid for: 44 effective layers out of 22 blocks' worth of weights.

Usage

bash

krill pull srv-sngh/Nanbeige4.2-3B-mlx-nvfp4
krill run Nanbeige4.2-3B-mlx-nvfp4 "Which number is bigger, 9.11 or 9.8?"

Nanbeige 4.2 is a reasoning model: its chat template opens an assistant turn inside a <think> block, so the model spends tokens reasoning before it emits any visible answer. Give it room — a few hundred tokens is often not enough to close the block, and you will see an empty reply. --max-tokens 4000 or more is a sensible floor for anything non-trivial.

Upstream's recommended sampling settings (from generation_config.json): temperature 0.6, topp 0.95, topk 20 for reasoning and chat; temperature 1.0 for agentic and tool-use work.

About the architecture

This is not a plain Llama clone, and the quantized layout reflects that:

—Looped transformer. num_loops: 2 runs the whole 22-block stack twice over the same weights — 44 effective layer executions from 22 blocks' worth of parameters. Each execution keeps its own KV cache slot, so a runtime must allocate num_loops * num_hidden_layers = 44 caches, not 22. The final model.norm is applied at the end of every loop.
—`head_dim` decoupled from `hidden_size`. 48 heads x 128 dims against a 3072-wide residual, so q_proj is 3072 -> 6144 and o_proj 6144 -> 3072. A loader that derives hidden_size / num_attention_heads gets 64 and mis-shapes every attention projection.

The optional research features described in upstream's configuration_nanbeige.py (multi-head hyper-connections with depth attention, concatenated n-gram embeddings, LoopSplit, loop-shared KV) are off in this checkpoint and its weight map contains none of their tensors.

Verification

The native Krill runtime backing this build is logit-parity gated against upstream's own modeling_nanbeige.py (mlx-lm has no nanbeige port), on both a synthetic fp32 fixture and these real weights — prefill and incremental cached decode, matching argmax and cosine > 0.9999. A live smoke gate additionally drives this checkpoint through the whole serving path (prompt → looped prefill → 44-cache decode → reasoning filter → detokenizer), and tool calling is verified end to end.

Reproducing the reference? NanbeigeRotaryEmbedding registers inv_freq with persistent=False, so it is absent from the checkpoint. Under transformers 5.x, from_pretrained materializes the model from checkpoint tensors and that buffer is left zero-initialized on every layer — RoPE then applies no rotation at all and the reference emits positionless gibberish. Recompute inv_freq after loading before trusting any comparison. The shipped config.json also has rope_scaling: null, which transformers 5.x repopulates, tripping upstream's _init_rope on a missing "type" key.

License

Apache 2.0, inherited from the base model. All credit for the model itself goes to the Nanbeige team at BOSS Zhipin.

About this build

Quantized and tested by Sourav Singh for Krill, a Mac-native LLM runtime and coding agent in one Swift + MLX binary. More Apple Silicon builds: huggingface.co/srv-sngh. Notes on building with local models and frontier AI: souravailabs.ai · GitHub · LinkedIn · YouTube