CoolFace
Modelpublic

iamimmanuelraj/Nanbeige4.2-3B-GGUF

sourceHugging Faceotherupdated 2mo agoView on Hugging Face
0likes771downloads
Model Card

Nanbeige4.2-3B — GGUF

GGUF quantizations of **Nanbeige/Nanbeige4.2-3B** — a 3B "Looped Transformer" (22 layers, num_loops=2, hidden 3072, untied 166k vocab, RoPE θ 70M).

33 quants from full-precision F16/BF16 down to 1-bit and ternary, including imatrix-guided IQ quants for best quality-per-byte at the low end.

Nanbeige4.2-3B support is merged into mainline [llama.cpp](https://github.com/ggml-org/llama.cpp) — just build the latest:

bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build -j

🚀 Run

llama.cpp (chat):

bash
./build/bin/llama-cli -hf iamimmanuelraj/Nanbeige4.2-3B-GGUF:Q4_K_M -cnv

or with a local file:

bash
./build/bin/llama-cli -m Nanbeige4.2-3B-Q4_K_M.gguf -cnv

llama-server (OpenAI-compatible API):

bash
./build/bin/llama-server -m Nanbeige4.2-3B-Q4_K_M.gguf -c 8192

Download a single quant:

bash
pip install -U "huggingface_hub[cli]"
hf download iamimmanuelraj/Nanbeige4.2-3B-GGUF Nanbeige4.2-3B-Q4_K_M.gguf --local-dir .

📦 Quants

⭐ = recommended. IQ quants are imatrix-guided (better quality at the same size).

Full precision

QuantSizeNotes
F168.34 GBFull 16-bit reference
BF168.34 GBFull 16-bit (bfloat)

High quality

QuantSizeNotes
⭐ Q8_04.43 GBNear-lossless, best quality
Q6_K3.42 GBExcellent, hard to distinguish from F16
⭐ Q5KM2.99 GBVery high quality, great default for quality
Q5KS2.94 GBVery high quality, slightly smaller
Q5_02.94 GBLegacy 5-bit
Q5_13.17 GBLegacy 5-bit, larger

Balanced (recommended range)

QuantSizeNotes
⭐ Q4KM2.57 GBBest size/quality balance — start here
Q4KS2.50 GBBalanced, slightly smaller
Q4_02.48 GBLegacy 4-bit
Q4_12.71 GBLegacy 4-bit, larger
⭐ IQ4_XS2.38 GBImatrix 4-bit, beats Q4 at smaller size
IQ4_NL2.49 GBImatrix 4-bit non-linear

Small

QuantSizeNotes
Q3KL2.31 GB3-bit, larger/better
Q3KM2.17 GB3-bit balanced
Q3KS2.00 GB3-bit smaller
IQ3_M2.08 GBImatrix 3-bit, good
IQ3_S2.00 GBImatrix 3-bit
IQ3_XS1.93 GBImatrix 3-bit smaller
IQ3_XXS1.78 GBImatrix 3-bit, very small

Very small

QuantSizeNotes
Q2_K1.76 GB2-bit, usable
Q2KS1.68 GB2-bit smaller
Q2_01.60 GBLegacy 2-bit
IQ2_M1.66 GBImatrix 2-bit, best small quality
IQ2_S1.59 GBImatrix 2-bit
IQ2_XS1.46 GBImatrix 2-bit smaller
IQ2_XXS1.37 GBImatrix 2-bit, very small

Extreme / experimental

QuantSizeNotes
IQ1_M1.27 GBImatrix 1-bit, experimental
IQ1_S1.21 GBImatrix 1-bit, most degraded
Q1_00.94 GB1-bit, smallest, heavy quality loss
TQ1_01.37 GBTernary, experimental
TQ2_01.52 GBTernary, experimental

Which should I pick?

  • —Most people → `Q4_K_M` (2.57 GB): best balance of size, speed, quality.
  • —Want max quality → `Q8_0` or Q6_K.
  • —Tight on RAM/VRAM → `IQ4_XS` (2.38 GB) or Q3_K_M (2.17 GB).
  • —Tiny devices → `IQ2_M` (1.66 GB); below that quality drops sharply.
  • —≤ 1.3 GB (`IQ1`/`Q1`/`TQ`): experimental — expect noticeable degradation.

Prefer IQ over the same-size legacy quant when available — imatrix calibration gives better quality per byte.

Notes

  • —IQ quants built with an importance matrix (imatrix) for better low-bit fidelity.
  • —Q8_0 KV cache recommended for this architecture.
  • —License follows the base model's license.

Quantized with [llama.cpp](https://github.com/ggml-org/llama.cpp).