tinyopsec/Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B — GGUF Quantizations

Community-provided GGUF quantizations of Nanbeige/Nanbeige4.2-3B, a compact agentic LLM developed by Nanbeige LLM Lab.

All weight files were converted from the original BF16 SafeTensors checkpoint using the Nanbeige fork of llama.cpp (-b nanbeige42). No fine-tuning or architectural changes were applied.

Model Overview

Nanbeige4.2-3B is a supervised fine-tuned + RL-trained agentic model built on Nanbeige4.2-3B-Base. Its Looped Transformer architecture reuses transformer layers to increase effective model capacity without increasing parameter count. At 3B non-embedding parameters it outperforms models such as Qwen3.5-9B and Gemma4-12B on several agentic and reasoning benchmarks.

Available Quantizations

Recommendation: Q4_K_M is the best starting point for most deployments. Use Q6_K or Q8_0 if you have sufficient RAM and need maximum fidelity.

Quickstart

llama.cpp

Install (macOS / Linux)

curl -LsSf https://llama.app/install.sh | sh

Install (Windows via WinGet)

winget install llama.cpp

Run a local OpenAI-compatible server

llama serve -hf tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

Run inference in the terminal

llama cli -hf tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

Ollama

ollama run hf.co/tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

Docker

docker model run hf.co/tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

LM Studio

Download any GGUF file from the Files tab and open it directly in LM Studio.

Note: The bundled llama.cpp inside LM Studio does not yet support the nanbeige architecture. You must replace the LM Studio backend binaries with the Nanbeige fork: 1. Build llama.cpp from the `nanbeige42` branch 2. Copy build/bin/* into your LM Studio backend directory (e.g. ~/.lmstudio/extensions/backends/llama.cpp-<version>)

Inference Parameters

The following settings are recommended by the original authors:

The chat template exposes two configuration flags:

• —enable_thinking — enables chain-of-thought reasoning (on by default; set to false for non-thinking mode)
• —preserve_thinking — controls whether reasoning tokens from previous turns are retained in multi-turn conversations (recommended true for tool-use and code-agent workflows; false for general chat)

For tool-calling, tool_call_format="xml" yields the best performance; json is supported for compatibility.

Benchmark Summary

Results reported by the original authors (thinking mode, preserve_thinking=true):

Full benchmark tables and methodology are available in the original model card and technical report.

Conversion Notes

GGUF files were produced using the Nanbeige fork of llama.cpp. The standard upstream llama.cpp (ggerganov/llama.cpp) does not support the nanbeige architecture at the time of publication; use the forked build for any operations that require re-quantizing or converting these files.

# Clone the Nanbeige fork
git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# BF16 → GGUF (reference step)
python3 convert_hf_to_gguf.py /path/to/Nanbeige4.2-3B \
  --outfile Nanbeige4.2-3B-BF16.gguf \
  --outtype bf16

# Re-quantize to Q4_K_M
./build/bin/llama-quantize \
  Nanbeige4.2-3B-BF16.gguf \
  Nanbeige4.2-3B-Q4_K_M.gguf \
  Q4_K_M

Limitations

As noted by the original authors: the model may produce inaccurate, biased, or otherwise inappropriate outputs due to its probabilistic nature. These GGUF files are unmodified quantizations of the original checkpoint and inherit all limitations documented in the original model card.

License

The original model weights are released under the Apache 2.0 license by Nanbeige LLM Lab. This repository provides only format-converted quantizations. The Apache 2.0 license terms apply; see LICENSE in the original repository.

Citation

If you use this model, please cite the original work:

@article{lab2026nanbeige4,
  title={Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model},
  author={Lab, Nanbeige and Yang, Chen and Huang, Chengrui and Lan, Fufeng and others},
  journal={arXiv preprint arXiv:2607.22083},
  year={2026}
}

Links

• —Original model — Nanbeige/Nanbeige4.2-3B
• —Base model — Nanbeige/Nanbeige4.2-3B-Base
• —Technical report — arXiv:2607.22083
• —Nanbeige llama.cpp fork