Frosty40/Nex-N2-mini-B70-Turbo-GGUF

<p align="center"> <img src="https://huggingface.co/Frosty40/Nex-N2-mini-B70-Turbo-GGUF/resolve/main/cover.jpg" alt="NexN2 B70 Turbo" width="440"> </p>

NexN2 B70 Turbo — Nex-N2-mini GGUF

GGUF + importance-matrix quantizations of [Nex-N2-mini](https://huggingface.co/nex-agi/Nex-N2-mini) — the Qwen3.5-35B-A3B MoE (~3B active, multimodal reasoning) by the [Nex-AGI](https://nex-agi.com) team — tuned and measured for fast local inference on the Intel Arc Pro B70 (Battlemage) via llama.cpp's SYCL backend.

The model is [Nex-N2-mini](https://huggingface.co/nex-agi/Nex-N2-mini), built by the [Nex-AGI](https://nex-agi.com) team — the reasoning, multimodal capability, and training are entirely theirs; this is an independent community quantization that only repackages their model to run fast on Intel Arc, with sincere thanks to Nex-AGI for releasing it openly. The base model is © Nex-AGI (Apache-2.0), post-trained on Qwen3.5-35B-A3B-Base © Qwen Team (Apache-2.0); these quants are redistributed under Apache-2.0 and the names identify the upstream work.

Code, kernel patch, full methodology, and reproducible benchmarks: https://github.com/newjordan/NexN2-B70-Turbo

Files

Recommendation: `Q5_K_M` (best accuracy under the Q6K reference, near-fastest) or **`Q4KM`** for max speed. On SYCL, Q4K / Q5K carry the optimized kernels and lead the decode column — they're the speed picks. Accuracy is KL-divergence + top-1 agreement vs a Q6K reference (PPL 6.572, wikitext-2, 100 chunks). For vision, add mmproj-f16.gguf.

Benchmarks (Intel Arc Pro B70, real measurements)

With the project's Turbo package on Arc Pro B70:

Q5KM/Q4KM decode updated 2026-06-12 with the Turbo patch chain's Q6K MoE reorder (`patches/0003` in the GitHub repo). `patches/0004` adds default-on exact NX2 fused gate/up SwiGLU, post-down weighted-sum fusion, and fused F32 MoE tail add (Q5KM ctx0 reaches 88.1 t/s in the retained default; Q4KM reaches 93.5 t/s on the release-card evidence). The 131k Q5KM table entry is a full-package measurement, not a fused-MoE-only attribution: the audited release gate measured `0004` at 42.0952 -> 42.3148 t/s (+0.52%) for Q5K_M 131k. The large 20.0 -> 42.3 t/s 131k change is the overall Turbo deployment comparison against the fresh stock control.

Broader generic gate/up fusion experiments remain opt-in profiling modes. Shared-expert dense gate/up, activation-Q8 cache, gate/up XOR reduction, vec4 weighted-sum, local-weight weighted-sum, down-weighted-sum, atomic down-weighted-sum, SwiGLU activation variants, gate/up rowpack scheduling, gate/up local-Q8 activation caching, gate/up expert-pack scheduling, gate/up dual-dot vecdot, weighted-tail, dispatch guard, gate/up Q8 handoff, exact Q6 down specialization, shared-gate-tail, shared-gate-tail local-gate broadcast, shared-gate-sigmoid-tail, tail-add vec4, tail-add+RMSNORM, RMSNORM+MUL, post-norm Q8 handoff, and selective post-norm Q8 handoff probes are also opt-in only after failing to beat the retained default.

Accuracy guardrail for the retained kernel path: Q5KM 30-chunk WikiText PPL showed no measured loss in the fresh release-gate run (5.5682 +/- 0.15248 baseline vs 5.5676 +/- 0.15244 candidate), and targeted MUL_MAT_ID backend ops passed 690/690.

Run it (llama.cpp)

# build llama.cpp with the SYCL backend (oneAPI/icpx); see the GitHub repo for the Turbo kernel patch
llama-server -m Nex-N2-mini-B70-Turbo-Q5_K_M.gguf -ngl 99 -fa on -c 131072 \
  -ctk f16 -ctv f16 --jinja --host 127.0.0.1 --port 8090

OpenAI-compatible endpoint at http://127.0.0.1:8090/v1. Nex-N2-mini is a reasoning model — it emits a <think> trace, so give it generous max_tokens. Native recommended sampling (per Nex-AGI): temperature 0.7, top_p 0.95, top_k 40.

Reasoning channel: if the <think> trace shows up inline in the answer instead of a separate reasoning channel, your llama.cpp needs the forced-open <think> parser fix — Nex-N2-mini's template force-opens a bare <think>, and without the fix the server leaves the whole trace in message.content. Apply `patches/chat-forced-open-reasoning-leak.patch` (git apply) and rebuild.

Provenance & changes (Apache-2.0 §4(b))

• —Base model: nex-agi/Nex-N2-mini (Apache-2.0), post-trained on Qwen/Qwen3.5-35B-A3B-Base (Apache-2.0).
• —Changes from the original: converted to GGUF; quantized with a llama.cpp importance matrix (imatrix calibrated on Bartowski calibration_datav3); set GGUF metadata qwen35moe.block_count=40 and qwen35moe.nextn_predict_layers=0 so the model loads in llama.cpp (the MTP/NextN head is speculative-only and absent from the checkpoint — lossless for standard inference).
• —The weights are quantizations of the original; no other modifications.

License & attribution

⭐ Built on [Nex-N2-mini](https://huggingface.co/nex-agi/Nex-N2-mini) by the [Nex-AGI](https://nex-agi.com) team — their model, their intelligence; this project only quantizes it for Intel Arc. → nex-agi.com · GitHub · HuggingFace

Released under the Apache License 2.0, inherited from the base model. Retain the attribution above. See `NOTICE` for the full attribution chain (Qwen → Nex-AGI → these quants; plus llama.cpp / MIT).

@misc{qwen3.5,
  title  = {{Qwen3.5}: Towards Native Multimodal Agents},
  author = {{Qwen Team}},
  month  = {February},
  year   = {2026},
  url    = {https://qwen.ai/blog?id=qwen3.5}
}

Please also credit Nex-AGI's Nex-N2-mini (https://github.com/nex-agi/Nex-N2) as the base model.