julianmb/Nex-N2.5-mini-ROCmFP4-GGUF

Nex-N2.5-mini ROCmFP4

[!IMPORTANT] Custom Engine Required — Incompatible with Stock Ollama / Vanilla llama.cpp This repository provides custom ROCmFP4 quantized weights (Q4_0_ROCMFP4_STRIX_LEAN using custom GGML tensor types 100 & 101, file type 106) engineered specifically for AMD Strix Halo (`gfx1151`) and RDNA 3.5 architectures. - Engine Requirement: Requires [ROCmFPX](https://github.com/charlie12345/ROCmFPX) or [halofpx](https://github.com/julianmb/halofpx) to run. - Stock Ollama / llama.cpp Incompatibility: Stock llama.cpp and vanilla ollama will fail to load these weights (unknown tensor type 101 and unsupported qwen35moe Gated DeltaNet architecture). - Standard Quants: If you need standard vanilla GGUF quants (Q4KM, etc.) for general llama.cpp usage, please use abenzerps/Nex-N2.5-mini-GGUF.

ROCmFP4 (Q4_0_ROCMFP4_STRIX_LEAN) quantization of Nex-N2.5-mini for AMD Strix Halo (`gfx1151`) and RDNA 3.5 GPUs, engineered using ROCmFPX.

Nex-N2.5-mini is an open-source agentic multimodal MoE model built by Nex AGI on the Qwen3.5-35B-A3B architecture, unifying requirement understanding, code generation, tool use, and environment execution through an Agentic Thinking adaptive reasoning loop.

Quantization Details

Measured Performance (AMD Ryzen AI Max+ 395, Radeon 8060S, Mesa RADV Wave64)

⚡ Why ROCmFP4 is Faster Than Standard Quants (Q4KM) on AMD Strix Halo

Standard GGUF integer quants (such as Q4_K_M) were designed primarily for CPU cache architectures and CUDA tensor cores. On AMD Strix Halo APUs (gfx1151) and RDNA 3.5 architectures, `ROCmFP4_STRIX_LEAN` achieves both higher decode throughput and smaller footprint for four key architectural reasons:

1. Direct Cooperative Matrix Mapping (KHR_coopmat / Mesa RADV Wave64)

• —The Problem with Q4_K: Standard Q4_K blocks are non-uniform: 256-element blocks split into 8 sub-blocks of 32 elements with dual 6-bit scales and 6-bit offsets. Compute units must pay a complex, multi-pass unpack and ALU dequantization penalty in vector registers before data can feed matrix multiply units.
• —The ROCmFP4 Solution: ROCmFP4 formats (Q4_0_ROCMFP4 and Q4_0_ROCMFP4_FAST) use single-scale uniform FP4 quantization per 32 elements. In shader registers, unpacking is reduced to single-cycle bit shifts and direct table lookups. This dramatically reduces register pressure and instruction count, allowing Mesa RADV's Wave64 cooperative matrix pipelines to run near theoretical hardware saturation.

2. Lower Memory Bus Pressure Across Unified Memory (UMA)

• —Strix Halo shares ~200 GB/s of LPDDR5X unified memory bandwidth between 16 Zen 5 CPU cores and the 40 CU GPU.
• —In Mixture-of-Experts (MoE) generation, token throughput is strictly gated by how fast active expert weights stream across the memory bus on every step.
• —At 17.32 GiB (4.29 BPW) vs Q4KM's 19.71 GiB (4.88 BPW), ROCmFP4 transfers 12.1% fewer bytes across UMA per forward pass. That bandwidth saving directly translates into sustained token throughput (76.92 tok/s vs 72.76 tok/s on Vulkan; 68.62 vs 60.39 tok/s on ROCm).

3. MoE-Aware "Strix Lean" Mixed-Precision Policy

Rather than naively crushing all tensors to 4-bit, the STRIX_LEAN recipe selectively preserves precision where accuracy matters most:

• —Router Gate Logits (`ffn_gate_inp.weight`) & LayerNorms: Maintained in uncompressed FP32. Expert routing decisions and token assignments remain bit-exact, preventing expert collapse.
• —Token Embeddings (`token_embd.weight`): Quantized in higher-precision Q5_K to maintain vocabulary entropy and prevent prompt degradation.
• —Attention Projections: Preserved in dual-scale q4_0_rocmfp4 for clean KV heads.
• —Bulk Expert Feed-Forward Networks: Packed into high-speed q4_0_rocmfp4_fast for maximum memory streaming bandwidth.

4. Synergy with Hybrid Gated DeltaNet Linear Attention

• —Nex N2.5 Mini uses 30 linear-attention layers (DeltaNet / recurrent SSM state) interleaved with 10 full-attention layers.
• —Recurrent state updates require continuous cache access; smaller weight memory footprint prevents cache eviction and memory bus contention between linear-attention states and MoE routing.

Serving

HaloFPX (Recommended — Text + Vision)

halofpx pull downloads and verifies both the ROCmFP4 weights and vision projector:

halofpx pull nex-n2.5-mini
halofpx serve -m nex-n2.5-mini

Direct llama-server (ROCmFPX fork)

llama-server   -m Nex-N2.5-mini-ROCmFP4-STRIX_LEAN.gguf   --mmproj mmproj-Nex-N2.5-mini.gguf   -ngl 99 -c 32768 -fa on   --host 0.0.0.0 --port 8080

Note on Reasoning Mode: To enable thinking mode over the /v1/chat/completions API, request with --reasoning-format deepseek or pass chat_template_kwargs: {"enable_thinking": true}. To disable thinking for lower latency, pass chat_template_kwargs: {"enable_thinking": false}.