vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

K2-Horizon-MoVA-36B-A4B — Apex Quant GGUF Models

# 🛑 READ BEFORE YOU RUN THIS MODEL *The "Use this model" quick-start buttons above (llama serve -hf ..., winget, Docker, Ollama, etc.) point to vanilla/upstream llama.cpp. They will NOT work correctly with these files. Upstream llama.cpp has no support for the `k2-horizon` architecture yet, and even the community fork that adds support has two known bugs. If you use the auto-generated buttons above, you will get one of: - a load failure (`Failed to process regex ... regex_error`) on Windows, or - garbage/repeated output (`> > > ...`) on any OS. You must build the patched fork described in [Building the Runtime](#️-building-the-runtime-patched-llamacpp) below.* There is currently no other correct way to run this GGUF with llama.cpp.

This repository contains 3 quantized GGUF profiles of the K2-Horizon-MoVA-36B-A4B model, produced using Apex-Quant technology.

📦 Model Profile Summary

BPW = Bits Per Weight. Higher value = better quality. Note: unlike most model families, here i-balanced is larger than i-quality — it keeps all routed experts at Q5K or higher, while i-quality compresses mid-layer experts to IQ4XS. Choose based on your memory budget vs. precision preference.

📁 File Structure

models/
├── K2-Horizon-36B-i-quality.gguf    # 23.8 GB — highest quality
├── K2-Horizon-36B-i-balanced.gguf   # 26.2 GB — max expert precision
└── K2-Horizon-36B-i-compact.gguf    # 17.6 GB — compact

🔗 Source Model

These models were created based on the K2-Horizon-MoVA-36B-A4B model.

• —Model Page: https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
• —Architecture: K2Horizon MoE with MoVA (Mixture-of-Value Attention)
• —Total Parameters: ~36B (~4B active per token)
• —Context Length: 524,288 tokens (training context; usable range depends on KV-cache RAM)
• —Reference Implementation: vLLM (K2HorizonForCausalLM)
• —Official llama.cpp fork: https://github.com/MBZUAI-IFM/llama.cpp

🛠️ Technology

These quantized models were produced using Apex-Quant technology.

• —Apex-Quant: MoE-aware mixed-precision quantization (layer-band importance: edge / near / mid)
• —Infrastructure: llama.cpp (llama-quantize, --tensor-type-file)
• —Config Generator: apex-quant/scripts/generate_config.sh --profile <profile> --layers 48

⚙️ Building the Runtime (patched llama.cpp)

Why a patched fork is required

The K2-Horizon support fork (MBZUAI-IFM/llama.cpp) provides the base K2-Horizon support. On top of the pinned commit below, two fixes are required for reliable operation (both verified empirically in our test setup):

Both fixes are provided as ready-to-apply patches below.

Step-by-step

# 1. Clone the official K2-Horizon fork and pin the exact commit these models were validated against
git clone https://github.com/MBZUAI-IFM/llama.cpp llama-cpp-k2horizon
cd llama-cpp-k2horizon
git checkout 35999d101cf2233fc54f09c3c8d599da7303ce02

# 2. Apply the two patches (from this repo's patches/ directory)
git apply /path/to/patches/0001-fix-k2-horizon-msvc-regex.patch
git apply /path/to/patches/0002-fix-k2-horizon-parallel-norm.patch

Patch 1 — MSVC tokenizer regex fix (src/llama-vocab.cpp)

MSVC's std::regex rejects \uXXXX escape sequences, so loading any K2-Horizon model crashes on Windows before inference even starts. The zero-width joiner/non-joiner alternatives are irrelevant for BPE pre-tokenization quality and are removed:

             case LLAMA_VOCAB_PRE_TYPE_K2_HORIZON:
                 regex_exprs = {
-                    "...(?:\\p{L}|\\p{M}|\\u200C|\\u200D)+...",
+                    "...(?:\\p{L}|\\p{M})+...",
                 };

(Full, exact diff: [`patches/0001-fix-k2-horizon-msvc-regex.patch`](patches/0001-fix-k2-horizon-msvc-regex.patch))

Patch 2 — Parallel norm residual fix (src/models/k2-horizon.cpp)

This patch rewires the decoder residual path to follow the vLLM reference semantics (K2HorizonRMSNorm.forward(x, residual)), where the running sum of sublayer outputs is tracked across layers and folded in before normalization.

Status note: With this patch applied, the repeated-token (> > >) failure mode and the graph-reservation crash disappeared and generation became coherent. The exact causal mechanism has not been independently verified against the upstream weights — treat this patch as a validated fix rather than a proven diagnosis.

The new residual flow tracks an accumulated residual tensor across layers:

ggml_tensor * residual = nullptr;          // outside the layer loop

for (int il = 0; il < n_layer; ++il) {
    // --- parallel norm BEFORE attention ---
    if (il == 0) {
        residual = inpL;                   // layer 0: residual = embedding
        cur = group_rms_norm(inpL, attn_norm);
    } else {
        cur      = ggml_add(ctx0, inpL, residual);  // prev MLP out + accumulated sum
        residual = cur;
        cur      = group_rms_norm(cur, attn_norm);
    }
    /* ...attention... */

    // --- parallel norm BEFORE FFN ---
    cur      = ggml_add(ctx0, cur, residual);       // attn out + accumulated sum
    residual = cur;
    cur      = group_rms_norm(cur, ffn_norm);
    /* ...MoE FFN... */

    cur  = build_cvec(cur, il);
    inpL = cur;                                 // NO residual add here (deferred)
}

// final layer: fold last MLP output into the accumulated sum, then normalize
cur = ggml_add(ctx0, inpL, residual);
cur = group_rms_norm(cur, output_norm);

(Full, exact diff: [`patches/0002-fix-k2-horizon-parallel-norm.patch`](patches/0002-fix-k2-horizon-parallel-norm.patch))

Compile

Windows (Visual Studio 2022 + CUDA):

cmake -B build -G "Visual Studio 17 2022" -A x64 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
cmake --build build --config Release

Linux (GCC/Clang + CUDA):

cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
cmake --build build --config Release -j$(nproc)

CPU-only builds work too (drop -DLLAMA_CUDA=ON).

Verify your build

./build/bin/Release/llama-cli -m K2-Horizon-36B-i-compact.gguf -n 30 --temp 0.7 --top-p 0.9 -p "Merhaba dünya"

✅ Working: coherent natural-language answer (~27–45 tok/s decode with full CUDA offload). ❌ Broken: repeated > or command-line echo → one of the patches was not applied, or you are running an unpatched (vanilla/upstream) llama.cpp build.

📋 Technical Details

Architecture Information (from GGUF metadata)

• —Architecture: k2-horizon
• —Block Count: 48 layers (3 leading dense blocks, 45 MoE blocks)
• —Expert Count: 100 routed experts/layer
• —Expert Used Count: 8 (top-8 routing)
• —Shared Experts: 1/layer
• —Expert Gating: softmax with weight normalization (scale 2.5)
• —Hidden Size: 2,560
• —Feed Forward Size: 6,144 (dense) / 768 (per expert)
• —Attention Heads: 32 (KV heads: 8, grouped query attention)
• —Head Dimension: 128 (full RoPE rotation)
• —RoPE Frequency Base: 10,000,000
• —Layer Norm: Grouped RMSNorm (2 groups), ε = 1e-6
• —MoVA: Mixture-of-Value Attention — 64 value experts/layer, top-4 routed
• —Residual Scheme: Parallel (pre-norm), as implemented by Patch 2 above — see the status note for verification caveats.

Quantize Profile Details

Layer bands (48 layers): EDGE = L0–4 & L43–47 · NEAR = L5–9 & L38–42 · MID = L10–37

i-quality (Q6K/Q5K/IQ4_XS)

• —Routed Expert FFN: EDGE Q6K / NEAR Q5K / MID IQ4_XS
• —Shared FFN: Q8_0
• —Attention: Q6_K
• —BPW: 5.47
• —File Size: 23.8 GB

i-balanced (Q6K/Q5K)

• —Routed Expert FFN: EDGE Q6K / NEAR Q5K / MID Q5_K
• —Shared FFN: Q8_0
• —Attention: Q6_K
• —BPW: 6.02
• —File Size: 26.2 GB

i-compact (Q4K/Q3K)

• —Routed Expert FFN: EDGE Q4K / NEAR Q3K / MID Q3_K
• —Shared FFN: Q6_K
• —Attention: Q4_K
• —BPW: 4.04
• —File Size: 17.6 GB

💻 Usage

⚠️ Both examples below assume you already built the patched fork from the section above. Running these commands against a stock llama-cli/llama-server build (including the one installed via the "Use this model" buttons at the top of this page) will not work.

With llama.cpp (patched fork)

# Interactive chat
./build/bin/Release/llama-cli -m models/K2-Horizon-36B-i-quality.gguf \
    -n 256 --temp 0.7 --top-p 0.9 -p "Hello, how are you?"

# Single-shot completion
./build/bin/Release/llama-completion -m models/K2-Horizon-36B-i-compact.gguf \
    -n 128 -p "Explain mixture-of-experts models:"

With Python (llama-cpp-python)

Build llama-cpp-python against the same patched source tree (see above), e.g.:

CMAKE_BUILD_ARGS="-DLLAMA_CURL=OFF" pip install llama-cpp-python \
    --global-option=build_ext \
    --global-option="--library_dir=$(pwd)/build/lib" \
    --global-option="--include_dir=$(pwd)/include"

from llama_cpp import Llama

llm = Llama(
    model_path="models/K2-Horizon-36B-i-quality.gguf",
    n_ctx=8192,
    n_threads=8,
)

output = llm("Hello, how are you?", max_tokens=256)
print(output["choices"][0]["text"])

📊 Model Comparison

📝 Notes

• —All models are in GGUF format and were generated from the BF16 master checkpoint.
• —The K2Horizon architecture combines Mixture-of-Experts (100×8) with MoVA (64×4 value experts) and grouped RMSNorm. Residual wiring follows the patched behavior described in runtime fixes — see the status note there before treating it as a confirmed architectural fact.
• —Measured performance (full CUDA offload, 49/49 layers): prompt eval ≈ 28–139 tok/s, generation ≈ 27–45 tok/s (i-compact, consumer GPU).
• —Only the 3 APEX profiles are published here; the 70 GB BF16 master stays upstream.
• —If you arrived here via the "Use this model" quick-start buttons at the top of this page: those commands run vanilla llama.cpp and will not work with this GGUF yet. Please use the patched-fork instructions above instead.

📄 License

⚠️ Please review the original model's license/terms before commercial use. The GGUF files are repackaged quantizations of the upstream weights.

🙏 Acknowledgments

• —[MBZUAI-IFM](https://github.com/MBZUAI-IFM) — K2-Horizon model and official llama.cpp fork
• —[localai-org/apex-quant](https://github.com/localai-org/apex-quant) — Apex-Quant MoE-aware mixed-precision quantization framework
• —[ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) — GGUF format and quantization engine

🔗 Related Links

• —Source GGUF: https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
• —llama.cpp fork: https://github.com/MBZUAI-IFM/llama.cpp
• —Apex-Quant: https://github.com/localai-org/apex-quant
• —GGUF Format: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md