VladHong/K2-Horizon-MoVA-36B-A4B-APEX-Mini-GGUF

K2-Horizon-MoVA-36B-A4B APEX Mini (GGUF)

This repository contains a compact GGUF quantization of IFM/K2-Horizon-MoVA-36B-A4B for local llama.cpp inference.

K2-Horizon-MoVA-36B-A4B is a sparse Mixture-of-Experts model with Mixture-of-Values attention (MoVA). It has 36B total parameters and activates approximately 4B parameters per token. The native context length is 524,288 tokens.

This is the vanilla upstream APEX Mini mixed-precision profile, regenerated for K2-Horizon with a K2-shaped importance matrix computed from a cleaned, high-quality multilingual calibration corpus. The profile prioritizes normal generation quality and CUDA compatibility over the earlier 11 GB experimental target: Q3K routed experts in edge layers, IQ2S routed experts in middle layers, higher-precision shared experts and attention, and standard F32/Q6_K tensors where appropriate.

File

Quick start — llama.cpp

K2-Horizon requires the MBZUAI-IFM `model/K2Horizon` llama.cpp branch, or another build that includes the k2-horizon architecture and tokenizer support. A vanilla llama.cpp release may not recognize this GGUF yet.

Interactive chat:

llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf \
  -c 4096 \
  -ngl 99 \
  -cnv

For CPU-only inference, omit -ngl 99 (or set -ngl 0). The embedded chat template is used automatically.

The K2-Horizon GGUF carries IFM's own chat template; no external template file is required.

OpenAI-compatible local server:

llama-server \
  -m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf \
  -c 4096 \
  -ngl 99 \
  --host 127.0.0.1 \
  --port 8080

Then send requests to http://127.0.0.1:8080/v1/chat/completions with any OpenAI-compatible client. Increase -c only when sufficient RAM/VRAM is available; the model's native maximum is 524,288 tokens.

Quantization profile

The profile was produced from the BF16 checkpoint with the upstream APEX Mini tensor map (k2-vanilla-apex-mini-48.txt) and a K2-Horizon calibration matrix. A short CUDA-server smoke test produced coherent instruction-following and factual responses on the special K2-Horizon llama.cpp build.

This file contains no TQ1_0 or TQ2_0 tensors. IQ2_S is intentional: it is the standard APEX Mini middle-expert type and has CUDA kernels in the required K2-Horizon build.

Calibration and imatrix process

The calibration source is the combined_all_medium set from the MIT-licensed eaddario/imatrix-calibration dataset. The Parquet source was exported to UTF-8 plain text and cleaned before use:

• —Unicode NFKC and control-character normalization
• —whitespace and blank-line normalization
• —removal of samples shorter than 80 or longer than 16,000 characters
• —removal of adult/erotic material, including multilingual English and Chinese terms
• —removal of gambling, SEO, credential-request, repeated-token, and low-diversity spam
• —exact duplicate removal

The resulting corpus contains 15,654 samples and is used as plain text. Its local SHA-256 is 010725E8BFA24511B5FEE811E5AC066F6C7BDE41FB73E6FC1C7C24FED302B58D.

Using the special K2-Horizon llama.cpp build, the importance matrix was generated with 12 chunks, a 512-token context, automatic GPU fitting, no perplexity pass, and GGUF output:

llama-imatrix \
  -m K2-Horizon-36B-BF16.gguf \
  -f calibration-high-quality-combined-all-medium.txt \
  -ngl auto \
  --chunks 12 \
  --no-ppl \
  --parse-special \
  -o k2-horizon-high-quality.imatrix.gguf

The matrix contains 609 tensor entries over 12 chunks. Its SHA-256 is 12AE83A745C36045C6929C0907F924F97CF75C4B4D8B867C73D11544E7D32982. The final APEX Mini was then produced from BF16 with the upstream tensor map and this matrix:

llama-quantize \
  --tensor-type-file k2-vanilla-apex-mini-48.txt \
  --imatrix k2-horizon-high-quality.imatrix.gguf \
  K2-Horizon-36B-BF16.gguf \
  K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf \
  Q3_K_M

The resulting GGUF retains the model's embedded IFM chat template and requires the K2-Horizon llama.cpp architecture support described above.

License and attribution

This quantization is derived from IFM/K2-Horizon-MoVA-36B-A4B and follows the upstream Apache-2.0 license. Please read the upstream model card for the original model's capabilities, limitations, citation, and usage guidance.