ngquocvinh/K2-Horizon-MoVA-36B-A4B-GGUF
K2-Horizon-MoVA-36B-A4B GGUF
Community GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B.
<div align="center" style="background-color:#f59e0b;color:#ffffff;padding:16px 20px;border-radius:10px;line-height:1.7;"> ☕ If this GGUF made your day easier, a coffee would make mine.<br> <a href="https://ko-fi.com/ngquocvinh" style="color:#ffffff;"><strong style="color:#ffffff;">Send a coffee ☕</strong></a><br> I build and test these releases myself. Your coffee helps keep me going.<br> Thank you for supporting this work. </div>
About K2-Horizon-MoVA-36B-A4B
K2-Horizon-MoVA-36B-A4B is a sparse Mixture-of-Experts model with Mixture-of-Values attention (MoVA). The upstream checkpoint has 36B total parameters and activates about 4B parameters per token. It advertises a native 524,288-token context window and is intended for reasoning, coding, agentic, and long-context workloads. See the official model card for the original model, serving recipes, prompt conventions, and upstream results.
Upstream K2-Horizon-MoVA-36B-A4B benchmark results; the image and scores are from the [official model card](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B).
Fidelity measurements
The table below compares every published GGUF file against the BF16 reference on a held-out WikiText pilot: eight chunks from wiki.test.raw and eight chunks from wiki.valid.raw, using a 4,096-token context and the same K2 llama.cpp runtime. Values are averaged across the two splits. Size (GB) uses decimal GB (file_size_bytes / 1,000,000,000). Lower Mean KLD, ΔPPL, and RMS Δp, and higher Top-1 agreement, indicate closer next-token behavior to BF16. The BF16 reference mean PPL was 6.815612 in this pilot. These are next-token fidelity measurements, not a direct percentage of capabilities retained and not a substitute for task-specific evaluation.
Bold rows mark practical sweet spots: Q5_K_M for extra fidelity headroom, Q4_K_M as the balanced default, and IQ3_M as the compact importance-aware option. Q8_0 remains the high-bit reference profile.
In this pilot, Q1_0 shows very large divergence from the BF16 reference; the extreme-compression profiles should therefore be treated as experimental.
The machine-readable results are available in `reproducibility/quality-summary.tsv`, with corpus hashes, evaluation settings, and runtime provenance in `reproducibility/manifest.md`.
Quick start
Use a K2-Horizon-capable llama.cpp build. The explicit partial offload and CPU MoE options in this example keep the model within the available accelerator memory when full offload is not possible; adjust the GPU-layer count for your hardware.
./llama-cli \
-m K2-Horizon-MoVA-36B-A4B-Q6_K.gguf \
--chat-template-file reproducibility/chat_template_smoke_user.jinja \
--jinja \
--reasoning off \
-p 'Answer briefly in English: What is GGUF, and why is it useful for running language models locally?' \
-n 128 -c 4096 \
--device CUDA0 \
--gpu-layers 8 \
--cpu-moeThe included template is the compatible single-turn template used by the release smoke test. The upstream full tool-aware Jinja template is not claimed as certified by this package. Long-context use may require substantially more memory than the short example above.
Reproducibility and validation
The GGUF files were quantized directly from the locked upstream BF16 input. Q8_0 was generated without an imatrix; the other ladder members used the model-specific combined importance matrix. Every published file passed the load/generate smoke test before upload. Runtime throughput data, when present, is supplementary and is recorded separately from the BF16 fidelity comparison above in `reproducibility/runtime-summary.tsv`.
The public package includes compact reproduction inputs and scripts. Raw conversion, imatrix, quantization, smoke-test, fidelity, and benchmark logs are kept locally under reports/ and are intentionally not uploaded.
Checksums for all published artifacts and public reproduction files are in `SHA256SUMS.txt`. The locked source revision, BF16 checksum, runtime commit, calibration inputs, and validation settings are in `reproducibility/manifest.md`.
License and attribution
The upstream model is released under the Apache License 2.0. Preserve upstream attribution and the included `LICENSE` when redistributing these derivative artifacts.
These are community GGUF quantizations, not an official IFM release or endorsement.