CoolFace
Modelpublic

Abiray/K2-Horizon-MoVA-36B-A4B-GGUF

sourceHugging Faceapache-2.0updated 18d agoView on Hugging Face
4likes1.1kdownloads
Model Card

K2-Horizon-MoVA-36B-A4B-GGUF

This repository contains GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B, a sparse Mixture-of-Experts model utilizing Mixture-of-Values Attention (MoVA). It features 36B total parameters with only ~4B activated per token, enabling near-small-model inference speeds with 36B-scale intelligence.

[!IMPORTANT] Architecture Support Required: As of early 2026, k2-horizon is a newly introduced architecture not yet merged into upstream mainline llama.cpp. To run or quantize these GGUFs, you must build from the official MBZUAI-IFM fork (model/K2Horizon branch). Running on vanilla llama.cpp will result in unknown model architecture: 'k2-horizon'.

Quantization Overview & Hardware Recommendations

File NameQuantSizeRecommended VRAM (Full Offload)Minimum System RAM (CPU)Description
K2-Horizon-MoVA-36B-A4B-Q3_K_S.ggufQ3KS16.4 GB20 GB (RTX 3090/4090)24 GBSmallest footprint; lowest memory usage.
K2-Horizon-MoVA-36B-A4B-Q3_K_M.ggufQ3KM17.7 GB22 GB (RTX 3090/4090)28 GBBalanced 3-bit quantization.
K2-Horizon-MoVA-36B-A4B-Q4_K_S.ggufQ4KS21.4 GB24 GB (RTX 3090/4090)32 GBGreat performance/size ratio for 24 GB GPUs.
K2-Horizon-MoVA-36B-A4B-Q4_K_M.ggufQ4KM22.4 GB24 GB+ (RTX 3090/4090)32 GBRecommended balance of quality and memory efficiency.
K2-Horizon-MoVA-36B-A4B-Q5_K_M.ggufQ5KM26.4 GB32 GB36 GBNear-lossless retention; high precision.
K2-Horizon-MoVA-36B-A4B-Q6_K.ggufQ6_K30.8 GB36 GB40 GBHigh precision; preserves subtle router logits.
K2-Horizon-MoVA-36B-A4B-Q8_0.ggufQ8_039.8 GB48 GB (2× 24 GB / A6000)48 GBMaximum precision; near identical to original BF16.

Installation & Setup

1. Build llama.cpp with K2-Horizon Support

bash
# Clone the official architecture fork
git clone -b model/K2Horizon [https://github.com/MBZUAI-IFM/llama.cpp.git](https://github.com/MBZUAI-IFM/llama.cpp.git)
cd llama.cpp

# Build with hardware acceleration (enable CUDA if running on NVIDIA GPUs)
cmake -B build -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release # Add -DGGML_CUDA=ON for NVIDIA GPU
cmake --build build -j$(nproc) --target llama-cli llama-server

Quickstart Guide

Running via llama-cli

bash
./llama.cpp/build/bin/llama-cli \
  -m ./K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --temp 1.0 \
  --top-p 0.95 \
  -p "<|ifm|im_start|>user\nWrite an efficient Python script for asynchronous data fetching.<|ifm|im_end|>\n<|ifm|im_start|>assistant\n"

Prompt Format & Reasoning Traces

K2-Horizon uses ChatML-style delimiters with native thinking tags:

bash
<|ifm|im_start|>system
You are a helpful assistant.<|ifm|im_end|>
<|ifm|im_start|>user
Your query goes here.<|ifm|im_end|>
<|ifm|im_start|>assistant
<ifm|think>
[Model generates intermediate reasoning steps here]
</ifm|think>
[Final output generated here]<|ifm|im_end|>