CoolFace
Modelpublic

AMAImedia/K2-Horizon-7B-GGUF

sourceHugging Faceapache-2.0updated 11d agoView on Hugging Face
4likes6.8kdownloads
Model Card

<!-- apex-banner --> <div style="background-color: #f59e0b; color: white; padding: 20px; border-radius: 10px; text-align: center; margin: 20px 0;"> <h2 style="color: white; margin: 0 0 10px 0;">⚡ Each donation funds the next large quant.</h2> <p style="font-size: 16px; margin: 0 0 15px 0;">I host free GGUF or MoE quants as independent research.<br> Local hardware: <b>Mechrevo Kuangshi GM7AG0M</b> — RTX 3060 Laptop 6GB GDDR6, 64GB DDR5, i7-12700H (14C/20T, 4.7GHz), Windows 11, Samsung 990 Pro.<br> Good for imatrix and 0.6–35B-class work in RAM. <b>9B+ and searches need rented H200/Blackwell</b>, typically $100 per quant.</p> <p style="font-size: 20px; margin: 0;"> <a href="https://boosty.to/amaimediacom" style="color: white; text-decoration: underline;">🎉 Boosty🦄</a>  |  <a href="https://donatex.gg/widgets/donation-goal/89bc59e8-b69c-467a-9aa0-5e1c3f8842d6" style="color: white; text-decoration: underline;">☕ Buy Me a Coffee🦄</a>  |  <a href="https://www.donationalerts.com/r/djbionicl" style="color: white; text-decoration: underline;">⭐ DonationAlerts🦄</a> </p> <p style="font-size: 14px; margin: 10px 0 0 0; opacity: 0.9;">💚 Thanks to Hugging Face for extra storage.🦄</p> </div>

K2-Horizon-7B — GGUF Quantizations

GGUF quantizations of IFM/K2-Horizon-7B, a dense 7B-parameter causal decoder (K2HorizonForCausalLM, model_type: k2_horizon).

Quantized by NANI-Nithin using a custom pipeline built on the MBZUAI-IFM llama.cpp fork (branch model/K2Horizon).

Model Details

PropertyValue
Base modelIFM/K2-Horizon-7B
ArchitectureK2HorizonForCausalLM (k2_horizon)
Parameters~7B (dense decoder, no MoE)
Original dtypeBF16
BF16 GGUF size~18.01 GB
Source GGUFIFM/K2-Horizon-7B-GGUF
llama.cpp forkMBZUAI-IFM/llama.cpp @ model/K2Horizon
Note: These GGUFs carry the k2-horizon architecture token and require the MBZUAI-IFM fork (or upstream llama.cpp once support is merged) to run. Vanilla upstream llama.cpp (as of September 2026) does not support K2HorizonForCausalLM.

Included Files

Standard Quantizations

FileBits/WeightNotes
K2-Horizon-7B-BF16.gguf16 bpwSource quant, full precision
K2-Horizon-7B-Q8_0.gguf8 bpwNear-lossless, recommended reference
K2-Horizon-7B-Q6_K.gguf6 bpwNear-lossless K-quant
K2-Horizon-7B-Q5_K_M.gguf5 bpwBest quality/size in the 5-bit range
K2-Horizon-7B-Q5_K_S.gguf5 bpwSmaller 5-bit variant
K2-Horizon-7B-Q5_1.gguf5 bpwLegacy 5-bit
K2-Horizon-7B-Q5_0.gguf5 bpwLegacy 5-bit
K2-Horizon-7B-Q4_K_M.gguf4 bpwRecommended general use
K2-Horizon-7B-Q4_K_S.gguf4 bpwSmaller 4-bit K-quant
K2-Horizon-7B-Q4_1.gguf4 bpwLegacy 4-bit
K2-Horizon-7B-Q4_0.gguf4 bpwLegacy 4-bit
K2-Horizon-7B-Q3_K_L.gguf3 bpwLarge 3-bit K-quant
K2-Horizon-7B-Q3_K_M.gguf3 bpwMedium 3-bit K-quant
K2-Horizon-7B-Q3_K_S.gguf3 bpwSmall 3-bit K-quant
K2-Horizon-7B-Q2_K.gguf2 bpwAggressive compression
K2-Horizon-7B-Q2_K_S.gguf2 bpwSmaller 2-bit K-quant (imatrix-guided)
K2-Horizon-7B-Q2_0.gguf2.25 bpwGroup-64 2-bit
K2-Horizon-7B-Q1_0.gguf1.125 bpwMaximum compression

IQ (Importance-Matrix) Quantizations

FileBits/Weight
K2-Horizon-7B-IQ4_NL.gguf~4 bpw
K2-Horizon-7B-IQ4_XS.gguf~4 bpw
K2-Horizon-7B-IQ3_M.gguf~3 bpw
K2-Horizon-7B-IQ3_S.gguf~3 bpw
K2-Horizon-7B-IQ3_XS.gguf~3 bpw
K2-Horizon-7B-IQ3_XXS.gguf~3 bpw
K2-Horizon-7B-IQ2_M.gguf~2 bpw
K2-Horizon-7B-IQ2_S.gguf~2 bpw
K2-Horizon-7B-IQ2_XS.gguf~2 bpw
K2-Horizon-7B-IQ2_XXS.gguf~2 bpw
K2-Horizon-7B-IQ1_M.gguf1.75 bpw
K2-Horizon-7B-IQ1_S.gguf1.56 bpw

Quantization Method

  • —Source: IFM's official BF16 GGUF (K2-Horizon-7B-BF16.gguf).
  • —imatrix: Computed from Salesforce/wikitext (wikitext-2-raw-v1, 500 rows) with 12 GPU layers offloaded on an RTX 4060 Laptop (8 GB VRAM) due to the 18 GB model size. Applied to all K-quants below Q6 and all IQ quants.
  • —Fork: MBZUAI-IFM/llama.cpp, branch model/K2Horizon.

Usage

Requires the MBZUAI-IFM llama.cpp fork (model/K2Horizon branch).
bash
git clone -b model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release

./build/bin/llama-cli \
  -m K2-Horizon-7B-Q4_K_M.gguf \
  -p "Hello, I am" \
  -n 128 \
  -ngl 35

License

Weights are released under the same license as the original IFM/K2-Horizon-7B model. Please refer to the original repository for full license terms.

Credits