NovaAI6868/Qwen3.8-4B-Distill-GGUF-Q2_K

Qwen3.8-4B — GGUF (Q2_K)

Q2_K imatrix quantization of [empero-ai/Qwen3.8-4B](https://huggingface.co/empero-ai/Qwen3.8-4B) — a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture — for llama.cpp, Ollama, LM Studio, Jan, KoboldCpp, and other stock GGUF runtimes.

This repo hosts the Q2_K quant only. For higher-quality quantizations (Q4KM and up), see [empero-ai/Qwen3.8-4B-Distill-GGUF](https://huggingface.co/empero-ai/Qwen3.8-4B-Distill-GGUF). Capability details and full benchmark results live on the [main model card](https://huggingface.co/empero-ai/Qwen3.8-4B).

[!Note] Qwen3.5-class models are hybrids: three Gated DeltaNet layers for every full-attention layer. A recent llama.cpp build with Qwen3.5 / Gated DeltaNet support is required — older builds will fail to load the architecture.

[!Warning] Q2K is an aggressive 2-bit quantization. Expect a noticeable quality drop compared to Q4K_M and above, especially on long reasoning chains. Use this file when memory is the hard constraint.

Files

Size is exact decimal GB from the uploaded file (1 GB = 1,000,000,000 bytes).

Quantization details

• —Method: Q2_K with importance matrix (imatrix) weighting
• —Calibration: imatrix.dat computed from a diverse English calibration dataset (~4,900 chunks)
• —Source weights: Qwen3.8-4B-BF16.gguf (full-precision reference)

Usage

llama.cpp

llama-cli -m Qwen3.8-4B-Q2_K.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv

Use the built-in chat template (-cnv). The model is a reasoning model: every answer opens with a <think> block, so allow a generous -n and strip the <think>...</think> span for end users.

Ollama / LM Studio / Jan / KoboldCpp

Download the GGUF and load it directly; the chat template is embedded in the file. Recommended sampling: temperature=0.6, top_p=0.95, top_k=20.

Model architecture (from GGUF metadata)

Provenance & licensing

Quantization of [empero-ai/Qwen3.8-4B](https://huggingface.co/empero-ai/Qwen3.8-4B), a distillation of Qwen3.8 2.4T A95B into Qwen/Qwen3.5-4B. Weights are Apache-2.0, inherited from the Qwen base, shared as-is.

Acknowledgements

• —Source model: Empero
• —Base model: Qwen3.5-4B (Alibaba Qwen team)
• —GGUF quantization: llama.cpp (ggml-org)