rkumar70900/qwen2.5-1.5b-gguf-experiments

Qwen2.5-1.5B-Instruct — GGUF Quantization Experiments

This repo contains Qwen2.5-1.5B-Instruct quantized into multiple GGUF formats using llama.cpp. It was created as part of a hands-on quantization experiment documenting the full process from raw HuggingFace weights → multiple GGUF formats → quality evaluation.

What's in This Repo

gguf/
├── qwen2.5-1.5b-f16.gguf          ~2.9 GB   source of truth — full precision
├── qwen2.5-1.5b-Q8_0.gguf         ~1.6 GB   near-lossless
├── qwen2.5-1.5b-Q5_K_M.gguf       ~1.0 GB   great quality/size tradeoff
├── qwen2.5-1.5b-Q4_K_M.gguf       ~935 MB   recommended — sweet spot ★
├── qwen2.5-1.5b-Q4_K_S.gguf       ~865 MB   leaner 4-bit variant
├── qwen2.5-1.5b-Q2_K.gguf         ~530 MB   aggressive K-quant baseline
├── qwen2.5-1.5b-Q2_K_S.gguf       ~530 MB   aggressive K-quant — needs imatrix
├── qwen2.5-1.5b-IQ3_M.gguf        ~680 MB   importance-weighted 3-bit
├── qwen2.5-1.5b-IQ2_XS.gguf       ~480 MB   importance-weighted 2-bit — needs imatrix
├── qwen2.5-1.5b-IQ2_XXS.gguf      ~420 MB   most aggressive — needs imatrix
├── qwen2.5-1.5b-IQ2_S.gguf        ~450 MB   importance-weighted 2.5-bit — needs imatrix
├── qwen2.5-1.5b-IQ1_M.gguf        ~300 MB   extreme 1.75-bit — needs imatrix
└── qwen2.5-1.5b-IQ1_S.gguf        ~280 MB   extreme 1.56-bit — needs imatrix

Note on f16: The F16 file is included as the reference baseline for perplexity comparisons. It is not intended for general inference use — at 2.9 GB it offers no practical advantage over Q8_0 for local deployment.

Which File Should I Use?

If you're not sure, start with Q4_K_M. It's the most tested format in the community and gives you ~68% size reduction with minimal quality loss.

⚠️ IQ1 and IQ2 formats (IQ1_S, IQ1_M, IQ2_S, IQ2_XS, IQ2_XXS, Q2_K_S) were all generated with an importance matrix. Without one, these formats produce significantly degraded output. See the Imatrix Calibration section below for details.

Format Guide

K-Quant Family (Q\\_K\_\)

Standard llama.cpp quantization using superblocks of 256 weights. The suffix means:

• —_S (Small) — more aggressive, smaller file
• —_M (Medium) — mixed-precision, smarter assignment of bits to sensitive layers

Despite the "4" in Q4KM, it is not uniform 4-bit. Critical tensors like the embedding table and output projection are bumped to 6-bit internally. The "4" is the average bits-per-weight.

IQ Family (IQ\\_\)

Importance-weighted quantization. These formats use an importance matrix — calibration data was run through the base model to identify which weights matter most, and precision was distributed accordingly. This is why IQ formats punch above their weight class at the same file size compared to K-quants.

The IQ2 files in this repo were generated with a WikiText-2 calibration dataset (see below). Without an importance matrix, these formats produce near-incoherent output — the imatrix is what makes them viable.

Quantization Details

Base model: Qwen/Qwen2.5-1.5B-Instruct Quantization tool: llama.cpp build 7074 (commit 22e1ce2f8) Source precision: F16 GGUF (converted from original SafeTensors) Platform: Apple Silicon (arm64)

Imatrix Calibration

The IQ2 formats were quantized using an importance matrix generated from WikiText-2:

from datasets import load_dataset

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")

with open("calibration.txt", "w") as f:
    for row in dataset:
        text = row["text"].strip()
        if len(text) > 100:
            f.write(text + "\n")

./build/bin/llama-imatrix \
    -m qwen2.5-1.5b-f16.gguf \
    -f calibration.txt \
    -o imatrix.dat \
    --ctx-size 512 \
    -ngl -1 \
    --chunks 100

How to Run

llama.cpp CLI

./build/bin/llama-cli \
    -m qwen2.5-1.5b-Q4_K_M.gguf \
    -n 512 \
    -ngl 99 \
    --prompt "Explain the difference between supervised and unsupervised learning."

llama.cpp Server (OpenAI-compatible)

./build/bin/llama-server \
    -m qwen2.5-1.5b-Q4_K_M.gguf \
    -ngl 99 \
    --port 8080

Then hit http://localhost:8080/v1/chat/completions like any OpenAI endpoint.

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="qwen2.5-1.5b-Q4_K_M.gguf",
    n_gpu_layers=-1,  # full GPU offload
    n_ctx=4096,
)

output = llm(
    "Explain quantization in simple terms:",
    max_tokens=256,
    temperature=0.7,
)
print(output["choices"][0]["text"])

Ollama

ollama run hf.co/your-username/qwen2.5-1.5b-gguf-experiments:Q4_K_M

Model Architecture (from metadata)

License

The quantized weights in this repo are derived from Qwen/Qwen2.5-1.5B-Instruct and inherit its Apache 2.0 license.

Citation

If you use these files in your work, please also cite the original Qwen2.5 model:

@misc{qwen2.5,
    title  = {Qwen2.5: A Party of Foundation Models},
    author = {Qwen Team},
    year   = {2024},
    url    = {https://qwenlm.github.io/blog/qwen2.5/}
}