abhinavdread/qwen-1.5B-Q4_k_M

Qwen-1.5B-Q4KM (Quantized GGUF Model)

This repository provides a quantized GGUF build of Qwen-1.5B-Instruct using Q4_K_M quantization, optimized for CPU-only inference with llama.cpp.

The model is designed for efficient local execution, making it suitable for research, RAG pipelines, and edge deployments without requiring GPUs or external API keys.

Model Overview

Motivation

Large language models are often impractical for local or edge deployment due to hardware and memory constraints. This quantized release focuses on:

• —Reducing memory footprint
• —Preserving instruction-following capability
• —Enabling fully offline inference
• —Supporting reliable retrieval-augmented generation (RAG)

The Q4_K_M quantization strikes a balance between model size, speed, and output quality.

Recommended Usage

This model is intended to be used with llama.cpp or compatible runtimes.

Python Example (llama.cpp)

from llama_cpp import Llama

llm = Llama(
    model_path="qwen2.5-1.5b-instruct-q4_k_m.gguf",
    n_ctx=4096,
    n_threads=8,
    n_gpu_layers=0,
    verbose=False
)

response = llm(
    "Explain Retrieval Augmented Generation in simple terms.",
    max_tokens=256,
    temperature=0.2
)

print(response["choices"][0]["text"])