zhangsq-nju/Qwen3-0.6B-EdgeRazor-GGUF

<div align="center"> <br/> <img src="./asset/Logo-HF.png" alt="EdgeRazor Logo" width="60%"> <h3> EdgeRazor for Lightweight LLMs </h3>

<p> <a href="https://arxiv.org/abs/2605.04062" target="blank"> <img src="https://img.shields.io/badge/arXiv-EdgeRazor-b31b1b?style=flat&logo=arxiv" alt="arXiv EdgeRazor"> </a> <a href="https://github.com/zhangsq-nju/EdgeRazor" target="blank"> <img src="https://img.shields.io/badge/GitHub-EdgeRazor-blue?style=flat&logo=github" alt="GitHub EdgeRazor"> </a> <a href="https://pypi.org/project/edgerazor/" target="blank"> <img src="https://img.shields.io/pypi/v/edgerazor?style=flat&logo=pypi" alt="PyPI EdgeRazor"> </a> </p>

</div>

Contents

• —Contents
• —Model Overview
• —Model Bit-Widths
• —Get Started
• —Citation

Model Overview

• —Base Model: Qwen/Qwen3-0.6B
• —Training: zhangsq-nju/EdgeRazor
• —Inference: ggml-org/llama.cpp

Model Bit-Widths

Get Started

Use llama.cpp to conduct efficient inference on edge devices.

Check the cli.sh script for basic usage.

Model list:

• —Qwen3-0.6B-BF16.gguf: BF16 model from the original Qwen3-0.6B
• —Qwen3-0.6B-EdgeRazor-Q4_0.gguf: Q4_0 model from the Qwen3-0.6B-EdgeRazor-4bit
• —Qwen3-0.6B-EdgeRazor-TQ1_0.gguf: TQ1_0 model from Qwen3-0.6B-EdgeRazor-1.58bit
• —Qwen3-0.6B-EdgeRazor-TQ2_0.gguf: TQ2_0 model from Qwen3-0.6B-EdgeRazor-1.58bit

Citation

If you find our project useful in your research, please consider kindly citing our papers ✏️:

@article{zhangsh-edgerazor,
  title={{EdgeRazor}: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation},
  author={Shu-Hao Zhang and Le-Tong Huang and Xiang-Sheng Deng and Xin-Yi Zou and Chen Wu and Nan Li and Shao-Qun Zhang},
  year={2026},
  journal={arXiv preprint arXiv:2605.04062}
}