boboliu/Qwen3-Reranker-0.6B-W4A16-G128

Qwen3-Reranker-0.6B-W4A16-G128

GPTQ Quantized Qwen/Qwen3-Reranker-0.6B with Ultrachat, THUIR/T2Ranking and m-a-p/COIG-CQIA for calibration set.

What's the benefit?

VRAM Usage: 3228M -> 2124M (w/o FA2, according to Embedding model's result).

What's the cost?

I think <5% accuracy, further evaluation on the way...

The Embedding one shows ~0.7%.

How to use it?

pip install compressed-tensors optimum and auto-gptq / gptqmodel, then goto the official usage guide.