ethicalabs/Qwen3-Reranker-8B-GGUF

ethicalabs/Qwen3-Reranker-8B-GGUF

This is a Q80 quantized GGUF of [Qwen/Qwen3-Reranker-8B](https://huggingface.co/Qwen/Qwen3-Reranker-8B) converted with llama.cpp's `converthftogguf.py (commit b820cc8`).

Why this exists

All publicly available Q8_0 GGUF conversions of Qwen3-Reranker-8B (as of July 2026) are broken for use with llama.cpp's /v1/rerank endpoint — they return null relevance scores on every query.

Root cause

The broken GGUFs are missing the tokenizer.chat_template.rerank key in the GGUF metadata. Without it, llama.cpp's /v1/rerank endpoint uses the default chat template to format query+document pairs, which truncates the input (128 tokens instead of 560) and produces null relevance scores.

Converting from the original HuggingFace model with a recent llama.cpp convert_hf_to_gguf.py correctly adds this key, producing a working GGUF.

The file size difference (8.04 GB vs 8.71 GB) is because the broken conversions duplicate shared weights (embedding/output layer), bloating the file by ~670 MB.

Verification

Tested with llama.cpp server-vulkan image (commit 7af4279f4)

Conversion command

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install transformers torch sentencepiece
python convert_hf_to_gguf.py Qwen/Qwen3-Reranker-8B --outtype q8_0 --outfile Qwen3-Reranker-8B.Q8_0.gguf

models.ini config

[qwen3-reranker-8b]
model = ../models/Qwen3-Reranker-8B.Q8_0.gguf
load-on-startup = 1
reranking = true
embeddings = true
pooling = rank
spec-default = 1
ctx-size = 32768

Usage with llama.cpp

Start the server:

llama-server --models-preset models.ini --port 9001 --host 0.0.0.0

Rerank API:

curl -X POST http://localhost:9001/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-reranker-8b","query":"your query","documents":["doc1","doc2"]}'

Original Model Card — Qwen3-Reranker-8B

Highlights

The Qwen3 Embedding model series is the latest proprietary model of the Qwen family, specifically designed for text embedding and ranking tasks. Building upon the dense foundational models of the Qwen3 series, it provides a comprehensive range of text embeddings and reranking models in various sizes (0.6B, 4B, and 8B). This series inherits the exceptional multilingual capabilities, long-text understanding, and reasoning skills of its foundational model.

Exceptional Versatility: The embedding model has achieved state-of-the-art performance across a wide range of downstream application evaluations. The 8B size embedding model ranks No.1 in the MTEB multilingual leaderboard (as of June 5, 2025, score 70.58), while the reranking model excels in various text retrieval scenarios.

Multilingual Capability: The Qwen3 Embedding series offer support for over 100 languages, thanks to the multilingual capabilites of Qwen3 models.

Model Overview

• —Model Type: Text Reranking
• —Supported Languages: 100+ Languages
• —Number of Parameters: 8B
• —Context Length: 32k

For more details, see the Qwen3 Embedding blog and GitHub.

Citation

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}