Modelpublic

mlx-community/Qwen3-Reranker-0.6B-4bit

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

2likes1.4kdownloads

Model Card

mlx-community/Qwen3-Reranker-0.6B-4bit

This model was converted to MLX format from Qwen/Qwen3-Reranker-0.6B using mlx-lm 0.31.3.

—Quantization: affine 4-bit, group_size=64 (~4.5 bits/weight)
—On-disk size: ~331 MB
—Task: text reranking (cross-encoder, yes/no relevance scoring)

Scoring recipe

Qwen3-Reranker is a causal LM used as a reranker: the relevance score of a (query, document) pair is softmax([logit("no"), logit("yes")])[1] at the last position of the prompt below.

python

import mlx.core as mx
from mlx_lm import load

model, tok = load("mlx-community/Qwen3-Reranker-0.6B-4bit")
hf = getattr(tok, "_tokenizer", tok)

INSTRUCT = "Given a web search query, retrieve relevant passages that answer the query"
PREFIX = ('<|im_start|>system\nJudge whether the Document meets the requirements '
          'based on the Query and the Instruct provided. Note that the answer can '
          'only be "yes" or "no".<|im_end|>\n<|im_start|>user\n')
SUFFIX = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
true_id, false_id = hf.convert_tokens_to_ids("yes"), hf.convert_tokens_to_ids("no")
pre, suf = hf.encode(PREFIX, add_special_tokens=False), hf.encode(SUFFIX, add_special_tokens=False)

def rerank_score(query, doc):
    content = f"<Instruct>: {INSTRUCT}\n<Query>: {query}\n<Document>: {doc}"
    ids = pre + hf.encode(content, add_special_tokens=False) + suf
    logits = model(mx.array([ids]))[:, -1, :]
    pair = mx.stack([logits[0, false_id], logits[0, true_id]])
    return float(mx.exp((pair - mx.logsumexp(pair))[1]))

print(rerank_score("What is the capital of China?", "The capital of China is Beijing."))