KoliaNik/Qwen3-Reranker-4B-NVFP4A16

Qwen3-Reranker-4B-NVFP4A16

NVFP4 weight-only quantization (W4A16) of `Qwen/Qwen3-Reranker-4B`, produced with llm-compressor 0.13.0 / compressed-tensors 0.18.0. Weights are FP4 (E2M1, group size 16, FP8 per-group scales); activations stay bf16. Data-free — no calibration set was used.

A W4A4 sibling (weights and activations in FP4, calibrated) is published at `KoliaNik/Qwen3-Reranker-4B-NVFP4`. See Which variant should I use? below.

Reading the Safetensors widget above. It says BF16 · U8 and 4B params, which looks like an unquantized model. It is not. U8 is the packed FP4 — two 4-bit values per byte — and covers 3.63 B of the 4.02 B parameters, i.e. every Linear in the 36 decoder layers. BF16 is the 0.39 B parameters deliberately left alone: the embedding table (which is also the LM head, see below) and the norms. The FP8 per-group scales do not appear in that widget at all. The file is 2.63 GiB, against 7.49 GiB for the bf16 original.

Why another NVFP4 reranker

The point of a 4-bit reranker is to leave VRAM for something else — a main model's KV cache, or a co-located embedding model. Existing NVFP4 checkpoints of this model spend ~0.72 GiB on a tensor that is never used at inference. Verified by reading the safetensors headers directly:

That is ~0.72 GiB back. In a deployment where the freed memory becomes KV cache, at a typical ~54k tokens/GiB for a 27B-class model that is roughly 39 000 extra context tokens.

Nothing is wrong with those checkpoints numerically — this is purely about not shipping a redundant copy of a 151669 × 2560 table.

What is quantized — and what deliberately is not

targets=["Linear"], ignore=["lm_head"]. All 36 decoder layers' q/k/v/o_proj and gate/up/down_proj are FP4. Embeddings, norms and the LM head are untouched.

`lm_head` must not be quantized. vLLM converts this model to Qwen3ForSequenceClassification and builds the ranking head out of two rows of lm_head — the "no" and "yes" token embeddings (classifier_from_token). It computes score_weight = lm_head[yes] − lm_head[no], loads that into a 1-row score layer, and then does del lm_head. Quantizing lm_head therefore injects error directly into the only two rows that produce the score. Because tie_word_embeddings: true, lm_head is embed_tokens, which is an nn.Embedding and so is already outside targets=["Linear"] — but the ignore entry keeps that explicit and survives future recipe edits.

The same choice is made by the reference FP8 checkpoint `DCC-BS/Qwen3-Reranker-4B-FP8-Dynamic`.

Serving with vLLM

vllm serve KoliaNik/Qwen3-Reranker-4B-NVFP4A16 \
  --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}'

vLLM does not apply the reranker prompt template for you — this model has no get_score_template, so for "LLM as reranker" the score endpoint simply concatenates text_1 + text_2 and tokenizes the result. Build the canonical prompt yourself and split it across the two fields:

import requests

PREFIX = ('<|im_start|>system\n'
          'Judge whether the Document meets the requirements based on the Query and the '
          'Instruct provided. Note that the answer can only be "yes" or "no".'
          '<|im_end|>\n<|im_start|>user\n')
SUFFIX = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
INSTRUCT = "Given a web search query, retrieve relevant passages that answer the query"

def text_1(query, instruct=INSTRUCT):
    return PREFIX + f"<Instruct>: {instruct}\n<Query>: {query}\n<Document>: "

def text_2(doc):
    return doc + SUFFIX

r = requests.post("http://127.0.0.1:8000/v1/score", json={
    "model": "KoliaNik/Qwen3-Reranker-4B-NVFP4A16",
    "text_1": text_1("What is the capital of France?"),
    "text_2": [text_2("Paris is the capital and largest city of France."),
               text_2("The blue whale is the largest marine mammal.")],
})
print([d["score"] for d in r.json()["data"]])
# [0.9912..., 0.000005...]

The separators are single \n, taken from the model's own chat_template.jinja. (The base model card's vLLM snippet uses \n\n; the template does not. Pick one and keep it consistent between indexing and evaluation.) /v1/rerank works the same way. The returned score is sigmoid(logit_yes − logit_no), i.e. P("yes").

The template trap — read this before you benchmark

--hf-overrides builds the score head, but it does not make vLLM apply the reranker prompt template. If neither side supplies it, vLLM receives a bare query glued to a bare document. The engine still starts, still registers /v1/score and /v1/rerank, still returns plausible scores in 0–1 — and silently ranks nonsense.

Measured on the bf16 base model with no template, for the query "What is the capital of France?":

The model separates on-topic from off-topic but cannot tell which capital was asked about, because the query never lands in its slot. With the template applied, the same three documents score 0.991 / 0.000007 / 0.0000004 and order correctly.

The tell is the token count: ~25 prompt tokens per pair without the template, 80–100 with it. Two ways to supply it correctly:

1. 1.Server-side — pass --chat-template <vllm>/examples/pooling/score/template/qwen3_reranker.jinja alongside --hf-overrides (the vLLM example also passes --runner pooling). Then send plain queries and documents; /v1/score and /v1/rerank additionally accept an instruction field.
2. 2.Client-side — build the canonical prompt yourself and split it so that text_1 + text_2 reconstructs it exactly, as in the snippet above.

Either way, verify it: POST /tokenize on the concatenated prompt must return exactly the usage.total_tokens that /v1/score reports. And after any config change make the smoke test "does a known-relevant document outrank a known-irrelevant one", not "does the endpoint answer 200" — the failure above passes the second check.

Every number in this card was produced with the client-side form and verified against /tokenize, so none of them are affected by this trap.

Measurements

All numbers below were measured directly, on 2× RTX 5090 (SM120) with vLLM 0.28.0, --tensor-parallel-size 2 --enforce-eager --max-model-len 768 --kv-cache-memory-bytes 134217728 --max-num-batched-tokens 768. The GPUs were shared with another resident workload, so these are not peak-throughput figures — but every variant ran under an identical configuration, so they are comparable to each other.

Size and VRAM

The FP8 file is 4.833 GiB but only 4.22 GiB reaches VRAM — vLLM drops lm_head after building the score head. Per-card occupancy includes the CUDA context, KV cache and activations. W4A4 needs ~0.23 GiB/card more than A16 at runtime for the FlashInfer workspace, despite identical weights.

Composition of this checkpoint: 1.692 GiB packed FP4 (uint8) + 0.211 GiB FP8 group scales

• —0.724 GiB bf16 (embeddings and norms).

Single-GPU gotcha. At TP=1 vLLM allocates a full ParallelLMHead (151669 × 2560 bf16 = 742 MiB) before tying it to embed_tokens and deleting it. Load peak therefore exceeds steady state by that much; at TP=2 it is split across cards. Budget ~0.75 GiB of transient headroom when starting this model on a single GPU.

Quality

Public reranking benchmarks, 150 sampled queries each, documents truncated to 800 characters. Δ is the mean per-query difference against bf16 with its standard error; Spearman/Pearson and mean |Δscore| are computed over all pairs.

MTEB RuBQReranking (Russian, 3535 pairs):

MTEB SciDocs-reranking (English, 3600 pairs):

For scale: scoring the same checkpoint twice gives Pearson 0.99975 and mean |Δscore| 0.0022 (vLLM batching is not bit-deterministic). So A16's score deviation is real but small, and its ranking loss is within noise on both sets.

Independent in-house cross-check

Both quantizations were also run against a private Russian-language retrieval benchmark that is not public and therefore not reproducible from this repository: 100 queries, top-50 candidates from a BM25 first stage, with a relevant document present among the candidates in 89% of cases.

On that set neither quantization lost anything against bf16 — both landed slightly ahead, within noise at 100 queries. Note the ceiling: R@5 = R@10 = 0.89 for every Qwen variant, which is exactly the share of relevant documents the first stage retrieved at all, so the reranker is not what limits the pipeline there.

Treat this as a directional cross-check, not as headline numbers: the corpus is private, the sample is small, and the timings come from a different serving configuration than the table below.

Throughput

RuBQReranking, 3535 pairs, warm run, 4 concurrent requests, identical engine settings:

Which variant should I use?

Both this checkpoint and the W4A4 sibling occupy the same 2.627 GiB on disk, so the memory argument does not separate them. What separates them is the kernel vLLM picks:

• —A16 (this one) — vLLM forces MarlinNvFp4LinearKernel for weight-only NVFP4 (use_a16=True short-circuits kernel selection), and logs "Your GPU does not have native support for FP4 computation… Weight-only FP4 compression will be used". Mature path, best numerical fidelity of the two, ~0.23 GiB/card less runtime memory. Slower than FP8.
• —W4A4 — kernel selection runs normally and finds FlashInferCutlassNvFp4LinearKernel, which does work on SM120. Fastest of everything measured. Slightly lower fidelity, and a measurable −0.008 nDCG@10 on the English set.

Pick A16 if the reranker is a low-QPS auxiliary component and you want the smallest runtime footprint and the most conservative numerics. Pick W4A4 if reranker throughput matters; it is the only 4-bit variant that does not regress against FP8 on speed.

On the private domain set above the two were level on nDCG@10 while W4A4 ran twice as fast per query, so the English-set gap did not transfer there. If you have a domain benchmark, run both — that measurement will decide this better than either table here.

How it was made

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

MODEL = "Qwen/Qwen3-Reranker-4B"  # revision 22e683669bc0f0bd69640a1354a6d0aebcfeede5

model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16, device_map="cpu")
tok = AutoTokenizer.from_pretrained(MODEL)

oneshot(
    model=model,
    recipe=QuantizationModifier(targets="Linear", scheme="NVFP4A16", ignore=["lm_head"]),
    output_dir="Qwen3-Reranker-4B-NVFP4A16",
    processor=tok,
)

Note that in compressed-tensors, scheme="NVFP4" is W4A4 — its input_activations are 4-bit with a static_minmax observer and therefore require calibration data. The data-free weight-only scheme is NVFP4A16, whose input_activations is None. Getting these two confused is easy and the failure is silent.

Ran on CPU in 24 seconds; no GPU and no calibration data required. llm-compressor logs Re-tied input/output embeddings; saving a single shared table — that is the line that keeps the checkpoint at 2.627 GiB instead of 3.35 GiB.

Evaluation methodology

Datasets: `mteb/RuBQReranking` and `mteb/scidocs-reranking`, test splits, 150 queries sampled with seed 20260907, up to 24 candidates per query, documents truncated to 800 characters so that every variant sees byte-identical prompts.

Scores are P("yes") from the 2-way softmax over the "no"/"yes" logits at the final position — identical by construction to what vLLM's from_2_way_softmax pooler computes. The harness was cross-checked against an independent transformers implementation on CPU in float32: Pearson 0.999967, mean |Δscore| 0.0007.

The bf16 baseline was served through the same vLLM engine (with --cpu-offload-gb 2, since 7.48 GiB of weights did not fit in the available VRAM), so all four rows in the tables come from one engine and one prompt builder.

Limitations

• —Throughput figures come from GPUs shared with another workload, with a deliberately tiny KV cache and --enforce-eager. Treat them as a relative ranking, not as absolute performance.
• —Measured only on SM120 (RTX 5090). On SM100 (B200) the kernel selection, and therefore the A16-vs-W4A4 speed relationship, will differ.
• —Evaluated on two reranking benchmarks (one Russian, one English). SciDocs documents are paper titles only (~71 characters on average), so it stresses short-document ranking.
• —150 queries per set bounds the resolution of the absolute nDCG figures; the paired Δ columns are the meaningful comparison.

Compute infrastructure

Quantization — CPU only, no GPU involved. Intel Core i9-14900K (24 cores / 32 threads, AVX2, no AVX-512 and no AMX), 125 GB RAM, PyTorch 2.13 CPU build, llm-compressor 0.13.0 with compressed-tensors 0.18.0. Wall time: 24 seconds. That is the practical argument for the data-free scheme — reproducing this checkpoint needs no accelerator and no calibration corpus. The W4A4 sibling, by contrast, took 104 minutes on the same machine for 512 calibration samples at 512 tokens.

Evaluation and serving. 2× NVIDIA RTX 5090 (32 GB, SM120 / compute capability 12.0), vLLM 0.28.0 on the CUDA 13.0 image, --tensor-parallel-size 2 --enforce-eager. The GPUs were shared with another resident workload for the entire measurement window, which is why the throughput figures are presented as a relative ranking rather than as peak performance, why the KV cache was pinned to 128 MiB, and why the bf16 baseline had to run with --cpu-offload-gb 2 — 7.48 GiB of weights did not fit in what was free.

License and attribution

Apache-2.0, inherited from `Qwen/Qwen3-Reranker-4B`. This repository contains only quantized weights derived from that model; all model credit belongs to the Qwen team. Quantized with llm-compressor.