kftof/bge-reranker-v2-m3-onnx-int8-avx2

bge-reranker-v2-m3 — ONNX INT8 (CPU / AVX2)

ONNX, INT8-quantized build of `BAAI/bge-reranker-v2-m3`, packaged for Hugging Face Text Embeddings Inference (TEI).

Goal: run this reranking cross-encoder on CPU, without AVX-512 or VNNI, with a smaller memory and disk footprint (~0.55 GB vs ~2.3 GB in FP32).

Quantization details

Target hardware: quantized with the AVX2 preset. Intended for x86-64 CPUs that have AVX2 but no AVX-512/VNNI (e.g. AMD Ryzen Zen/Zen+/Zen2 such as the 2600X). The model runs on any x86-64 CPU; the AVX2 preset only tunes the quantization parameters for that instruction profile.

Accuracy note: reduce_range=True is enabled. It keeps INT8 weights on 7 bits to avoid u8×s8 saturation (the VPMADDUBSW instruction) on AVX2 CPUs without VNNI, as recommended by ONNX Runtime. This trades a negligible amount of headroom for stable accuracy on non-VNNI hardware; on VNNI-capable CPUs it makes no difference.

Usage with TEI

docker run --rm -p 8080:80 \
  ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 \
  --model-id kftof/bge-reranker-v2-m3-onnx-int8-avx2

Call the /rerank endpoint:

curl http://127.0.0.1:8080/rerank \
  -H 'Content-Type: application/json' \
  -d '{
        "query": "What is the capital of France?",
        "texts": ["Paris is the capital of France.",
                  "Chocolate cake recipe"]
      }'

The response ranks the documents by relevance (highest score first).

Reproduce this build

pip install "optimum[exporters,onnxruntime]"
optimum-cli export onnx --model BAAI/bge-reranker-v2-m3 \
  --task text-classification onnx-fp32
optimum-cli onnxruntime quantize --avx2 --onnx_model onnx-fp32 -o onnx-int8
# then rename onnx-int8/model_quantized.onnx -> model.onnx for TEI

License

Inherits the upstream license of BAAI/bge-reranker-v2-m3: Apache-2.0. No re-licensing.