CoolFace
Modelpublic

woxpas-ai/bge-reranker-v2-m3-onnx

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes73downloads
Model Card

bge-reranker-v2-m3-onnx

ONNX export of BAAI/bge-reranker-v2-m3 for use with `onnxruntime` and `transformers.js`. No PyTorch required at inference time.

The original model is a multilingual cross-encoder reranker. Given a (query, passage) pair, the model returns a single relevance logit. Larger logits mean more relevant. Apply sigmoid if you want a [0, 1] probability.

Variants

This repo follows the onnx/ subdir convention used by `Xenova` so that transformers.js can pick a variant automatically.

FilePrecisionOn diskCosine vs PyTorch
onnx/model.onnx (+ model.onnx_data)fp32~2.1 GB1.000000
onnx/model_fp16.onnx (+ model_fp16.onnx.data)fp16~1.1 GB0.99999996
onnx/model_quantized.onnxint8 (dynamic)~544 MB0.99964

The fp32 and fp16 variants use ONNX's external-data format (the .onnx file is a small graph stub plus a sidecar with the weights). Both files in a pair must be downloaded together. The int8 variant is a single file.

Usage

Python (onnxruntime + transformers tokenizer)

python
import onnxruntime as ort
from transformers import AutoTokenizer

repo = "woxpas-ai/bge-reranker-v2-m3-onnx"
tokenizer = AutoTokenizer.from_pretrained(repo)
session = ort.InferenceSession(
    "onnx/model_quantized.onnx",   # or model.onnx / model_fp16.onnx
    providers=["CPUExecutionProvider"],
)

pairs = [
    ("What is database migration?", "We're migrating from MySQL to PostgreSQL"),
    ("How to ride a bike?", "We're migrating from MySQL to PostgreSQL"),
]
queries, docs = zip(*pairs)
enc = tokenizer(list(queries), list(docs), padding=True, truncation=True,
                return_tensors="np", max_length=512)
input_names = {i.name for i in session.get_inputs()}
feed = {k: v for k, v in enc.items() if k in input_names}
logits = session.run(None, feed)[0]
print(logits.reshape(-1))   # higher = more relevant

To download just the variant you need:

python
from huggingface_hub import snapshot_download
snapshot_download(
    "woxpas-ai/bge-reranker-v2-m3-onnx",
    allow_patterns=["onnx/model_quantized.onnx", "*.json", "*.txt", "sentencepiece.bpe.model"],
)

JavaScript (transformers.js)

js
import { AutoTokenizer, AutoModelForSequenceClassification } from "@xenova/transformers";

const repo = "woxpas-ai/bge-reranker-v2-m3-onnx";
const tokenizer = await AutoTokenizer.from_pretrained(repo);
const model = await AutoModelForSequenceClassification.from_pretrained(repo, { quantized: true });

const inputs = await tokenizer(["What is database migration?"], {
  text_pair: ["We're migrating from MySQL to PostgreSQL"],
  padding: true,
  truncation: true,
});
const { logits } = await model(inputs);
console.log(logits.data);

Verification

The export was numerically validated against the PyTorch reference (AutoModelForSequenceClassification.from_pretrained("BAAI/bge-reranker-v2-m3")) on a mixed test set: English query/doc, multilingual translation pair, a negative pair, a topic mismatch, and an empty-query edge case.

fp32

Cosine similarity: 1.00000000   Max |Δ|: 0.000012

  pt=-6.08043   onnx=-6.08044   Δ=-0.000004   What is database migration?         ⇄ MySQL→PostgreSQL
  pt=-10.98419  onnx=-10.98419  Δ=+0.000000   How to ride a bike?                  ⇄ MySQL→PostgreSQL  (negative)
  pt=+10.79895  onnx=+10.79895  Δ=-0.000001   ¿Cómo funciona el modelo?           ⇄ How does the model work?  (multilingual)
  pt=-10.48121  onnx=-10.48121  Δ=+0.000004   memory architecture                  ⇄ knowledge graph extraction
  pt=-10.24049  onnx=-10.24051  Δ=-0.000012   <empty>                              ⇄ empty query edge case

fp16

Cosine similarity: 0.99999996   Max |Δ|: 0.006321

int8 dynamic

Cosine similarity: 0.99964245   Max |Δ|: 0.504600

The int8 variant has measurable per-pair drift (up to ~0.5 logits) but ranking and sign are preserved on this test set; the ordering of relevant vs irrelevant docs is unchanged. For ranking-quality-critical use cases, prefer fp32 or fp16. For latency/size-critical deployments, int8 is generally fine.

The full per-pair numbers are saved in `verification_fp32.json`, `verification_fp16.json`, and `verification_int8.json` in this repo.

Export details

  • —Source model: `BAAI/bge-reranker-v2-m3`
  • —Tooling:
  • —optimum-cli export onnx --task text-classification (fp32)
  • —onnxruntime.transformers.optimizer with convert_float_to_float16(keep_io_types=True) (fp16)
  • —optimum-cli onnxruntime quantize --avx2 (int8 dynamic)
  • —ONNX opset: 14
  • —Library versions used during export:
  • —optimum==1.24.0
  • —transformers==4.48.3
  • —torch==2.2.2
  • —onnxruntime==1.23.2
  • —onnx==1.21.0
  • —Date: 2026-05-04

The --avx2 quantization flag selects the calibration heuristic; the resulting int8 model is portable and runs on any CPU architecture supported by onnxruntime (including arm64 / Apple Silicon), just with lower throughput on CPUs without AVX2.

License

MIT, inherited from the upstream model. See BAAI/bge-reranker-v2-m3.

If you use this model, please cite the original BAAI work:

bibtex
@misc{li2023makinglargelanguagemodels,
  title  = {Making Large Language Models A Better Foundation For Dense Retrieval},
  author = {Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
  year   = {2023},
  eprint = {2312.15503},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url    = {https://arxiv.org/abs/2312.15503},
}