ConfidentialMind/gte-multilingual-reranker-base-onnx-op14-opt-gpu-int8

gte-multilingual-reranker-base-onnx-op14-opt-gpu-int8-quantized

This model is a quantized ONNX version of Alibaba-NLP/gte-multilingual-reranker-base using ONNX opset 14.

Model Details

• —Quantization Type: INT8
• —ONNX Opset: 14
• —Task: text-classification
• —Target Device: GPU
• —Optimized: Yes
• —Framework: ONNX Runtime
• —Original Model: Alibaba-NLP/gte-multilingual-reranker-base
• —Quantized On: 2025-03-27
• —Author This model was modified by Jaro

Environment and Package Versions

Applied Optimizations

Usage

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

# Load model and tokenizer
model = ORTModelForSequenceClassification.from_pretrained("quantized_model")
tokenizer = AutoTokenizer.from_pretrained("quantized_model")

# Prepare input
text = "Your text here"
inputs = tokenizer(text, return_tensors="pt")

# Run inference
outputs = model(**inputs)

Quantization Process

This model was quantized using ONNX Runtime with int8 quantization. The quantization was performed using the Optimum library from Hugging Face with opset 14. Graph optimization was applied during export, targeting GPU devices.

Performance Comparison

Quantized models generally offer better inference speed with a slight trade-off in accuracy. This INT8 quantized model should provide significantly faster inference than the original model.