monju-lab/ruri-v3-130m-int8-onnx

ruri-v3-130m-int8-onnx

ONNX export of `cl-nagoya/ruri-v3-130m` (132.1M params, ModernBERT-ja architecture fine-tuned for Japanese sentence embeddings), converted for the feature-extraction task and dynamically quantized to INT8.

An fp32 ONNX reference (505 MB, zero measured quality loss, ~3x faster than PyTorch) was also produced during conversion but isn't included in this upload — ask if you want it added.

Important: this is a prompted embedding model

Ruri v3 uses a 1+3 prefix scheme — prepend one of these to your text before embedding, depending on the use case:

Pooling is mean pooling over last_hidden_state (with attention mask), matching the original Sentence-Transformers 1_Pooling config included in this repo. Output dimensionality is 512.

Benchmark

Measured on a CPU-only sandbox (onnxruntime CPUExecutionProvider), single-sequence inference, over the same 20-sentence Japanese set used for modernbert-ja-130m (no prefix, general semantic similarity setting). Quality is reported against the original PyTorch fp32 model both as mean cosine similarity of individual embeddings and as Spearman rank correlation of the full pairwise similarity matrix.

\* fp16 shows no real latency benefit here because this is a CPU-only benchmark; fp16 is meant for GPU tensor cores. Not included in this upload.

Recommendation: onnx/model_quantized.onnx (int8) gives the best combination of size and CPU latency, at the cost of some embedding precision (~0.99 cosine similarity, 0.94 ranking correlation vs. the original). If your downstream task is sensitive to fine-grained similarity ranking (e.g. re-ranking, nearest-neighbor retrieval with tight margins), validate against your own eval set before relying on it, or ask for the fp32 ONNX variant instead (same latency-class win as modernbert-ja-130m's fp32 export, with zero measured quality loss).

Usage

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
session = ort.InferenceSession("onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])

def embed(sentences, prefix=""):
    texts = [prefix + s for s in sentences]
    enc = tokenizer(texts, padding=True, return_tensors="np")
    last_hidden_state, = session.run(
        None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]}
    )
    mask = enc["attention_mask"][..., None].astype(np.float32)
    summed = (last_hidden_state.astype(np.float32) * mask).sum(axis=1)
    counts = np.clip(mask.sum(axis=1), 1e-9, None)
    return summed / counts

queries = embed(["東京の人口は？"], prefix="検索クエリ: ")
docs = embed(["東京都の人口は約1400万人です。"], prefix="検索文書: ")
cos_sim = (queries[0] @ docs[0]) / (np.linalg.norm(queries[0]) * np.linalg.norm(docs[0]))
print(cos_sim)

Conversion details

• —Exported with optimum (optimum.exporters.onnx), task feature-extraction, opset 18.
• —Dynamic INT8 quantization via onnxruntime.quantization.quantize_dynamic (QInt8 weights).
• —Sentence-Transformers config files (modules.json, config_sentence_transformers.json, sentence_bert_config.json, 1_Pooling/config.json) are carried over from the original repo for reference/compatibility, though this ONNX export is loaded via onnxruntime / optimum directly rather than the sentence_transformers library.

Base model card, license (Apache-2.0), training details, and MTEB scores: cl-nagoya/ruri-v3-130m.