ldwformat/jina-embeddings-v3-Q8-onnx

Jina Embeddings v3 — ONNX Int8 Quantized

Multilingual text embedding model (jinaai/jina-embeddings-v3) exported to ONNX and dynamically quantized to Int8 for CPU-friendly inference.

Overview

• —Base model: jinaai/jina-embeddings-v3 (XLM-R, 24 layers, hidden_size=1024).
• —Tasks: retrieval.query / retrieval.passage / text-matching / classification / separation.
• —Matryoshka dims: 32 / 64 / 128 / 256 / 512 / 768 / 1024 (you can truncate to any of these).
• —Files: model.onnx.

Quantization details

• —Tooling: optimum.onnxruntime ORTQuantizer.
• —Method: dynamic quantization (is_static=False), QuantType.QInt8, operators: MatMul / Attention / Gather / Gemm.
• —Export: opset 22, dynamic batch/seq, inputs: input_ids, attention_mask, task_id.

Quickstart

pip install onnxruntime transformers

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

model_id = "ldwformat/jina-embeddings-v3-Q8-onnx"  # this folder
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
session = ort.InferenceSession(f"{repo_path}/model.onnx")

def encode(text, task_id=0):
    inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)
    feed = {
        "input_ids": inputs["input_ids"].astype(np.int64),
        "attention_mask": inputs["attention_mask"].astype(np.int64),
        "task_id": np.array([task_id], dtype=np.int64),
    }
    last_hidden = session.run(None, feed)[0]          # [B, L, H]
    mask = np.expand_dims(feed["attention_mask"], -1) # [B, L, 1]
    summed = (last_hidden * mask).sum(axis=1)         # [B, H]
    lengths = np.clip(mask.sum(axis=1), 1e-9, None)
    pooled = summed / lengths                         # mean pooling
    emb = pooled[0]
    return emb / np.linalg.norm(emb)

# Example: query (task_id=0) vs passage (task_id=1)
q = encode("How to fix a phone battery?", task_id=0)
d = encode("A detailed guide to replacing a mobile device battery.", task_id=1)

Tip: choose task_id per task (0=query, 1=passage; see task_instructions in config). For Matryoshka use-cases, truncate to the leading N dims (e.g., emb[:256]).

Evaluation and self-check

• —Script qt/jina-quality-check.py tests semantic separation on a small CN example, quantization fidelity vs FP32, and 256-dim truncation robustness.
• —For production, rerun evaluations on your retrieval/matching datasets and compare against the FP32 model.

License

• —Inherits the upstream jinaai/jina-embeddings-v3 license; follow the original terms.