Nicolassuez/bekko-embedding-v1-a25m-onnx-q8

bekko-embedding-v1-a25m, ONNX INT8

Pre-quantized INT8 ONNX export of hotchpotch/bekko-embedding-v1-a25m (ModernBERT, 384-dim mean-pooled sentence embeddings, no prompt prefix, 8192-token context, MIT license).

All credit for the model goes to hotchpotch; this repository only adds the onnx/model_quantized.onnx file that quantized runtimes such as transformers.js resolve for dtype: 'q8' (the upstream repository ships the fp32 export only, so a q8 request 404s there).

The fp32 weights are not duplicated here: use the upstream repository for those.

How it was built

Dynamic INT8 weight quantization, no calibration data:

from onnxruntime.quantization import QuantType, quantize_dynamic
quantize_dynamic("onnx/model.onnx", "onnx/model_quantized.onnx", weight_type=QuantType.QUInt8)

QUInt8 rather than QInt8: both score identically on the downstream benchmark below, but unsigned is measurably more faithful to the fp32 reference (mean cosine 0.98817 vs 0.98600 over 438 texts, and worst case 0.974 vs 0.954), at equal speed on the hosts tested.

Note that only half of this model is quantizable: embeddings.tok_embeddings.weight (98.3M values, the 256k-token vocabulary table) is already INT8 in the upstream fp32 export, and only the 25.2M float transformer values are affected. That is why the file shrinks 1.6x (199 MB to 124 MB) rather than the usual 4x, and why the speedup is ~1.2x rather than 2x.

Measured impact

Quantization is quality-neutral on our workload. Measured on a semantic routing benchmark (kNN over a 345-example labelled prototype set, 250 held-out queries across ~30 languages, plus an adversarial out-of-domain set), with guard floors calibrated on the fp32 build and left untouched:

No recalibration was needed: the guard floors fitted on fp32 transfer unchanged. Individual cases do flip in both directions (quantization moves a vector by ~1.2% of cosine, enough to tip a query sitting exactly on a decision threshold), so read this as "no measurable degradation" rather than as an improvement. Speed gain is ~1.2x short text / ~1.27x long text, limited by the already-INT8 embedding table described above.

Usage (transformers.js)

import { pipeline } from '@huggingface/transformers'
const extract = await pipeline('feature-extraction', 'Nicolassuez/bekko-embedding-v1-a25m-onnx-q8', { dtype: 'q8' })
const vec = await extract('query text', { pooling: 'mean', normalize: true })