jsonMartin/voyage-4-nano-ONNX-int8

voyage-4-nano-ONNX-int8

Int8 dynamically quantized ONNX conversion of Voyage AI's voyage-4-nano embedding model for use with Transformers.js and ONNX Runtime.

Model Details

Why Int8?

• —Smaller model size: 329 MB vs 705 MB (FP16) - 53% reduction
• —Faster CPU inference: Int8 weights are faster to process on CPU
• —Maintained quality: ~97% cosine similarity with FP16 model
• —Perfect for edge/mobile: Smaller download, lower memory usage

voyage-4-nano was trained with Quantization-Aware Training (QAT), making it well-suited for int8 quantization with minimal quality loss.

Usage

Transformers.js (Browser/Node.js)

import { pipeline } from '@huggingface/transformers';

const extractor = await pipeline(
  'feature-extraction',
  'YOUR_USERNAME/voyage-4-nano-ONNX-int8'
);

// Document embedding (for indexing)
const docPrefix = "Represent the document for retrieval: ";
const docEmbedding = await extractor(docPrefix + "Your document text", {
  pooling: 'mean',
  normalize: true
});

// Query embedding (for search)
const queryPrefix = "Represent the query for retrieving supporting documents: ";
const queryEmbedding = await extractor(queryPrefix + "Your search query", {
  pooling: 'mean',
  normalize: true
});

ONNX Runtime (Python)

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/voyage-4-nano-ONNX-int8")
session = ort.InferenceSession("model.onnx")

text = "Represent the document for retrieval: Your text here"
inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)

# Create position_ids
seq_len = inputs["input_ids"].shape[1]
position_ids = np.arange(seq_len).reshape(1, -1).astype(np.int64)

outputs = session.run(None, {
    "input_ids": inputs["input_ids"],
    "attention_mask": inputs["attention_mask"],
    "position_ids": position_ids
})

# Mean pooling
embeddings = outputs[0]
mask = inputs["attention_mask"]
pooled = (embeddings * mask[:, :, None]).sum(1) / mask.sum(1, keepdims=True)

# L2 normalize
normalized = pooled / np.linalg.norm(pooled, axis=1, keepdims=True)

Instruction Prefixes

Important: Use the appropriate prefix for best results:

• —Documents (indexing): "Represent the document for retrieval: "
• —Queries (search): "Represent the query for retrieving supporting documents: "

Quantization Details

This model was quantized using ONNX Runtime's dynamic quantization:

• —Quantization type: Dynamic (weights quantized to int8, activations remain float32)
• —Source model: FP16 ONNX (converted to FP32 intermediate for quantization)
• —Target operators: MatMul, Attention layers
• —Quality retention: ~97% cosine similarity vs FP16 original

When to use Int8 vs FP16

Model Comparison

Validation Results

All validation tests pass:

• —Model loads and runs correctly
• —Embedding dimensions: 1024
• —Cosine similarity vs FP16: ~0.97 (average)
• —Semantic ranking preserved
• —L2 normalization correct
• —Transformers.js compatible

License

Apache 2.0 (same as original model)

Acknowledgments

• —Voyage AI for the original model
• —Hugging Face for Transformers.js and ONNX Runtime