sanjay920/voyage-4-nano-mlx-6bit

voyage-4-nano-mlx-6bit

`voyageai/voyage-4-nano` converted for MLX, so it runs on Apple silicon. Smallest size that keeps cosine above 0.996.

voyage-4-nano is not served by the Voyage API — the open checkpoint is the only way to run it. Embeddings from this port land in the same space as the hosted voyage-4 models, so you can index locally and query with those: same-text cosine 0.969 against voyage-4-lite, 0.952 against voyage-4, 0.894 against voyage-4-large (12 texts, versus 0.32–0.36 for non-matching text).

Unofficial community port. Not affiliated with Voyage AI or MongoDB.

Use

pip install voyage-4-nano-mlx

from voyage_4_nano_mlx import load

emb = load("sanjay920/voyage-4-nano-mlx-6bit")

q = emb.encode_query("Which planet is known as the Red Planet?")
d = emb.encode_document([
    "Venus is often called Earth's twin because of its similar size.",
    "Mars, known for its reddish appearance, is called the Red Planet.",
])
emb.similarity(q, d)          # [[0.4052 0.6514]]

Queries and documents take different trained prefixes; encode_query and encode_document apply them. Use encode() for symmetric tasks.

Matryoshka dims (2048 / 1024 / 512 / 256) and compact outputs:

emb.encode_document(docs, dims=256)                        # 256-d, unit norm
emb.encode_document(docs, dims=512, output_dtype="int8")   # 512 int8 values
emb.encode_document(docs, output_dtype="ubinary")          # 2048 bits = 256 bytes

To shrink an index, prefer dims=256 + int8 over a smaller weight file — it cuts the index 32x and leaves the encoder faithful.

Variants

Weight quantization saves disk, not time: at typical sequence lengths this model is compute-bound, and 8-bit and 4-bit both run ~7% slower than bf16.

Verification

Against the reference PyTorch implementation, with a float64 run as ground truth:

On MLX's CPU backend the unquantized port matches float64 as closely as PyTorch's own float32 does (max error 1.03e-04 vs 1.07e-04). Tokenization is byte-identical to the reference across 14 texts covering CJK, emoji, URLs and whitespace edge cases. Method and full numbers: docs/PARITY.md.

Model

A Qwen3 backbone run bidirectionally, not an encoder-only model:

Qwen3, 12 layers, d=1024, 16 heads / 8 KV heads, head_dim=128
RMSNorm, SwiGLU, RoPE theta=1e6, per-head q/k RMSNorm, bidirectional attention
-> final norm
-> linear 1024 -> 2048   (per token, before pooling)
-> mean pool over the attention mask
-> L2 normalize

346M parameters, 32k context, multilingual.

Notes

• —Metal's float32 matmul accumulates at reduced precision, so GPU fp32 is slightly less exact than CPU fp32 (0.9999995 vs 1.0000000 cosine). It does not affect retrieval.
• —32k-token inputs use dense attention and take ~6 s each.

Source, tests and benchmarks: https://github.com/sanjay920/voyage-4-nano-mlx