CoolFace
Modelpublic

kzzalews/Nemotron-3-Embed-1B-BF16-onnx

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
1likes25downloads
Model Card

NVIDIA Nemotron-3 Embed 1B BF16 — ONNX

ONNX export of `nvidia/Nemotron-3-Embed-1B-BF16` for use with fastembed-rs and onnxruntime.

Released 2026-07-16. #1 on RTEB (Retrieval Text Embedding Benchmark) at 1B parameter scale — 27% improvement over the previous Nemotron embedding model.

Model details

PropertyValue
Base modelnvidia/Nemotron-3-Embed-1B-BF16
ArchitectureMinistral3 (no trustremotecode required)
Parameters~1B
Embedding dimension2048
PoolingMean pooling over token hidden states
Export dtypefloat32 (default) / float16 (fp16/ subfolder)
Opset17

Files

PathSizeDescription
model.onnx~2 MBONNX graph (float32)
model.onnx.data~4.25 GBExternal weights (float32)
fp16/model.onnx~2 MBONNX graph (float16)
fp16/model.onnx.data~2.63 GBExternal weights (float16)
tokenizer.json16 MBTokenizer (Mistral BPE)

Usage with onnxruntime

python
import onnxruntime as ort
import numpy as np
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("kzzalews/Nemotron-3-Embed-1B-BF16-onnx")
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

def embed(texts: list[str]) -> np.ndarray:
    enc = tokenizer(texts, return_tensors="np", padding=True, truncation=True)
    hidden = sess.run(["last_hidden_state"], {
        "input_ids": enc["input_ids"],
        "attention_mask": enc["attention_mask"],
    })[0]
    mask = enc["attention_mask"][:, :, np.newaxis].astype(np.float32)
    return (hidden * mask).sum(axis=1) / mask.sum(axis=1)

# query prefix required for retrieval
embeddings = embed(["query: How does Kubernetes handle OOMKill events?"])
print(embeddings.shape)  # (1, 2048)

fastembed-rs integration (pending PR)

float32 variant:

rust
// src/models/text_embedding.rs
ModelInfo {
    model: EmbeddingModel::NvidiaNemotron3Embed1BBF16,
    dim: 2048,
    description: String::from("NVIDIA Nemotron-3 Embed 1B BF16 — #1 RTEB, float32"),
    model_code: String::from("kzzalews/Nemotron-3-Embed-1B-BF16-onnx"),
    model_file: String::from("model.onnx"),
    additional_files: vec!["model.onnx.data".to_string()],
    output_key: None,
},

// src/text_embedding/impl.rs
EmbeddingModel::NvidiaNemotron3Embed1BBF16 => Some(Pooling::Mean),

float16 variant (2.63 GB, identical quality, same CPU speed):

rust
ModelInfo {
    model: EmbeddingModel::NvidiaNemotron3Embed1BBF16Fp16,
    dim: 2048,
    description: String::from("NVIDIA Nemotron-3 Embed 1B BF16 — #1 RTEB, float16"),
    model_code: String::from("kzzalews/Nemotron-3-Embed-1B-BF16-onnx"),
    model_file: String::from("fp16/model.onnx"),
    additional_files: vec!["fp16/model.onnx.data".to_string()],
    output_key: None,
},

EmbeddingModel::NvidiaNemotron3Embed1BBF16Fp16 => Some(Pooling::Mean),

Variants

VariantPathSizeQuality
float32 (default)model.onnx + model.onnx.data4.25 GBmax_diff vs PyTorch = 1.14e-04
float16fp16/model.onnx + fp16/model.onnx.data2.63 GBcos_sim = 1.000000 vs float32
Note: INT8 dynamic quantization was tested on the predecessor model and rejected (cos_sim = 0.65–0.70 without calibration — unusable for retrieval). Float16 saves ~40% space but does not improve CPU inference speed (CPU emulates fp16 without AVX-512 FP16 hardware).

Performance (CPU: Intel i9-13950HX, 16 threads, no GPU)

VariantShort text (~11 tokens)
float32~129ms
float16~128ms (CPU emulation — no speedup without AVX-512 FP16)

Numerical validation (ONNX float32 vs PyTorch): max diff = 1.14e-04, mean diff ≈ 1e-05.

Prompts

This model uses instruction prefixes for retrieval:

  • —Queries: "query: <text>"
  • —Passages/documents: "passage: <text>"

See also