groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16

Languages / Idiomas: 🇬🇧 English · 🇪🇸 Español

English

<!-- polished-overview:start -->

Overview

Nemotron-3-Embed-1B-AWQ-W4A16 is a weight-quantized checkpoint intended for efficient GPU inference, published by `groxaxo`. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• —*.safetensors (1 file)
• —config.json
• —tokenizer.json
• —tokenizer_config.json
• —Additional configuration, tokenizer, processor, or shard files (23 visible artifacts total)

Quick start

vLLM (AWQ-compatible runtimes)

vllm serve groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16 \
  --quantization awq_marlin \
  --dtype float16 \
  --trust-remote-code

The exact kernel and flags depend on the quantizer and architecture. Check the files and source model card before selecting a production serving configuration.

Compatibility and responsible use

• —Use a runtime that explicitly supports this format, architecture, and modality.
• —Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• —Review the source model card and license before redistribution or deployment.
• —Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• —Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws. <!-- polished-overview:end -->

Nemotron-3-Embed-1B-AWQ-W4A16

Quality-maxed 4-bit AWQ of NVIDIA's Nemotron-3-Embed-1B — half the size, 0.0 pp Recall@10 loss on English AND on Spanish, despite being calibrated English-only.

1.0 GB on disk · 2048-dim retrieval encoder · vLLM Marlin W4A16 accelerated · G32 asymmetric · MSE observer · 1024 calibration samples

This is a W4A16 AWQ rebuild of `nvidia/Nemotron-3-Embed-1B-BF16` using `llmcompressor` 0.12.0 and serialized as compressed-tensors for native vLLM/Marlin serving. Every Linear weight is INT4 (group 32, asymmetric, zero-point) with BF16 activations; the 131 072 × 2 048 token-embedding table and all RMSNorms stay BF16 to protect semantic fidelity.

The result: 56 % smaller, 0.0 percentage-point Recall@10 loss versus BF16 across two languages (English PAQ and Spanish MIRACL-ES, 512 held-out pairs each) and three dimensions (2 048, 1 024, 512). Mean BF16↔AWQ vector cosine sits at 0.995–0.997 everywhere.

Why this checkpoint

G32 asymmetric is the quality-max target inside W4A16: it minimizes per-group reconstruction error and is the smallest group size vLLM's Marlin W4A16 path will accept. Pairing it with an MSE observer that searches 200 clipping candidates over a 20 % shrink range gives the lowest published-style reconstruction error you can reach without going to W8A16 or BF16.

Quick start — serve with vLLM

pip install "vllm==0.25.0"

vllm serve groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16 \
  --host 0.0.0.0 --port 8000 \
  --served-model-name nemotron-embed \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --max-num-batched-tokens 4096 \
  --max-cudagraph-capture-size 4096

Do not pass `--quantization awq`. The checkpoint already carries compressed-tensors metadata and vLLM picks the W4A16 Marlin kernel automatically.

Query the /v2/embed endpoint:

import requests

r = requests.post(
    "http://localhost:8000/v2/embed",
    json={
        "model": "nemotron-embed",
        "input_type": "query",          # or "document" / "passage"
        "texts": ["How do I rotate an IAM access key?"],
        "embedding_types": ["float"],
        "truncate": "END",
    },
    timeout=60,
).json()

vec = r["embeddings"]["float"][0]        # 2048-dim, L2-normalized

The model expects query: / passage: prefixes; vLLM applies them from input_type via the bundled Sentence Transformers metadata — do not prefix the strings yourself.

Quick start — transformers / sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16")

queries  = ["¿Cómo renuevo una clave de acceso IAM?", "How do I rotate an IAM key?"]
passages = ["Para rotar una clave IAM, primero crea una segunda clave...",
            "To rotate an IAM access key, first create a second key..."]

q = model.encode(queries,  prompt_name="query",    normalize_embeddings=True)
d = model.encode(passages, prompt_name="document", normalize_embeddings=True)

scores = q @ d.T   # 2x2 cosine-similarity matrix

Benchmarks — BF16 vs AWQ, English + Spanish

Two completely separate held-out evaluations, both run on a single RTX 3090 through vLLM 0.25.0:

• —English — 512 query/passage pairs from sentence-transformers/paq, disjoint from the English-only calibration set (seed 999, deduped against the seed-42 calibration slice).
• —Spanish — 512 query/passage pairs from Hieuman/es_MIRACL (MIRACL-ES), disjoint seed 999. No Spanish was seen during AWQ calibration — this measures how the English-calibrated checkpoint transfers to a second language.

Headline numbers (TL;DR):

• —✅ Recall@10 = 1.0000 in every cell, every language, every dimension. The positive passage is always in the top-10 for both BF16 and AWQ.
• —✅ English: literally identical to BF16 — 0.0 pp on Recall@1, Recall@10, MRR@10 at all three dimensions.
• —✅ Spanish (never seen during calibration): −0.2 pp Recall@1, −0.1 pp MRR@10 at 2 048-d and 512-d, 0.0 pp at 1 024-d. All within the suggested 0.5 pp gate.
• —✅ Mean BF16↔AWQ vector cosine 0.9947–0.9971 across both languages and all dimensions.

Full machine-readable reports: `validation_report_en.json`, `validation_report_es.json`. Methodology and reproduction: `BENCHMARKS.md`.

Caveat — read this before deploying: both PAQ and MIRACL-ES positives are easy (positive passage is nearly always rank-1 for both BF16 and AWQ), so the deltas above are lower bounds on quality. They prove the build is sound (no broken layers, no silently-loaded BF16 weights, no language-specific collapse), not that AWQ will be lossless on your traffic. Re-validate on your own held-out production pairs (queries + passages, with realistic length/language/domain mix) before committing to a recall budget. NVIDIA's NVFP4 derivative benchmarks NDCG@10 72.00 vs 72.38 BF16 on RTEB — expect AWQ to land in the same ballpark.

What's in the box

Source scripts (quantize_nemotron_awq.py, prepare_paq_calibration.py, prepare_bilingual_calibration.py, validate_awq_embeddings.py, dump_and_validate_gpu2.py) live alongside this README.

Reproduce this exact checkpoint

Single-GPU (RTX 3090, ~35 min wall-clock):

python3.12 -m venv ~/.venvs/nemotron-awq
source ~/.venvs/nemotron-awq/bin/activate
pip install "llmcompressor==0.12.0" "datasets" "huggingface_hub[hf_xet]"

./prepare_paq_calibration.py --output paq.jsonl --num-pairs 4096 --seed 42

CUDA_VISIBLE_DEVICES=0 python ./quantize_nemotron_awq.py \
  --model nvidia/Nemotron-3-Embed-1B-BF16 \
  --calibration-jsonl ./paq.jsonl \
  --output ./Nemotron-3-Embed-1B-AWQ-W4A16-G32-ASYM \
  --num-samples 1024 --max-seq-length 4096 \
  --group-size 32 --asymmetric --observer mse \
  --n-grid 100 --mse-grid 200 --mse-maxshrink 0.20 --mse-patience 40 --mse-norm 2.4 \
  --attn-implementation sdpa --overwrite

For 3× 3090s use torchrun --nproc_per_node=3 (see `BENCHMARKS.md`). The build settings and exact CLI are also persisted in `awq_build_settings.json`.

Spanish + English bilingual rebuild

This checkpoint was calibrated on English-only PAQ and already transfers to Spanish with only −0.2 pp Recall@1 (see benchmarks above). If your traffic is majority Spanish and you want to recover that last 0.2 pp, rebuild with the bilingual recipe in `RECIPE_BILINGUAL.md` — it pulls 50 % sentence-transformers/paq (English) and 50 % Hieuman/es_MIRACL (native Spanish Wikipedia QA), so AWQ scale search sees real Spanish retrieval prefixes and the model's response to Spanish activation distributions.

./prepare_bilingual_calibration.py \
  --output bilingual_calibration.jsonl \
  --num-pairs-en  512 \
  --num-pairs-es  512 \
  --seed 42

CUDA_VISIBLE_DEVICES=0 python ./quantize_nemotron_awq.py \
  --model nvidia/Nemotron-3-Embed-1B-BF16 \
  --calibration-jsonl ./bilingual_calibration.jsonl \
  --output ./Nemotron-3-Embed-1B-AWQ-W4A16-G32-ASYM-EN-ES \
  --num-samples 1024 --max-seq-length 4096 \
  --group-size 32 --asymmetric --observer mse \
  --n-grid 100 --mse-grid 200 --mse-maxshrink 0.20 --mse-patience 40 --mse-norm 2.4 \
  --attn-implementation sdpa --overwrite

How this differs from NVIDIA's NVFP4

NVFP4 is the right choice if you have Hopper+ and want maximum throughput; this AWQ build is the right choice for Ampere/Ada (RTX 3090/4090, A10/A100) where Marlin W4A16 is the fastest 4-bit path.

License

This model is a derivative of `nvidia/Nemotron-3-Embed-1B-BF16` and inherits its [OpenMDW-1.1](https://github.com/nvidia/OpenMDW-1.1/blob/main/OpenMDW-1.1.txt) license. The LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md files are preserved unchanged. Attribution: this checkpoint was produced by Facu Vlad J (HuggingFace groxaxo) using llmcompressor 0.12.0.

The accompanying Python scripts in this repository are MIT-licensed; see each file header.

Citation

If you use this checkpoint, please cite both NVIDIA's Nemotron model card and this rebuild:

@misc{nvidia_nemotron_3_embed_1b,
  title  = {Nemotron-3-Embed-1B-BF16},
  author = {NVIDIA Corporation},
  url    = {https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16},
  year   = {2026}
}

@misc{groxaxo_nemotron_3_embed_awq,
  title  = {Quality-maxed W4A16 AWQ rebuild of Nemotron-3-Embed-1B},
  author = {Vlad J, Facu (groxaxo)},
  url    = {https://huggingface.co/groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16},
  year   = {2026},
  note   = {Built with llmcompressor 0.12.0; compressed-tensors format for vLLM/Marlin}
}

Español

Nemotron-3-Embed-1B-AWQ-W4A16

AWQ de 4 bits de NVIDIA Nemotron-3-Embed-1B con calidad máxima — la mitad de tamaño, 0.0 pp de pérdida en Recall@10 en inglés Y en español, a pesar de estar calibrado únicamente en inglés.

1.0 GB en disco · encoder de recuperación de 2048 dimensiones · acelerado con vLLM Marlin W4A16 · G32 asimétrico · observador MSE · 1024 muestras de calibración

Esta es una reconstrucción W4A16 AWQ de `nvidia/Nemotron-3-Embed-1B-BF16` usando `llmcompressor` 0.12.0, serializada como compressed-tensors para servirse nativamente con vLLM/Marlin. Cada peso Linear está en INT4 (grupo 32, asimétrico, con zero-point) con activaciones en BF16; la tabla de embeddings de tokens (131 072 × 2 048) y todas las RMSNorms permanecen en BF16 para proteger la fidelidad semántica.

El resultado: 56 % más chico, 0.0 puntos porcentuales de pérdida en Recall@10 frente a BF16 en dos idiomas (inglés con PAQ y español con MIRACL-ES, 512 pares held-out cada uno) y tres dimensionalidades (2 048, 1 024, 512). El coseno vectorial BF16↔AWQ se mantiene entre 0.995 y 0.997 en todas partes.

Por qué este checkpoint

G32 asimétrico es el objetivo de calidad máxima dentro de W4A16: minimiza el error de reconstrucción por grupo y es el group size más chico que acepta el path W4A16 Marlin de vLLM. Combinarlo con un observador MSE que prueba 200 candidatos de clipping sobre un rango de shrink del 20 % entrega el menor error de reconstrucción publicable que se puede alcanzar sin irse a W8A16 o BF16.

Inicio rápido — servir con vLLM

pip install "vllm==0.25.0"

vllm serve groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16 \
  --host 0.0.0.0 --port 8000 \
  --served-model-name nemotron-embed \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --max-num-batched-tokens 4096 \
  --max-cudagraph-capture-size 4096

No pasar `--quantization awq`. El checkpoint ya trae metadata de compressed-tensors y vLLM selecciona el kernel W4A16 Marlin automáticamente.

Consultar el endpoint /v2/embed:

import requests

r = requests.post(
    "http://localhost:8000/v2/embed",
    json={
        "model": "nemotron-embed",
        "input_type": "query",          # o "document" / "passage"
        "texts": ["How do I rotate an IAM access key?"],
        "embedding_types": ["float"],
        "truncate": "END",
    },
    timeout=60,
).json()

vec = r["embeddings"]["float"][0]        # 2048-dim, L2-normalizado

El modelo espera los prefijos query: / passage: ; vLLM los aplica desde input_type usando la metadata de Sentence Transformers incluida — no agregar los prefijos a mano.

Inicio rápido — transformers / sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16")

queries  = ["¿Cómo renuevo una clave de acceso IAM?", "How do I rotate an IAM key?"]
passages = ["Para rotar una clave IAM, primero crea una segunda clave...",
            "To rotate an IAM access key, first create a second key..."]

q = model.encode(queries,  prompt_name="query",    normalize_embeddings=True)
d = model.encode(passages, prompt_name="document", normalize_embeddings=True)

scores = q @ d.T   # matriz 2x2 de similaridad coseno

Benchmarks — BF16 vs AWQ, inglés + español

Dos evaluaciones held-out completamente separadas, ambas corridas en una sola RTX 3090 con vLLM 0.25.0:

• —Inglés — 512 pares query/passage de sentence-transformers/paq, disjuntos del set de calibración inglés (seed 999, deduplicados contra la partición calibrada con seed 42).
• —Español — 512 pares query/passage de Hieuman/es_MIRACL (MIRACL-ES), seed 999 disjunto. No se usó nada de español durante la calibración AWQ — esto mide cómo el checkpoint calibrado en inglés se transfiere a un segundo idioma.

Números destacados (TL;DR):

• —✅ Recall@10 = 1.0000 en todas las celdas, en todos los idiomas, en todas las dimensiones. El passage correcto siempre aparece en el top-10 tanto para BF16 como para AWQ.
• —✅ Inglés: idéntico a BF16 — 0.0 pp en Recall@1, Recall@10 y MRR@10 en las tres dimensiones.
• —✅ Español (nunca visto durante la calibración): −0.2 pp Recall@1, −0.1 pp MRR@10 en 2 048-d y 512-d, 0.0 pp en 1 024-d. Todos dentro del gate sugerido de 0.5 pp.
• —✅ Coseno vectorial BF16↔AWQ entre 0.9947 y 0.9971 en ambos idiomas y todas las dimensiones.

Reportes completos legibles por máquina: `validation_report_en.json`, `validation_report_es.json`. Metodología y reproducción: `BENCHMARKS.md`.

Advertencia — leer antes de desplegar a producción: los positives de PAQ y MIRACL-ES son fáciles (el passage correcto casi siempre es rank-1 tanto para BF16 como para AWQ), así que los deltas de arriba son cotas inferiores de calidad. Demuestran que el build es sano (sin capas rotas, sin pesos BF16 cargados silenciosamente, sin colapso por idioma), no que AWQ vaya a ser lossless en tu tráfico. Re-validá con tus propios pares held-out de producción (queries + passages, con mix realista de longitud/idioma/dominio) antes de comprometer un presupuesto de recall. La derivada NVFP4 de NVIDIA publica NDCG@10 72.00 vs 72.38 BF16 en RTEB — AWQ debería caer en el mismo ballpark.

Qué hay en la caja

Los scripts fuente (quantize_nemotron_awq.py, prepare_paq_calibration.py, prepare_bilingual_calibration.py, validate_awq_embeddings.py, dump_and_validate_gpu2.py) viven junto a este README.

Reproducir este checkpoint exacto

Una sola GPU (RTX 3090, ~35 min reloj):

python3.12 -m venv ~/.venvs/nemotron-awq
source ~/.venvs/nemotron-awq/bin/activate
pip install "llmcompressor==0.12.0" "datasets" "huggingface_hub[hf_xet]"

./prepare_paq_calibration.py --output paq.jsonl --num-pairs 4096 --seed 42

CUDA_VISIBLE_DEVICES=0 python ./quantize_nemotron_awq.py \
  --model nvidia/Nemotron-3-Embed-1B-BF16 \
  --calibration-jsonl ./paq.jsonl \
  --output ./Nemotron-3-Embed-1B-AWQ-W4A16-G32-ASYM \
  --num-samples 1024 --max-seq-length 4096 \
  --group-size 32 --asymmetric --observer mse \
  --n-grid 100 --mse-grid 200 --mse-maxshrink 0.20 --mse-patience 40 --mse-norm 2.4 \
  --attn-implementation sdpa --overwrite

Para 3× 3090 usar torchrun --nproc_per_node=3 (ver `BENCHMARKS.md`). Los settings del build y el CLI exacto también están persistidos en `awq_build_settings.json`.

Rebuild bilingüe español + inglés

Este checkpoint se calibró sobre PAQ inglés únicamente y ya se transfiere a español con apenas −0.2 pp de Recall@1 (ver benchmarks arriba). Si tu tráfico es mayoritariamente español y querés recuperar esos últimos 0.2 pp, reconstruilo con la receta bilingüe en `RECIPE_BILINGUAL.md` — toma 50 % sentence-transformers/paq (inglés) y 50 % Hieuman/es_MIRACL (QA nativo de Wikipedia en español), así la búsqueda de escalas AWQ ve prefijos de recuperación reales en español y la respuesta del modelo a las distribuciones de activación del español.

./prepare_bilingual_calibration.py \
  --output bilingual_calibration.jsonl \
  --num-pairs-en  512 \
  --num-pairs-es  512 \
  --seed 42

CUDA_VISIBLE_DEVICES=0 python ./quantize_nemotron_awq.py \
  --model nvidia/Nemotron-3-Embed-1B-BF16 \
  --calibration-jsonl ./bilingual_calibration.jsonl \
  --output ./Nemotron-3-Embed-1B-AWQ-W4A16-G32-ASYM-EN-ES \
  --num-samples 1024 --max-seq-length 4096 \
  --group-size 32 --asymmetric --observer mse \
  --n-grid 100 --mse-grid 200 --mse-maxshrink 0.20 --mse-patience 40 --mse-norm 2.4 \
  --attn-implementation sdpa --overwrite

Diferencias frente al NVFP4 de NVIDIA

NVFP4 es la opción correcta si tenés Hopper+ y querés máximo throughput; este build AWQ es la opción correcta para Ampere/Ada (RTX 3090/4090, A10/A100), donde Marlin W4A16 es el path de 4 bits más rápido.

Licencia

Este modelo es un derivado de `nvidia/Nemotron-3-Embed-1B-BF16` y hereda su licencia [OpenMDW-1.1](https://github.com/nvidia/OpenMDW-1.1/blob/main/OpenMDW-1.1.txt). Los archivos LICENSE, NOTICE y THIRD_PARTY_NOTICES.md se preservan sin cambios. Atribución: este checkpoint fue producido por Facu Vlad J (HuggingFace groxaxo) usando llmcompressor 0.12.0.

Los scripts Python adjuntos en este repositorio están bajo licencia MIT; ver el header de cada archivo.

Cita

Si usás este checkpoint, citá tanto la card del modelo Nemotron de NVIDIA como este rebuild:

@misc{nvidia_nemotron_3_embed_1b,
  title  = {Nemotron-3-Embed-1B-BF16},
  author = {NVIDIA Corporation},
  url    = {https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16},
  year   = {2026}
}

@misc{groxaxo_nemotron_3_embed_awq,
  title  = {Quality-maxed W4A16 AWQ rebuild of Nemotron-3-Embed-1B},
  author = {Vlad J, Facu (groxaxo)},
  url    = {https://huggingface.co/groxaxo/Nemotron-3-Embed-1B-AWQ-W4A16},
  year   = {2026},
  note   = {Built with llmcompressor 0.12.0; compressed-tensors format for vLLM/Marlin}
}