groxaxo/Nemotron-3-Embed-8B-AWQ-W4A16-G32-ASYM

Nemotron 3 Embed 8B — 4-bit AWQ (W4A16, G32, Asymmetric)

<!-- polished-overview:start -->

Overview

Nemotron-3-Embed-8B-AWQ-W4A16-G32-ASYM is a weight-quantized checkpoint intended for efficient GPU inference, published by `groxaxo`. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• —*.safetensors (1 file)
• —config.json
• —tokenizer.json
• —tokenizer_config.json
• —Additional configuration, tokenizer, processor, or shard files (13 visible artifacts total)

Quick start

vLLM (AWQ-compatible runtimes)

vllm serve groxaxo/Nemotron-3-Embed-8B-AWQ-W4A16-G32-ASYM \
  --quantization awq_marlin \
  --dtype float16 \
  --trust-remote-code

The exact kernel and flags depend on the quantizer and architecture. Check the files and source model card before selecting a production serving configuration.

Compatibility and responsible use

• —Use a runtime that explicitly supports this format, architecture, and modality.
• —Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• —Review the source model card and license before redistribution or deployment.
• —Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• —Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws. <!-- polished-overview:end -->

A quality-first, deployment-ready 4-bit quantization of NVIDIA's multilingual 8B embedding model.

This checkpoint turns NVIDIA Nemotron-3-Embed-8B-BF16 into a single 5.36 GB safetensors artifact using asymmetric AWQ with a fine-grained group size of 32. It retains 4096-dimensional embeddings, loads natively through vLLM's compressed-tensors path, and was validated with real embedding requests on an RTX 3090.

This is an independently produced community quantization. It is not an official NVIDIA release.

Why this build

• —4-bit weights, BF16 activations — materially lower storage and weight memory than the BF16 source.
• —Quality-first G32 quantization — smaller groups preserve more local weight detail than coarser AWQ configurations.
• —Native vLLM serving — embedded compressed-tensors metadata enables Marlin-compatible execution without a manual quantization flag.
• —Reproducible provenance — the exact source revision, calibration recipe, and quantizer settings ship with the model.
• —Measured, not merely converted — validated for finite 4096D output, BF16-to-AWQ cosine fidelity, and code-retrieval quality.

Quick start with vLLM

Validated with vllm==0.25.0:

pip install "vllm==0.25.0" openai

vllm serve groxaxo/Nemotron-3-Embed-8B-AWQ-W4A16-G32-ASYM \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name nemotron-embed-8b-awq \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --max-num-batched-tokens 4096

Do not add --quantization awq. vLLM detects the checkpoint's compressed-tensors metadata automatically.

Embed queries and documents with the model's required role prefixes:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

response = client.embeddings.create(
    model="nemotron-embed-8b-awq",
    input=[
        "query: How do I rotate an API key?",
        "passage: API keys can be rotated from the security settings page.",
    ],
)

query_vector, passage_vector = [item.embedding for item in response.data]
print(len(query_vector))  # 4096

For retrieval, use query: for queries and passage: for candidate documents. Compare L2-normalized vectors with cosine similarity or dot product.

Validation results

Quantization fidelity

The checkpoint was loaded with vLLM 0.25.0 on an RTX 3090 and served a real embeddings request. A four-vector BF16-versus-AWQ smoke comparison produced:

These cosine figures measure output fidelity to the BF16 checkpoint; they are not a substitute for a full retrieval benchmark.

Code-retrieval evaluation

A controlled evaluation used the same five-repository corpus for both models: 10,403 function/method chunks and 200 labeled queries.

On this workload, retrieval quality was effectively tied. The 8B model should therefore be chosen for workloads where its larger capacity demonstrates an advantage on representative data—not simply because it has more parameters. Results are specific to this custom code corpus and are not RTEB scores.

Dimensionality and index compatibility

The native output is 4096 dimensions. The upstream model supports client-side prefix slicing (for example, keeping the first 2048 dimensions) followed by L2 normalization. However, in the validated vLLM 0.25.0 configuration, requesting output_dimension=2048 was rejected; client-side slicing after inference worked.

Do not mix 4096D and 2048D vectors in one index. Re-embed the corpus, slice queries and documents identically, normalize after slicing, and measure retrieval quality before migrating a production index.

Quantization details

See `awq_build_settings.json` and `recipe.yaml` for the machine-readable build configuration.

Intended use

This model is intended for multilingual retrieval, semantic search, dense indexing, clustering, and RAG systems that can serve a compressed 8B encoder and store 4096D vectors. Evaluate it on your own corpus before production use, especially for domain-specific, safety-critical, or latency-sensitive applications.

License and attribution

The upstream model and its configuration are licensed under the OpenMDW License Agreement 1.1. Review that license and the original NVIDIA model card before use. This repository contains a separately produced quantization of the pinned upstream checkpoint.