1kxia/Qwen3-Embedding-0.6B-modelopt-fp8

Qwen3-Embedding-0.6B-modelopt-fp8

FP8 (E4M3) quantized version of Qwen/Qwen3-Embedding-0.6B, quantized using NVIDIA ModelOpt static FP8 quantization.

Model Details

Quantization Details

Method

• —Tool: NVIDIA ModelOpt static FP8 quantization
• —Format: FP8 E4M3 (torch.float8_e4m3fn)
• —Scope: All linear layers (QKV projections, output projections, MLP layers) are quantized to FP8. Embeddings and LayerNorms remain in BF16.
• —Scales: Per-tensor weight scales and input activation scales are stored alongside the quantized weights.

Calibration

• —Dataset: CNN/DailyMail (real text data)
• —Samples: 64
• —Sequence Length: 256
• —Batch Size: 4
• —Activation Scales: Collected at 4 points per layer (post-layernorm, attention output, MLP input, SiLU output), saved in calib.json

Precision Evaluation

Cosine similarity between this FP8 model and the original BF16 model, measured on CNN/DailyMail text inputs (threshold: 0.99):

All configurations achieve >0.99 cosine similarity with the BF16 baseline.

File Structure

.
├── config.json              # Model config with quantization_config
├── model.safetensors        # FP8 quantized weights + scales
├── calib.json               # Activation scales per layer
├── tokenizer.json           # Tokenizer
├── tokenizer_config.json    # Tokenizer config
├── vocab.json               # Vocabulary
├── merges.txt               # BPE merges
└── generation_config.json   # Generation config

Intended Use

This model is intended for efficient FP8 inference of text embeddings on NVIDIA GPUs with FP8 support (Hopper architecture and above).