groxaxo/qwen3-embed-8b-gptq

Qwen3-Embedding-8B-GPTQ (4-bit Quantized)

<!-- polished-overview:start -->

Overview

qwen3-embed-8b-gptq is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by `groxaxo`. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• —*.safetensors (1 file)
• —config.json
• —tokenizer.json
• —tokenizer_config.json
• —chat_template.jinja
• —Additional configuration, tokenizer, processor, or shard files (10 visible artifacts total)

Quick start

vLLM (documented configuration)

CUDA_VISIBLE_DEVICES=0 vllm serve groxaxo/qwen3-embed-8b-gptq \
  --task embed \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.5 \
  --dtype float16 \
  --quantization gptq

This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.

Compatibility and responsible use

• —Use a runtime that explicitly supports this format, architecture, and modality.
• —Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• —Review the source model card and license before redistribution or deployment.
• —Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• —Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws. <!-- polished-overview:end -->

GPTQ W4A16 quantized version of Qwen3-Embedding-8B

• —Original Size: 15GB → Quantized Size: 4.5GB (70% reduction)
• —Semantic Accuracy: 99.14% preserved
• —Context Window: Up to 32k tokens
• —Quantization: GPTQ W4A16 (4-bit weights, 16-bit activations)

Model Details

• —Base Model: Qwen/Qwen3-Embedding-8B
• —Parameters: 8B
• —Embedding Dim: 3072
• —Max Sequence Length: 32768
• —Quantization: GPTQ (4-bit weights, 16-bit activations)
• —Calibration Samples: 128
• —Dataset: ultrachat-200k

Performance

Test Results

Comprehensive testing with 11 test pairs:

• —Mean Absolute Difference: 0.007 (0.86%)
• —Maximum Difference: 0.019 (1.9%)
• —Rating: ✅ EXCELLENT (>95% preserved)

Usage

vLLM (Recommended)

CUDA_VISIBLE_DEVICES=0 vllm serve groxaxo/qwen3-embed-8b-gptq \
  --task embed \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.5 \
  --dtype float16 \
  --quantization gptq

Python

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "groxaxo/qwen3-embed-8b-gptq",
    torch_dtype="float16",
    trust_remote_code=True,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "groxaxo/qwen3-embed-8b-gptq",
    trust_remote_code=True
)

# Generate embeddings
inputs = tokenizer("Your text here", return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model(**inputs)
embeddings = outputs.last_hidden_state.mean(dim=1)

OpenAI-compatible API

curl http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "groxaxo/qwen3-embed-8b-gptq",
    "input": "Your text here"
  }'

Quantization Details

Method: GPTQ (Group-wise Post-Training Quantization)

Configuration:

• —Weights: 4-bit
• —Activations: 16-bit
• —Group Size: 128
• —Scheme: W4A16
• —Damping Factor: Default (auto)

Why GPTQ over AWQ?

• —AWQ calibration produced NaN values for Qwen3-Embedding-8B
• —GPTQ more stable for embedding models
• —GPTQ provides better accuracy for this architecture

Benchmark Results

Inference Speed: ~1.2-1.5x slower than FP16 Memory Usage: ~30% of FP16 model Batch Size: 512 (recommended for vLLM) Throughput: Excellent for batch processing

Recommendations

Use this model when:

• —✅ Semantic accuracy is critical (>95% required)
• —✅ Want 70% memory reduction
• —✅ Production deployment with vLLM
• —✅ Batch processing of embeddings

Consider alternatives when:

• —⚠️ Need <25% memory (use Int8)
• —⚠️ Maximum inference speed (use FP16)
• —⚠️ Edge devices with limited compute

Evaluation

Comprehensive evaluation report available in the repository or QUANTIZATION_REPORT.md.

Test Suite:

• —11 diverse test pairs (similar, different, related concepts)
• —Cosine similarity measurements
• —Statistical analysis of degradation

Limitations

1. 1.Quantization Error: Small accuracy loss (~0.86%)
2. 2.Inference Speed: ~20-50% slower than FP16
3. 3.Not Compatible: With AWQ-optimized servers (use GPTQ)
4. 4.Architecture: Specific to Qwen3-Embedding-8B

Citation

If you use this model, please cite:

@article{qwen2024qwen,
  title={Qwen Technical Report},
  author={Qwen Team},
  year={2024}
}

License

Apache 2.0

Acknowledgments

• —Original Model: Alibaba Qwen Team
• —Quantization Tool: vLLM/llmcompressor
• —Testing: Comprehensive semantic similarity evaluation

Model Card Version: 1.0 Last Updated: February 23, 2026