gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit

Qwen3-VL-Embedding-8B — AWQ INT4 (W4A16)

4-bit AWQ quantization of `Qwen/Qwen3-VL-Embedding-8B`, exported in the compressed-tensors format for fast serving with vLLM (Marlin INT4 kernels on Ampere/Ada/Hopper and Jetson Orin sm_87).

The model produces dense multimodal embeddings (text, image, video) in a shared 4096-dim space using last-token pooling. This quant keeps the entire vision tower in BF16 and only quantizes the language decoder, which preserves embedding quality (see Evaluation).

What is and isn't quantized

Only the language-model `Linear` layers are quantized to INT4. Quantizing the vision encoder of a VLM embedding model causes severe "cosine drift", so the following are explicitly excluded and remain bit-identical to the BF16 reference:

ignore = ["re:.*lm_head", "re:.*visual.*", "re:.*vision_tower.*",
          "re:.*merger.*", "re:.*patch_embed.*"]

The AWQ smoothing uses the standard Qwen3 mappings (input_layernorm→q/k/v, v→o, post_attention_layernorm→gate/up, up→down). The v_proj→o_proj smoothing is skipped per layer because of grouped-query attention (different head dims) — this is expected and harmless.

Usage

vLLM (recommended — OpenAI-compatible /v1/embeddings)

vllm serve gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit \
  --runner pooling --convert embed \
  --max-model-len 8192 --gpu-memory-utilization 0.40

Notes - vLLM ≥ 0.15 uses --convert embed (older docs say --task embed). - Do not pass --load-format fastsafetensors — it regresses multimodal embedding similarity (vLLM issue #33954). - compressed-tensors quantization is auto-detected; no --quantization flag needed.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
e = client.embeddings.create(model="gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit",
                             input=["A dog on the beach", "A cat on a sofa"])
print(len(e.data[0].embedding))  # 4096

Transformers / compressed-tensors

import torch
from transformers import AutoConfig, AutoProcessor, Qwen3VLForConditionalGeneration

repo = "gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit"
cfg = AutoConfig.from_pretrained(repo); cfg.tie_word_embeddings = True
model = Qwen3VLForConditionalGeneration.from_pretrained(
    repo, config=cfg, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

msgs = [{"role": "user", "content": [{"type": "text", "text": "Represent this sentence."}]}]
text = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inp = processor(text=[text], return_tensors="pt").to(model.device)
with torch.no_grad():
    h = model(**inp, output_hidden_states=True).hidden_states[-1][:, -1, :]
emb = torch.nn.functional.normalize(h, dim=-1)  # last-token pooling

(compressed-tensors must be installed to load the packed INT4 weights.)

Calibration

Activation statistics were collected with llm-compressor (AWQModifier + QuantizationModifier, duo_scaling, n_grid=20) using the sequential pipeline (one decoder subgraph at a time).

Evaluation

BF16 vs. this AWQ model, both served through vLLM, comparing /v1/embeddings output on mixed EN/PL text:

The similarity geometry is preserved — similar inputs stay close, unrelated inputs stay far — so retrieval ranking matches the BF16 model. (Validation shown here is text-only; the vision tower is unquantized BF16 and runs identically to the base model.)

Reproduce

Quantized with llm-compressor 0.11 on an NVIDIA Jetson AGX Orin (JetPack 6.2, CUDA 12.6). One upstream workaround is required before loading the base model: set tie_word_embeddings = true to handle the missing lm_head.weight shard (Qwen3-VL-Embedding bug #89). The full Dockerized recipe (quantize.py, recipe, serve script) is included as recipe.yaml in this repo.

Limitations

• —INT4 introduces ~2-3% per-vector cosine drift vs BF16; for most RAG retrieval this does not change top-k ranking, but exact-similarity-threshold workflows should re-tune thresholds.
• —Validation above is text-only.
• —Inherits all capabilities and limitations of the base Qwen/Qwen3-VL-Embedding-8B.

License & citation

Released under Apache-2.0, inherited from the base model. Please cite the original Qwen3-VL-Embedding work and credit Qwen/Qwen3-VL-Embedding-8B as the base model.