CoolFace
Modelpublic

dam2452/Qwen3-VL-Embedding-8B-GGUF

sourceHugging Faceapache-2.0updated 9mo agoView on Hugging Face
4likes313downloads
Model Card

Qwen3-VL-Embedding-8B GGUF

GGUF quantizations of Qwen/Qwen3-VL-Embedding-8B for efficient CPU inference with llama.cpp.

Model Description

Qwen3-VL-Embedding-8B is a multimodal embedding model for information retrieval and cross-modal understanding. It supports text, images, screenshots, videos, and mixed multimodal inputs.

Original model specs:

  • —Parameters: 8B
  • —Context Length: 32K tokens
  • —Embedding Dimension: 64-4096 (configurable)
  • —Languages: 30+
  • —Input Modalities: Text, Images, Videos

Available Quantizations

FileSizeUse Case
Qwen3-VL-Embedding-8B-F16.gguf15GBMaximum quality, baseline reference
Qwen3-VL-Embedding-8B-Q8_0.gguf7.5GBRecommended - minimal quality loss
Qwen3-VL-Embedding-8B-Q6_K.gguf5.8GBHigh quality, good balance
Qwen3-VL-Embedding-8B-Q5KM.gguf5.1GBGood quality, balanced size
Qwen3-VL-Embedding-8B-Q5KS.gguf5.0GBGood quality, smaller variant
Qwen3-VL-Embedding-8B-Q4KM.gguf4.4GBDecent quality, smaller size
Qwen3-VL-Embedding-8B-Q4KS.gguf4.2GBDecent quality, more compressed
Qwen3-VL-Embedding-8B-Q3KM.gguf3.6GBLower quality, significant compression
Qwen3-VL-Embedding-8B-Q2_K.gguf2.9GBLowest quality, maximum compression

Recommendation: Start with Q80 for production use. Use Q4KM or Q5K_M for resource-constrained environments.

Usage with llama.cpp

Installation

bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

Download Model

bash
huggingface-cli download dam2452/Qwen3-VL-Embedding-8B-GGUF \
  Qwen3-VL-Embedding-8B-Q8_0.gguf \
  --local-dir ./models

Run Embedding Server

bash
./llama-server \
  -m models/Qwen3-VL-Embedding-8B-Q8_0.gguf \
  --embedding \
  --port 8080 \
  --host 0.0.0.0

Generate Embeddings (API)

bash
curl http://localhost:8080/embedding \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Your text or image data here"
  }'

Generate Embeddings (Python)

python
import requests

response = requests.post(
    "http://localhost:8080/embedding",
    json={"content": "A woman playing with her dog on a beach"}
)

embedding = response.json()["embedding"]
print(f"Embedding dimension: {len(embedding)}")

Performance

Original model performance on benchmarks:

  • —MMEB-V2: 77.9 overall score
  • —MMTEB: 67.88 mean task score
  • —Retrieval: 81.08

Note: Quantized models may show slightly reduced performance, with Q8_0 typically having less than 1% degradation.

License

Apache 2.0 (inherited from original model)

Citation

bibtex
@article{qwen3vlembedding,
  title={Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking},
  author={Li, Mingxin and Zhang, Yanzhao and Long, Dingkun and Chen Keqin and Song, Sibo and Bai, Shuai and Yang, Zhibo and Xie, Pengjun and Yang, An and Liu, Dayiheng and Zhou, Jingren and Lin, Junyang},
  journal={arXiv},
  year={2026}
}

Resources