webAI-Official/webAI-ColVec1.1-8b

1---2pipeline_tag: visual-document-retrieval3library_name: transformers4language:5  - multilingual6license: other7license_name: webai-non-commercial-license-v1.08license_link: https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b/blob/main/LICENSE.md9base_model: Qwen/Qwen3.5-9B10datasets:11  - vidore/colpali_train_set12  - Tevatron/docmatix-ir13  - openbmb/VisRAG-Ret-Train-In-domain-data14  - openbmb/VisRAG-Ret-Train-Synthetic-data15  - llamaindex/vdr-multilingual-train16  - Tevatron/wiki-ss-nq17tags:18  - text19  - image20  - multimodal-embedding21  - visual-document-retrieval22  - vidore23  - colbert24  - colqwen3_525  - multilingual-embedding26  - sentence-transformers27  - multi-vector28---29 30# webAI-Official/webAI-ColVec1.1-8b31 32## ⚡ Summary33 34**webAI-Official/webAI-ColVec1.1-8b** is a ColBERT-style multimodal35embedding model based on36[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B). It maps text37queries and visual documents (images or rendered PDF pages) into aligned,38L2-normalized multi-vector embeddings for late-interaction retrieval.39 40The model uses bidirectional attention in Qwen3.5's full-attention layers and41a learned 640-dimensional projection head. The unused language-model head has42been removed from the released checkpoint. As a result, the released43embedding model contains approximately 8.4B parameters, as reflected in the44`webAI-ColVec1.1-8b` repository name, even though its original backbone is45Qwen3.5-9B.46 47### Training data48 49We created filtered, balanced, and multilingual curated subsets from six public50datasets: [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set),51[Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir),52[VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data),53[VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data),54[VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train),55and [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq). This56Qwen3.5-9B-backbone model was trained on a 750,000-sample curated subset as well57as synthetically generated data.58 59## 🛠️ Model specifications60 61| Feature | Detail |62| :--- | :--- |63| **Architecture** | Qwen3.5-9B vision-language model + 640-dimensional linear projection |64| **Released parameters** | 8,395,317,104 |65| **Method** | ColBERT-style late interaction with MaxSim scoring |66| **Output** | L2-normalized multi-vector embeddings `(sequence_length, 640)` |67| **Modalities** | Text queries and document images |68| **Attention** | Bidirectional full-attention layers; selectable SDPA, FlashAttention 2, or FlashAttention 3 kernel |69| **Visual-token budget** | 1,792 tokens per image in the released processor |70| **Training** | LoRA adapters and a fully trained projection layer, merged for release |71| **Weights** | `bfloat16`; language-model head removed |72 73### Key properties74 75- **Unified encoder:** The same model encodes text and document images.76- **Token-level retrieval:** Multi-vector embeddings preserve fine-grained77  layout and content signals that single-vector pooling can discard.78- **Compact projection:** Hidden states are projected to 640 dimensions79  without an activation function.80- **Bidirectional retrieval attention:** Selecting SDPA, FlashAttention 2, or81  FlashAttention 3 changes the execution kernel, not the model's bidirectional82  attention mode.83 84## 📊 Evaluation results85 86The table reports NDCG@10 scores on the ViDoRe V3 tasks as percentages rather87than values between 0 and 1 (for example, 0.80 is shown as 80.00). Each task88value is the mean of its six language subsets. Mean (Public) is the unweighted89mean of the eight public task values, Mean (Private) covers the Nuclear and90Telecom tasks, and the final mean is the unweighted mean across all ten tasks.91Models are sorted by the final mean.92 93The evaluation software versions and setup are documented under94[Reproducing the evaluation environment](#reproducing-the-evaluation-environment).95For the reported evaluation, SDPA was selected as the attention96implementation, the released processor used a 1,792 visual-token budget, and97the batch size was 32.98 99Model encoding used `bfloat16`. Before MaxSim scoring, query and document100embeddings were moved to CPU and converted to `float32`; all reported ViDoRe101results use this FP32 scoring path. Because floating-point calculations and102kernel execution can vary across accelerator hardware, independent103evaluations may produce slightly different results. The submitted MTEB104artifacts are the canonical source for the reported scores.105 106All table values are shown to two decimal places. ColVec1.1 values are rounded107from the submitted artifacts using round-half-up. Comparator values were read108from the live109[ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29)110on July 31, 2026.111 112| Model | **Final mean** | **Mean (Public)** | **Mean (Private)** | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Nuclear | Pharmaceuticals | Physics | Telecom |113| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |114| **webAI-ColVec1.1-8b (this model)** | **64.95** | **65.32** | **63.47** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 53.66 | 67.88 | 51.50 | **73.29** |115| [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 64.26 | 64.72 | 62.43 | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | 53.59 | **68.19** | **51.73** | 71.27 |116| **[webAI-ColVec1.1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b)** | 63.90 | 64.24 | 62.53 | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 53.30 | 67.25 | 51.24 | 71.76 |117| [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 63.57 | 63.72 | 63.00 | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | **54.90** | 67.45 | 50.18 | 71.10 |118| [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 63.42 | 63.54 | 62.92 | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 53.84 | 67.19 | 50.84 | 72.00 |119| [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | 63.00 | 64.45 | 57.20 | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 47.66 | 67.32 | 48.38 | 66.74 |120| [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 62.22 | 63.39 | 57.55 | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 48.22 | 65.68 | 50.15 | 66.88 |121| [Tomoro ColQwen3 Embed 8B](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b) | 61.59 | 61.60 | 61.56 | 75.35 | 68.41 | 65.08 | 49.10 | 63.98 | 54.41 | 52.65 | 66.36 | 50.13 | 70.46 |122 123The current MTEB leaderboard entries named `webAI-ColVec1-4b` and124`webAI-ColVec1-9b` refer to the previous ColVec1 release, not these ColVec1.1125checkpoints.126 127## 💻 Usage128 129### Using Sentence Transformers130 131The checkpoint loads as a `MultiVectorEncoder`, which is available from132Sentence Transformers v6.0.0:133 134```bash135pip install "sentence-transformers[image]>=6.0.0"136```137 138`encode_query` and `encode_document` apply the query and document prompt139formats, the ten query-augmentation tokens, and the L2-normalized140640-dimensional projection. `similarity` computes the MaxSim score matrix.141 142```python143from io import BytesIO144 145import requests146from PIL import Image147from sentence_transformers import MultiVectorEncoder148 149model = MultiVectorEncoder("webAI-Official/webAI-ColVec1.1-8b", trust_remote_code=True)150 151queries = [152    "When was the United States Declaration of Independence proclaimed?",153    "Who printed the edition of Romeo and Juliet?",154]155document_urls = [156    "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg",157    "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg",158]159documents = [160    Image.open(BytesIO(requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30).content))161    for url in document_urls162]163 164query_embeddings = model.encode_query(queries)165document_embeddings = model.encode_document(documents)166print(query_embeddings[0].shape, document_embeddings[0].shape)167# (27, 640) (523, 640)168 169scores = model.similarity(query_embeddings, document_embeddings)170print(scores)171# tensor([[22.1212,  4.7561],172#         [ 6.7469, 22.3831]])173print("Best document per query:", scores.argmax(dim=1))174# Best document per query: tensor([0, 1])175```176 177Documents may also be given as file paths or URLs instead of `PIL.Image`178objects. The Wikimedia URLs above are fetched with a browser `User-Agent`179because Wikimedia rejects the default one. Both encode methods accept a180`batch_size`, and the loading options are forwarded through `model_kwargs`181(`dtype`, `attn_implementation`, `device_map`) and `processor_kwargs`182(`max_num_visual_tokens`):183 184```python185model = MultiVectorEncoder(186    "webAI-Official/webAI-ColVec1.1-8b",187    trust_remote_code=True,188    model_kwargs={"attn_implementation": "sdpa", "device_map": "cuda:0"},189    processor_kwargs={"max_num_visual_tokens": 1024},190)191```192 193The scores above come from the plain load, which uses the checkpoint's194`bfloat16` weights and SDPA. MaxSim is accumulated in `float32` rather195than in the embedding dtype, matching the scoring path behind the reported196evaluation, so a `bfloat16` `score_retrieval` call on the same embeddings197returns coarser values. The scores still move in the second decimal place198across PyTorch and Transformers builds. Text passed to `encode_document`199is rendered as a query, because the model defines no text-document format,200and a warning says so.201 202### Using transformers203 204The processor provides the current retrieval API:205 206- `process_images(images)` prepares one or more document images.207- `process_queries(texts)` prepares one or more natural-language queries.208- `score_retrieval(query_embeddings, document_embeddings)` computes a MaxSim209  score matrix with shape `(number_of_queries, number_of_documents)`.210 211### Quick start212 213Create and activate a Python 3.12 virtual environment, then install the214validated PyTorch CUDA 12.8 build and the minimal packages required for SDPA215inference. If you are already using an isolated Python environment, skip the216first two commands.217 218```bash219python3.12 -m venv .venv220source .venv/bin/activate221 222python -m pip install \223  torch==2.9.0 torchvision==0.24.0 \224  --index-url https://download.pytorch.org/whl/cu128225 226python -m pip install \227  "transformers>=5.14.1,<6.0.0" \228  accelerate pillow requests safetensors229```230 231The following example uses SDPA, the portable default and the attention232implementation used for the reported evaluation. It does not require233FlashAttention.234 235```python236from io import BytesIO237 238import requests239import torch240from PIL import Image241from transformers import AutoModel, AutoProcessor242 243MODEL_ID = "webAI-Official/webAI-ColVec1.1-8b"244DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu"245 246# Portable default and the backend used by the published evaluation:247ATTN_IMPLEMENTATION = "sdpa"248 249processor = AutoProcessor.from_pretrained(250    MODEL_ID,251    trust_remote_code=True,252    max_num_visual_tokens=1792,253)254model = AutoModel.from_pretrained(255    MODEL_ID,256    trust_remote_code=True,257    dtype=torch.bfloat16,258    attn_implementation=ATTN_IMPLEMENTATION,259    device_map=DEVICE,260).eval()261 262queries = [263    "When was the United States Declaration of Independence proclaimed?",264    "Who printed the edition of Romeo and Juliet?",265]266document_urls = [267    "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg",268    "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg",269]270 271 272def load_image(url: str) -> Image.Image:273    response = requests.get(274        url,275        headers={"User-Agent": "Mozilla/5.0"},276        timeout=30,277    )278    response.raise_for_status()279    return Image.open(BytesIO(response.content)).convert("RGB")280 281 282device = next(model.parameters()).device283query_inputs = processor.process_queries(queries)284document_inputs = processor.process_images(285    [load_image(url) for url in document_urls]286)287query_inputs = {288    key: value.to(device) if isinstance(value, torch.Tensor) else value289    for key, value in query_inputs.items()290}291document_inputs = {292    key: value.to(device) if isinstance(value, torch.Tensor) else value293    for key, value in document_inputs.items()294}295 296with torch.inference_mode():297    query_batch = model(**query_inputs)298    document_batch = model(**document_inputs)299 300query_embeddings = [embedding.cpu() for embedding in query_batch]301document_embeddings = [embedding.cpu() for embedding in document_batch]302scores = processor.score_retrieval(303    query_embeddings,304    document_embeddings,305    output_dtype=torch.float32,306)307 308print(scores)309print("Best document per query:", scores.argmax(dim=1))310```311 312The released processor uses a 1,792 visual-token budget by default. To reduce313memory use, pass a lower `max_num_visual_tokens` value to314`AutoProcessor.from_pretrained`; this changes document granularity and may315change retrieval scores.316 317### Optional acceleration318 319The model can run with compatible PyTorch and CUDA builds using SDPA,320FlashAttention 2, or FlashAttention 3. CPU execution is supported through321PyTorch's SDPA math fallback, but is generally impractical for a model of this322size.323 324Qwen3.5 uses a hybrid stack of full-attention and GatedDeltaNet325linear-attention layers. These kernels serve different parts of the model:326 327- `flash-attn` can accelerate the full-attention layers when selected.328- `causal-conv1d` and `flash-linear-attention` (`fla`) accelerate the329  GatedDeltaNet layers. Transformers can fall back to PyTorch implementations330  without them.331- `tilelang` provides optimized GPU kernels for some FLA operations. FLA uses332  these kernels when the operation and hardware are supported and uses another333  implementation otherwise. Pin `apache-tvm-ffi<0.1.10` alongside it to keep334  TileLang's TVM dependency compatible.335 336[flash-attn](https://github.com/Dao-AILab/flash-attention) and337[causal-conv1d](https://github.com/Dao-AILab/causal-conv1d) ship as prebuilt338wheels tied to a specific Python, PyTorch, CUDA, and C++ ABI combination, so339install the build that matches your environment;340[flash-linear-attention](https://github.com/fla-org/flash-linear-attention) and341[TileLang](https://github.com/tile-ai/tilelang) install from PyPI. The exact342versions used for the reported scores are pinned in343[Reproducing the evaluation environment](#reproducing-the-evaluation-environment).344 345`causal-conv1d`, `flash-linear-attention`, and `tilelang` are detected346automatically once installed, so the GatedDeltaNet layers need no configuration347change. Only the full-attention backend is selected explicitly, as a one-line348change to the model loading code in [Quick start](#quick-start):349 350```python351ATTN_IMPLEMENTATION = "flash_attention_2"  # or "flash_attention_3"352```353 354FlashAttention 2 and FlashAttention 3 both preserve the model's bidirectional355attention, and FlashAttention 3 can improve throughput on Hopper GPUs356(H100/H200). PyTorch's built-in SDPA remains the more portable choice because357it does not require a separate FlashAttention package or ABI-compatible wheel.358Changing the full-attention implementation may introduce small floating-point359differences and affect only those layers, not the GatedDeltaNet layers.360 361### Reproducing the evaluation environment362 363The complete pinned Python environment is provided in364[`evaluation-requirements-cu128.txt`](./evaluation-requirements-cu128.txt),365which reproduces the recorded Linux x86-64, CPython 3.12, CUDA 12.8, and366PyTorch 2.9 environment used for evaluation. Its pinned wheel URLs are specific367to that platform, so a different environment needs matching wheels or a source368build.369 370Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), ensure371Git is available, and then run:372 373```bash374uv venv --python 3.12 .venv375source .venv/bin/activate376 377uv pip install \378  torch==2.9.0 torchvision==0.24.0 \379  --index-url https://download.pytorch.org/whl/cu128380 381uv pip install -r evaluation-requirements-cu128.txt382uv pip check383```384 385The requirements include FlashAttention 2 as an optional supported backend;386its presence does not change the SDPA configuration used for the reported387scores.388 389The ViDoRe V3 scores used the following core software versions:390 391```text392Python 3.12393PyTorch 2.9.0 + CUDA 12.8394Transformers 5.14.1395MTEB 2.18.6 (commit d56a414b45ebad0d03495de000b4880d8b028d4a)396Sentence Transformers 5.6.0397causal-conv1d 1.6.2.post1398flash-linear-attention 0.5.1399TileLang 0.1.9400Attention implementation: SDPA401```402 403## ⚖️ Strengths and limitations404 405### Strengths406 407- **Performance:** State-of-the-art retrieval performance on the public ViDoRe408  V3 tasks, with excellent multimodal document retrieval results.409- **Complex layouts:** Excellent handling of chart-rich PDFs and410  domain-specific documents.411- **End-to-end retrieval:** OCR-free retrieval on unseen multimodal documents412  without using an intermediate vision-language model to generate summaries.413- **Multilingualism:** Strong performance on non-English document inputs.414 415### Limitations416 417- **Storage cost:** Still larger than single-vector baselines despite the418  smaller token dimension.419 420## License421 422Model weights are distributed under the423[webAI Non-Commercial License v1.0](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b/blob/main/LICENSE.md).424See the repository's `NOTICES.md` for upstream attribution.425 426## 📚 Citation427 428```bibtex429@misc{webai_colvec1_1_8b,430  title  = {webAI-ColVec1.1-8b: A Bidirectional Multi-Vector Model for Visual Document Retrieval},431  author = {webAI},432  year   = {2026},433  url    = {https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b}434}435```436