webAI-Official/webAI-ColVec1.1-4b

1---2pipeline_tag: visual-document-retrieval3library_name: transformers4language:5  - multilingual6license: other7license_name: webai-non-commercial-license-v1.08license_link: https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b/blob/main/LICENSE.md9base_model: Qwen/Qwen3.5-4B10datasets:11  - vidore/colpali_train_set12  - Tevatron/docmatix-ir13  - openbmb/VisRAG-Ret-Train-In-domain-data14  - openbmb/VisRAG-Ret-Train-Synthetic-data15  - llamaindex/vdr-multilingual-train16  - Tevatron/wiki-ss-nq17tags:18  - text19  - image20  - multimodal-embedding21  - visual-document-retrieval22  - vidore23  - colbert24  - colqwen3_525  - multilingual-embedding26  - sentence-transformers27  - multi-vector28---29 30# webAI-Official/webAI-ColVec1.1-4b31 32## ⚡ Summary33 34**webAI-Official/webAI-ColVec1.1-4b** is a ColBERT-style multimodal35embedding model based on36[Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). It maps text37queries and visual documents (images or rendered PDF pages) into aligned,38L2-normalized multi-vector embeddings for late-interaction retrieval.39 40The model uses bidirectional attention in Qwen3.5's full-attention layers and41a learned 640-dimensional projection head. The unused language-model head has42been removed from the released checkpoint.43 44### Training data45 46We created filtered, balanced, and multilingual curated subsets from six public47datasets: [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set),48[Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir),49[VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data),50[VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data),51[VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train),52and [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq). This53Qwen3.5-4B-backbone model was trained on a 500,000-sample curated subset as well54as synthetically generated data.55 56## 🛠️ Model specifications57 58| Feature | Detail |59| :--- | :--- |60| **Architecture** | Qwen3.5-4B vision-language model + 640-dimensional linear projection |61| **Released parameters** | 4,540,904,576 |62| **Method** | ColBERT-style late interaction with MaxSim scoring |63| **Output** | L2-normalized multi-vector embeddings `(sequence_length, 640)` |64| **Modalities** | Text queries and document images |65| **Attention** | Bidirectional full-attention layers; selectable SDPA, FlashAttention 2, or FlashAttention 3 kernel |66| **Visual-token budget** | 1,792 tokens per image in the released processor |67| **Training** | LoRA adapters and a fully trained projection layer, merged for release |68| **Weights** | `bfloat16`; language-model head removed |69 70### Key properties71 72- **Unified encoder:** The same model encodes text and document images.73- **Token-level retrieval:** Multi-vector embeddings preserve fine-grained74  layout and content signals that single-vector pooling can discard.75- **Compact projection:** Hidden states are projected to 640 dimensions76  without an activation function.77- **Bidirectional retrieval attention:** Selecting SDPA, FlashAttention 2, or78  FlashAttention 3 changes the execution kernel, not the model's bidirectional79  attention mode.80 81## 📊 Evaluation results82 83The table reports NDCG@10 scores on the ViDoRe V3 tasks as percentages rather84than values between 0 and 1 (for example, 0.80 is shown as 80.00). Each task85value is the mean of its six language subsets. Mean (Public) is the unweighted86mean of the eight public task values, Mean (Private) covers the Nuclear and87Telecom tasks, and the final mean is the unweighted mean across all ten tasks.88Models are sorted by the final mean.89 90The evaluation software versions and setup are documented under91[Reproducing the evaluation environment](#reproducing-the-evaluation-environment).92For the reported evaluation, SDPA was selected as the attention93implementation, the released processor used a 1,792 visual-token budget, and94the batch size was 32.95 96Model encoding used `bfloat16`. Before MaxSim scoring, query and document97embeddings were moved to CPU and converted to `float32`; all reported ViDoRe98results use this FP32 scoring path. Because floating-point calculations and99kernel execution can vary across accelerator hardware, independent100evaluations may produce slightly different results. The submitted MTEB101artifacts are the canonical source for the reported scores.102 103All table values are shown to two decimal places. ColVec1.1 values are rounded104from the submitted artifacts using round-half-up. Comparator values were read105from the live106[ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29)107on July 31, 2026.108 109| Model | **Final mean** | **Mean (Public)** | **Mean (Private)** | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Nuclear | Pharmaceuticals | Physics | Telecom |110| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |111| **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | **64.95** | **65.32** | **63.47** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 53.66 | 67.88 | 51.50 | **73.29** |112| [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 64.26 | 64.72 | 62.43 | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | 53.59 | **68.19** | **51.73** | 71.27 |113| **webAI-ColVec1.1-4b (this model)** | 63.90 | 64.24 | 62.53 | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 53.30 | 67.25 | 51.24 | 71.76 |114| [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 63.57 | 63.72 | 63.00 | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | **54.90** | 67.45 | 50.18 | 71.10 |115| [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 63.42 | 63.54 | 62.92 | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 53.84 | 67.19 | 50.84 | 72.00 |116| [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | 63.00 | 64.45 | 57.20 | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 47.66 | 67.32 | 48.38 | 66.74 |117| [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 62.22 | 63.39 | 57.55 | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 48.22 | 65.68 | 50.15 | 66.88 |118| [Tomoro ColQwen3 Embed 8B](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b) | 61.59 | 61.60 | 61.56 | 75.35 | 68.41 | 65.08 | 49.10 | 63.98 | 54.41 | 52.65 | 66.36 | 50.13 | 70.46 |119 120The current MTEB leaderboard entries named `webAI-ColVec1-4b` and121`webAI-ColVec1-9b` refer to the previous ColVec1 release, not these ColVec1.1122checkpoints.123 124## 💻 Usage125 126### Using Sentence Transformers127 128The checkpoint loads as a `MultiVectorEncoder`, which is available from129Sentence Transformers v6.0.0:130 131```bash132pip install "sentence-transformers[image]>=6.0.0"133```134 135`encode_query` and `encode_document` apply the query and document prompt136formats, the ten query-augmentation tokens, and the L2-normalized137640-dimensional projection. `similarity` computes the MaxSim score matrix.138 139```python140from io import BytesIO141 142import requests143from PIL import Image144from sentence_transformers import MultiVectorEncoder145 146model = MultiVectorEncoder("webAI-Official/webAI-ColVec1.1-4b", trust_remote_code=True)147 148queries = [149    "When was the United States Declaration of Independence proclaimed?",150    "Who printed the edition of Romeo and Juliet?",151]152document_urls = [153    "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg",154    "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg",155]156documents = [157    Image.open(BytesIO(requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30).content))158    for url in document_urls159]160 161query_embeddings = model.encode_query(queries)162document_embeddings = model.encode_document(documents)163print(query_embeddings[0].shape, document_embeddings[0].shape)164# (27, 640) (523, 640)165 166scores = model.similarity(query_embeddings, document_embeddings)167print(scores)168# tensor([[23.3935,  5.7660],169#         [ 4.8504, 23.1851]])170print("Best document per query:", scores.argmax(dim=1))171# Best document per query: tensor([0, 1])172```173 174Documents may also be given as file paths or URLs instead of `PIL.Image`175objects. The Wikimedia URLs above are fetched with a browser `User-Agent`176because Wikimedia rejects the default one. Both encode methods accept a177`batch_size`, and the loading options are forwarded through `model_kwargs`178(`dtype`, `attn_implementation`, `device_map`) and `processor_kwargs`179(`max_num_visual_tokens`):180 181```python182model = MultiVectorEncoder(183    "webAI-Official/webAI-ColVec1.1-4b",184    trust_remote_code=True,185    model_kwargs={"attn_implementation": "sdpa", "device_map": "cuda:0"},186    processor_kwargs={"max_num_visual_tokens": 1024},187)188```189 190The scores above come from the plain load, which uses the checkpoint's191`bfloat16` weights and SDPA. MaxSim is accumulated in `float32` rather192than in the embedding dtype, matching the scoring path behind the reported193evaluation, so a `bfloat16` `score_retrieval` call on the same embeddings194returns coarser values. The scores still move in the second decimal place195across PyTorch and Transformers builds. Text passed to `encode_document`196is rendered as a query, because the model defines no text-document format,197and a warning says so.198 199### Using transformers200 201The processor provides the current retrieval API:202 203- `process_images(images)` prepares one or more document images.204- `process_queries(texts)` prepares one or more natural-language queries.205- `score_retrieval(query_embeddings, document_embeddings)` computes a MaxSim206  score matrix with shape `(number_of_queries, number_of_documents)`.207 208### Quick start209 210Create and activate a Python 3.12 virtual environment, then install the211validated PyTorch CUDA 12.8 build and the minimal packages required for SDPA212inference. If you are already using an isolated Python environment, skip the213first two commands.214 215```bash216python3.12 -m venv .venv217source .venv/bin/activate218 219python -m pip install \220  torch==2.9.0 torchvision==0.24.0 \221  --index-url https://download.pytorch.org/whl/cu128222 223python -m pip install \224  "transformers>=5.14.1,<6.0.0" \225  accelerate pillow requests safetensors226```227 228The following example uses SDPA, the portable default and the attention229implementation used for the reported evaluation. It does not require230FlashAttention.231 232```python233from io import BytesIO234 235import requests236import torch237from PIL import Image238from transformers import AutoModel, AutoProcessor239 240MODEL_ID = "webAI-Official/webAI-ColVec1.1-4b"241DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu"242 243# Portable default and the backend used by the published evaluation:244ATTN_IMPLEMENTATION = "sdpa"245 246processor = AutoProcessor.from_pretrained(247    MODEL_ID,248    trust_remote_code=True,249    max_num_visual_tokens=1792,250)251model = AutoModel.from_pretrained(252    MODEL_ID,253    trust_remote_code=True,254    dtype=torch.bfloat16,255    attn_implementation=ATTN_IMPLEMENTATION,256    device_map=DEVICE,257).eval()258 259queries = [260    "When was the United States Declaration of Independence proclaimed?",261    "Who printed the edition of Romeo and Juliet?",262]263document_urls = [264    "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg",265    "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg",266]267 268 269def load_image(url: str) -> Image.Image:270    response = requests.get(271        url,272        headers={"User-Agent": "Mozilla/5.0"},273        timeout=30,274    )275    response.raise_for_status()276    return Image.open(BytesIO(response.content)).convert("RGB")277 278 279device = next(model.parameters()).device280query_inputs = processor.process_queries(queries)281document_inputs = processor.process_images(282    [load_image(url) for url in document_urls]283)284query_inputs = {285    key: value.to(device) if isinstance(value, torch.Tensor) else value286    for key, value in query_inputs.items()287}288document_inputs = {289    key: value.to(device) if isinstance(value, torch.Tensor) else value290    for key, value in document_inputs.items()291}292 293with torch.inference_mode():294    query_batch = model(**query_inputs)295    document_batch = model(**document_inputs)296 297query_embeddings = [embedding.cpu() for embedding in query_batch]298document_embeddings = [embedding.cpu() for embedding in document_batch]299scores = processor.score_retrieval(300    query_embeddings,301    document_embeddings,302    output_dtype=torch.float32,303)304 305print(scores)306print("Best document per query:", scores.argmax(dim=1))307```308 309The released processor uses a 1,792 visual-token budget by default. To reduce310memory use, pass a lower `max_num_visual_tokens` value to311`AutoProcessor.from_pretrained`; this changes document granularity and may312change retrieval scores.313 314### Optional acceleration315 316The model can run with compatible PyTorch and CUDA builds using SDPA,317FlashAttention 2, or FlashAttention 3. CPU execution is supported through318PyTorch's SDPA math fallback, but is generally impractical for a model of this319size.320 321Qwen3.5 uses a hybrid stack of full-attention and GatedDeltaNet322linear-attention layers. These kernels serve different parts of the model:323 324- `flash-attn` can accelerate the full-attention layers when selected.325- `causal-conv1d` and `flash-linear-attention` (`fla`) accelerate the326  GatedDeltaNet layers. Transformers can fall back to PyTorch implementations327  without them.328- `tilelang` provides optimized GPU kernels for some FLA operations. FLA uses329  these kernels when the operation and hardware are supported and uses another330  implementation otherwise. Pin `apache-tvm-ffi<0.1.10` alongside it to keep331  TileLang's TVM dependency compatible.332 333[flash-attn](https://github.com/Dao-AILab/flash-attention) and334[causal-conv1d](https://github.com/Dao-AILab/causal-conv1d) ship as prebuilt335wheels tied to a specific Python, PyTorch, CUDA, and C++ ABI combination, so336install the build that matches your environment;337[flash-linear-attention](https://github.com/fla-org/flash-linear-attention) and338[TileLang](https://github.com/tile-ai/tilelang) install from PyPI. The exact339versions used for the reported scores are pinned in340[Reproducing the evaluation environment](#reproducing-the-evaluation-environment).341 342`causal-conv1d`, `flash-linear-attention`, and `tilelang` are detected343automatically once installed, so the GatedDeltaNet layers need no configuration344change. Only the full-attention backend is selected explicitly, as a one-line345change to the model loading code in [Quick start](#quick-start):346 347```python348ATTN_IMPLEMENTATION = "flash_attention_2"  # or "flash_attention_3"349```350 351FlashAttention 2 and FlashAttention 3 both preserve the model's bidirectional352attention, and FlashAttention 3 can improve throughput on Hopper GPUs353(H100/H200). PyTorch's built-in SDPA remains the more portable choice because354it does not require a separate FlashAttention package or ABI-compatible wheel.355Changing the full-attention implementation may introduce small floating-point356differences and affect only those layers, not the GatedDeltaNet layers.357 358### Reproducing the evaluation environment359 360The complete pinned Python environment is provided in361[`evaluation-requirements-cu128.txt`](./evaluation-requirements-cu128.txt),362which reproduces the recorded Linux x86-64, CPython 3.12, CUDA 12.8, and363PyTorch 2.9 environment used for evaluation. Its pinned wheel URLs are specific364to that platform, so a different environment needs matching wheels or a source365build.366 367Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), ensure368Git is available, and then run:369 370```bash371uv venv --python 3.12 .venv372source .venv/bin/activate373 374uv pip install \375  torch==2.9.0 torchvision==0.24.0 \376  --index-url https://download.pytorch.org/whl/cu128377 378uv pip install -r evaluation-requirements-cu128.txt379uv pip check380```381 382The requirements include FlashAttention 2 as an optional supported backend;383its presence does not change the SDPA configuration used for the reported384scores.385 386The ViDoRe V3 scores used the following core software versions:387 388```text389Python 3.12390PyTorch 2.9.0 + CUDA 12.8391Transformers 5.14.1392MTEB 2.18.6 (commit d56a414b45ebad0d03495de000b4880d8b028d4a)393Sentence Transformers 5.6.0394causal-conv1d 1.6.2.post1395flash-linear-attention 0.5.1396TileLang 0.1.9397Attention implementation: SDPA398```399 400## ⚖️ Strengths and limitations401 402### Strengths403 404- **Performance:** State-of-the-art retrieval performance among 4B models on405  the public ViDoRe V3 tasks, with excellent multimodal document retrieval406  results.407- **Complex layouts:** Excellent handling of chart-rich PDFs and408  domain-specific documents.409- **End-to-end retrieval:** OCR-free retrieval on unseen multimodal documents410  without using an intermediate vision-language model to generate summaries.411- **Multilingualism:** Strong performance on non-English document inputs.412 413### Limitations414 415- **Storage cost:** Still larger than single-vector baselines despite the416  smaller token dimension.417 418## License419 420Model weights are distributed under the421[webAI Non-Commercial License v1.0](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b/blob/main/LICENSE.md).422See the repository's `NOTICES.md` for upstream attribution.423 424## 📚 Citation425 426```bibtex427@misc{webai_colvec1_1_4b,428  title  = {webAI-ColVec1.1-4b: A Bidirectional Multi-Vector Model for Visual Document Retrieval},429  author = {webAI},430  year   = {2026},431  url    = {https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b}432}433```434