CoolFace
Modelpublic

VAGOsolutions/SauerkrautLM-ColQwen3-2b-v0.1

sourceHugging Faceapache-2.0updated 25d agoView on Hugging Face
4likes411downloads
Model Card

SauerkrautLM-ColQwen3-2b-v0.1

<p align="center"> <img src="https://vago-solutions.ai/wp-content/uploads/2025/12/Sauerkrautlm-colpali-scaled.png" alt="VAGO Solutions Logo" width="75%"/> </p>

🥇 Best 128-dim Model in Medium (1-3B) Category | +1.01 over ColQwen2

SauerkrautLM-ColQwen3-2b-v0.1 achieves 90.24 NDCG@5 on ViDoRe v1, making it the #1 in the Medium (1-3B) category among 128-dim models - a significant +1.01 improvement over the baseline ColQwen2-v1.0.

<p align="center"> <img src="https://raw.githubusercontent.com/VAGOsolutions/sauerkrautlm-colpali/main/assets/benchmark128dimv1.png" alt="ViDoRe v1 Benchmark - 128-dim Models" width="100%"/> </p>

🎯 Why Visual Document Retrieval?

Traditional OCR-based retrieval loses layout, tables, and visual context. Our visual approach:

  • —✅ No OCR errors - Direct visual understanding
  • —✅ Layout-aware - Understands tables, forms, charts
  • —✅ End-to-end - Single model, no pipeline complexity

🏆 Key Achievements

BenchmarkScoreRank (128-dim)
ViDoRe v190.24#5
MTEB v1+v281.02#6
ViDoRe v354.32#5

Medium Category Comparison (1-3B, 128-dim)

ModelParamsDimViDoRe v1MTEB v1+v2ViDoRe v3
SauerkrautLM-ColQwen3-2b-v0.1 ⭐2.2B12890.2481.0254.32
colqwen2-v1.02.2B12889.2379.7444.18
SauerkrautLM-ColQwen3-1.7b-Turbo-v0.11.7B12888.8977.9448.76

#1 in Medium category on all three benchmarks!

Detailed Benchmark Results

<details> <summary><b>📊 ViDoRe v1 (NDCG@5) - Click to expand</b></summary>

TaskScore
ArxivQA91.24
DocVQA65.06
InfoVQA93.14
ShiftProject88.74
SyntheticDocQA-AI99.63
SyntheticDocQA-Energy96.91
SyntheticDocQA-Gov96.08
SyntheticDocQA-Health99.26
TabFQuAD90.32
TATDQA82.06
Average90.24

</details>

<details> <summary><b>📊 MTEB v1+v2 (NDCG@5) - Click to expand</b></summary>

ViDoRe v1 Tasks: | Task | Score | |------|-------| | ArxivQA | 91.24 | | DocVQA | 65.06 | | InfoVQA | 93.14 | | ShiftProject | 88.74 | | SyntheticDocQA-AI | 99.63 | | SyntheticDocQA-Energy | 96.91 | | SyntheticDocQA-Gov | 96.08 | | SyntheticDocQA-Health | 99.26 | | TabFQuAD | 90.32 | | TATDQA | 82.06 |

ViDoRe v2 Tasks (Multilingual): | Task | Score | |------|-------| | ViDoRe-v2-2BioMed | 58.62 | | ViDoRe-v2-2Econ | 54.64 | | ViDoRe-v2-2ESG-HL | 68.13 | | ViDoRe-v2-2ESG | 50.40 | | Combined Average | 81.02 |

</details>

<details> <summary><b>📊 ViDoRe v3 (NDCG@10) - Click to expand</b></summary>

TaskScore
ViDoRe-v3-CS73.70
ViDoRe-v3-Energy61.21
ViDoRe-v3-FinanceEn54.30
ViDoRe-v3-FinanceFr40.18
ViDoRe-v3-HR52.97
ViDoRe-v3-Industry44.01
ViDoRe-v3-Pharma60.64
ViDoRe-v3-Physics47.57
Average54.32

</details>

Improvement over Baseline

MetricColQwen3-2bColQwen2-v1.0Improvement
ViDoRe v190.2489.23+1.01
MTEB v1+v281.0279.74+1.28
ViDoRe v354.3244.18+10.14

📋 Summary Tables

128-dim Models Comparison

<p align="center"> <img src="https://raw.githubusercontent.com/VAGOsolutions/sauerkrautlm-colpali/main/assets/tablesummary128dim.png" alt="128-dim Models Summary" width="100%"/> </p>

Comparison vs High-dim Models

<p align="center"> <img src="https://raw.githubusercontent.com/VAGOsolutions/sauerkrautlm-colpali/main/assets/tablesummaryhighdim_comparison.png" alt="High-dim Comparison" width="100%"/> </p>

✨ Key Features

  • —🥇 #1 in Medium Category: Best 1-3B model among 128-dim models
  • —📈 +1.01 over ColQwen2: Significant improvement over baseline
  • —💾 Consumer GPU Ready: Only ~4.4GB VRAM
  • —⚡ Compact Embeddings: 128-dimensional
  • —🌍 Multilingual: 6 languages (EN, DE, FR, ES, IT, PT)

Model Details

PropertyValue
Base ModelQwen/Qwen3-VL-2B
Parameters2.2B
Embedding Dimension128
VRAM (bfloat16)~4.4 GB
Max Context Length262,144 tokens
LicenseApache 2.0

Training

Hardware & Configuration

SettingValue
GPUs4x NVIDIA RTX 6000 Ada (48GB)
Effective Batch Size256
Precisionbfloat16

Datasets

DatasetTypeDescription
vidore/colpali_train_setPublicColPali training data
openbmb/VisRAG-Ret-Train-In-domain-dataPublicVisual RAG training data
llamaindex/vdr-multilingual-trainPublicMultilingual document retrieval
VAGO Multilingual Dataset 1In-houseProprietary multilingual document-query pairs
VAGO Multilingual Dataset 2In-houseProprietary multilingual document-query pairs

Installation & Usage

Sentence Transformers

This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

bash
pip install "sentence-transformers[image]>=6.0.0"
python
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder(
    "VAGOsolutions/SauerkrautLM-ColQwen3-2b-v0.1",
    model_kwargs={"dtype": "bfloat16"},
)

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]

query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([25, 128]) torch.Size([1251, 128])

# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[15.1758,  9.4717],
#         [ 4.1470, 14.3262]], device='cuda:0')

SauerkrautLM ColPali

⚠️ Important: Install our package first before loading the model:
bash
pip install git+https://github.com/VAGOsolutions/sauerkrautlm-colpali
python
import torch
from PIL import Image
from sauerkrautlm_colpali.models import ColQwen3, ColQwen3Processor

model_name = "VAGOsolutions/SauerkrautLM-ColQwen3-2b-v0.1"

model = ColQwen3.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="cuda:0",
).eval()

processor = ColQwen3Processor.from_pretrained(model_name)

images = [Image.open("document.png")]
queries = ["What is the main topic?"]

batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_queries(queries).to(model.device)

with torch.no_grad():
    image_embeddings = model(**batch_images)
    query_embeddings = model(**batch_queries)

scores = processor.score(query_embeddings, image_embeddings)

📊 Additional Benchmark Visualizations

MTEB v1+v2 Benchmark (128-dim Models)

<p align="center"> <img src="https://raw.githubusercontent.com/VAGOsolutions/sauerkrautlm-colpali/main/assets/benchmark128dimv1v2.png" alt="MTEB v1+v2 Benchmark - 128-dim Models" width="100%"/> </p>

ViDoRe v3 Benchmark (128-dim Models)

<p align="center"> <img src="https://raw.githubusercontent.com/VAGOsolutions/sauerkrautlm-colpali/main/assets/benchmark128dimv3.png" alt="ViDoRe v3 Benchmark - 128-dim Models" width="100%"/> </p>

Our Models vs High-dim Models

<p align="center"> <img src="https://raw.githubusercontent.com/VAGOsolutions/sauerkrautlm-colpali/main/assets/benchmarkoursvshighdimv1.png" alt="ViDoRe v1 - Our Models vs High-dim" width="100%"/> </p>

Citation

bibtex
@misc{sauerkrautlm-colpali-2025,
  title={SauerkrautLM-ColPali: Multi-Vector Vision Retrieval Models},
  author={David Golchinfar},
  organization={VAGO Solutions},
  year={2025},
  url={https://github.com/VAGOsolutions/sauerkrautlm-colpali}
}

Contact