Qdrant/colmodernvbert

1---2license: mit3base_model: ModernVBERT/colmodernvbert4library_name: fastembed5tags:6  - colpali7  - vision-language8  - multimodal9  - onnx10  - fastembed11  - document-retrieval12pipeline_tag: feature-extraction13---14 15# ColModernVBERT ONNX16 17This repository contains an ONNX export of the [ModernVBERT/colmodernvbert](https://huggingface.co/ModernVBERT/colmodernvbert) model, converted by Qdrant for efficient inference with [FastEmbed](https://github.com/qdrant/fastembed).18 19**Original Model Authors:** Paul Teiletche, Quentin Macé, Max Conti, Antonio Loison, Gautier Viaud, Pierre Colombo, Manuel Faysse20 21**ONNX Conversion:** Kacper Łukawski (Qdrant)22 23## About This Repository24 25This is an ONNX-converted version of ColModernVBERT, optimized for deployment with FastEmbed. The original model was developed by the authors listed above and is described in detail in their [paper](https://arxiv.org/abs/2510.01149).26 27**What's Different:**28- Model format: PyTorch → ONNX29- Optimization: Configured for efficient CPU/GPU inference30- Integration: Ready for use with FastEmbed's multimodal embedding APIs31 32**What's Preserved:**33- Model architecture and weights34- Tokenizer and processor configurations35- Image preprocessing pipeline36- Embedding dimensions and behavior37 38## Model Description39 40ColModernVBERT is a vision-language model based on the ModernVBERT architecture, designed for document retrieval tasks. The [ModernVBERT paper](https://arxiv.org/abs/2510.01149) demonstrates that this 250M-parameter model achieves state-of-the-art performance in its size class, matching models up to 10x larger on visual document retrieval benchmarks.41 42It combines:43- ModernBERT text encoder (jhu-clsp/ettin-encoder-150m)44- SigLIP vision encoder (google/siglip2-base-patch16-512)45- Custom projection layer reducing to 128 dimensions46 47## Model Details48 49- **Architecture**: ColModernVBert50- **Projection Dimension**: 12851- **Max Sequence Length**: 8192 tokens52- **Image Size**: 512x512 (resized by processor)53- **Image Token ID**: 5040754- **Format**: ONNX (exported from PyTorch)55 56## Usage57 58This ONNX model is designed for use with FastEmbed for efficient document and query encoding.59 60### With FastEmbed61 62```python63from fastembed import LateInteractionMultimodalEmbedding64from PIL import Image65 66# Initialize the model67model = LateInteractionMultimodalEmbedding(68    model_name="Qdrant/colmodernvbert"69)70 71# Encode documents (images)72images = [Image.open("document1.png"), Image.open("document2.png")]73doc_embeddings = model.embed_image(images)74 75# Encode queries (text)76queries = ["What is the total amount?", "Find the invoice date"]77query_embeddings = model.embed_text(queries)78```79 80For more details on using this model with FastEmbed, see the [FastEmbed documentation](https://github.com/qdrant/fastembed).81 82### Original PyTorch Version83 84For the original PyTorch implementation with additional features, please refer to the [original model repository](https://huggingface.co/ModernVBERT/colmodernvbert) and the [colpali-engine library](https://github.com/illuin-tech/colpali).85 86## Performance87 88ColModernVBERT achieves competitive performance with models up to 10x larger on visual document retrieval tasks. For detailed benchmarks and evaluation results, please refer to the [original paper](https://arxiv.org/abs/2510.01149) and [model card](https://huggingface.co/ModernVBERT/colmodernvbert).89 90**Note:** ONNX inference performance may differ slightly from PyTorch due to optimizations and runtime differences. The model architecture and weights are preserved to maintain embedding quality.91 92## License93 94This model is released under the MIT License, consistent with the original ColModernVBERT model.95 96**Original Model License:**97- Copyright (c) 2025 Paul Teiletche, Quentin Macé, Max Conti, Antonio Loison, Gautier Viaud, Pierre Colombo, Manuel Faysse98- Licensed under MIT License99 100**ONNX Conversion:**101- Copyright (c) 2025 Qdrant102- Licensed under MIT License103 104See the [LICENSE](LICENSE) file for the full license text.105 106## Citation107 108If you use this model in your work, please cite the original ModernVBERT paper:109 110```bibtex111@misc{teiletche2025modernvbertsmallervisualdocument,112      title={ModernVBERT: Towards Smaller Visual Document Retrievers},113      author={Paul Teiletche and Quentin Macé and Max Conti and Antonio Loison and Gautier Viaud and Pierre Colombo and Manuel Faysse},114      year={2025},115      eprint={2510.01149},116      archivePrefix={arXiv},117      primaryClass={cs.IR},118      url={https://arxiv.org/abs/2510.01149},119}120```121 122If you use this ONNX conversion specifically, you may also acknowledge:123 124```bibtex125@software{qdrant_colmodernvbert_onnx,126  title={ColModernVBERT ONNX - FastEmbed Integration},127  author={Qdrant},128  year={2025},129  url={https://huggingface.co/Qdrant/colmodernvbert},130  note={ONNX conversion of ModernVBERT/colmodernvbert}131}132```133 134## Acknowledgments135 136This work builds upon the excellent ColModernVBERT model developed by Paul Teiletche, Quentin Macé, Max Conti, Antonio Loison, Gautier Viaud, Pierre Colombo, and Manuel Faysse. We thank them for releasing their work under the MIT License, enabling derivative works like this ONNX conversion.137 138The ONNX conversion was performed by the Qdrant team to enable efficient deployment through the FastEmbed library.139