answerdotai/answerai-colbert-small-v1
163895k
1---2license: apache-2.03language:4- en5tags:6- ColBERT7- multi-vector8- sentence-transformers9- RAGatouille10- passage-retrieval11---12 13# answerai-colbert-small-v114 15**answerai-colbert-small-v1** is a new, proof-of-concept model by [Answer.AI](https://answer.ai), showing the strong performance multi-vector models with the new [JaColBERTv2.5 training recipe](https://arxiv.org/abs/2407.20750) and some extra tweaks can reach, even with just **33 million parameters**.16 17While being MiniLM-sized, it outperforms all previous similarly-sized models on common benchmarks, and even outperforms much larger popular models such as e5-large-v2 or bge-base-en-v1.5.18 19For more information about this model or how it was trained, head over to the [announcement blogpost](https://www.answer.ai/posts/2024-08-13-small-but-mighty-colbert.html).20 21## Usage22 23This model was designed with the upcoming RAGatouille overhaul in mind. However, it's compatible with all recent ColBERT implementations!24 25### Sentence Transformers26 27This model can be used with [Sentence Transformers](https://www.sbert.net/) as a multi-vector (ColBERT-style late interaction) retriever via the `MultiVectorEncoder`:28 29```bash30pip install "sentence-transformers>=6.0.0"31```32 33```python34from sentence_transformers import MultiVectorEncoder35 36model = MultiVectorEncoder("answerdotai/answerai-colbert-small-v1")37 38query = "Which planet is known as the Red Planet?"39documents = [40 "Venus is often called Earth's twin because of its similar size and proximity.",41 "Mars, known for its reddish appearance, is often referred to as the Red Planet.",42 "Jupiter, the largest planet in our solar system, has a prominent red spot.",43 "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",44]45 46query_embeddings = model.encode_query(query)47document_embeddings = model.encode_document(documents)48print(query_embeddings.shape, document_embeddings[0].shape)49# (32, 96) (17, 96)50 51# MaxSim late-interaction scoring (higher is more relevant)52scores = model.similarity(query_embeddings, document_embeddings)53print(scores)54# tensor([[30.5692, 31.4895, 31.3029, 31.3072]])55```56 57### Rerankers58 59If you're interested in using this model as a re-ranker (it vastly outperforms cross-encoders its size!), you can do so via the [rerankers](https://github.com/AnswerDotAI/rerankers) library:60 61```bash62pip install --upgrade rerankers[transformers]63```64 65```python66from rerankers import Reranker67 68ranker = Reranker("answerdotai/answerai-colbert-small-v1", model_type='colbert')69docs = ['Hayao Miyazaki is a Japanese director, born on [...]', 'Walt Disney is an American author, director and [...]', ...]70query = 'Who directed spirited away?'71ranker.rank(query=query, docs=docs)72```73 74### RAGatouille75 76```bash77pip install --upgrade ragatouille78```79 80```python81from ragatouille import RAGPretrainedModel82 83RAG = RAGPretrainedModel.from_pretrained("answerdotai/answerai-colbert-small-v1")84 85docs = ['Hayao Miyazaki is a Japanese director, born on [...]', 'Walt Disney is an American author, director and [...]', ...]86 87RAG.index(docs, index_name="ghibli")88 89query = 'Who directed spirited away?'90results = RAG.search(query)91```92 93### Stanford ColBERT94 95```bash96pip install --upgrade colbert-ai97```98 99#### Indexing100 101```python102from colbert import Indexer103from colbert.infra import Run, RunConfig, ColBERTConfig104 105INDEX_NAME = "DEFINE_HERE" 106 107if __name__ == "__main__":108 config = ColBERTConfig(109 doc_maxlen=512,110 nbits=2111 )112 indexer = Indexer(113 checkpoint="answerdotai/answerai-colbert-small-v1",114 config=config,115 )116 docs = ['Hayao Miyazaki is a Japanese director, born on [...]', 'Walt Disney is an American author, director and [...]', ...]117 118 indexer.index(name=INDEX_NAME, collection=docs)119```120 121#### Querying122 123```python124from colbert import Searcher125from colbert.infra import Run, RunConfig, ColBERTConfig126 127INDEX_NAME = "THE_INDEX_YOU_CREATED" 128k = 10 129 130if __name__ == "__main__":131 config = ColBERTConfig(132 query_maxlen=32 # Adjust as needed, we recommend the nearest higher multiple of 16 to your query133 )134 searcher = Searcher(135 index=index_name, 136 config=config137 ) 138 query = 'Who directed spirited away?'139 results = searcher.search(query, k=k)140```141 142 143#### Extracting Vectors144 145Finally, if you want to extract individula vectors, you can use the model this way:146 147 148```python149from colbert.modeling.checkpoint import Checkpoint150 151ckpt = Checkpoint("answerdotai/answerai-colbert-small-v1", colbert_config=ColBERTConfig())152embedded_query = ckpt.queryFromText(["Who dubs Howl's in English?"], bsize=16)153```154 155 156## Results157 158### Against single-vector models159 160161 162 163| Dataset / Model | answer-colbert-s | snowflake-s | bge-small-en | bge-base-en |164|:-----------------|:-----------------:|:-------------:|:-------------:|:-------------:|165| **Size** | 33M (1x) | 33M (1x) | 33M (1x) | **109M (3.3x)** |166| **BEIR AVG** | **53.79** | 51.99 | 51.68 | 53.25 |167| **FiQA2018** | **41.15** | 40.65 | 40.34 | 40.65 |168| **HotpotQA** | **76.11** | 66.54 | 69.94 | 72.6 |169| **MSMARCO** | **43.5** | 40.23 | 40.83 | 41.35 |170| **NQ** | **59.1** | 50.9 | 50.18 | 54.15 |171| **TRECCOVID** | **84.59** | 80.12 | 75.9 | 78.07 |172| **ArguAna** | 50.09 | 57.59 | 59.55 | **63.61** |173| **ClimateFEVER**| 33.07 | **35.2** | 31.84 | 31.17 |174| **CQADupstackRetrieval** | 38.75 | 39.65 | 39.05 | **42.35** |175| **DBPedia** | **45.58** | 41.02 | 40.03 | 40.77 |176| **FEVER** | **90.96** | 87.13 | 86.64 | 86.29 |177| **NFCorpus** | 37.3 | 34.92 | 34.3 | **37.39** |178| **QuoraRetrieval** | 87.72 | 88.41 | 88.78 | **88.9** |179| **SCIDOCS** | 18.42 | **21.82** | 20.52 | 21.73 |180| **SciFact** | **74.77** | 72.22 | 71.28 | 74.04 |181| **Touche2020** | 25.69 | 23.48 | **26.04** | 25.7 |182 183### Against ColBERTv2.0184 185| Dataset / Model | answerai-colbert-small-v1 | ColBERTv2.0 |186|:-----------------|:-----------------------:|:------------:|187| **BEIR AVG** | **53.79** | 50.02 |188| **DBPedia** | **45.58** | 44.6 |189| **FiQA2018** | **41.15** | 35.6 |190| **NQ** | **59.1** | 56.2 |191| **HotpotQA** | **76.11** | 66.7 |192| **NFCorpus** | **37.3** | 33.8 |193| **TRECCOVID** | **84.59** | 73.3 |194| **Touche2020** | 25.69 | **26.3** |195| **ArguAna** | **50.09** | 46.3 |196| **ClimateFEVER**| **33.07** | 17.6 |197| **FEVER** | **90.96** | 78.5 |198| **QuoraRetrieval** | **87.72** | 85.2 |199| **SCIDOCS** | **18.42** | 15.4 |200| **SciFact** | **74.77** | 69.3 |201 202 203## Referencing204 205We'll most likely eventually release a technical report. In the meantime, if you use this model or other models following the JaColBERTv2.5 recipe and would like to give us credit, please cite the JaColBERTv2.5 journal pre-print:206 207```208@article{clavie2024jacolbertv2,209 title={JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resources},210 author={Clavi{\'e}, Benjamin},211 journal={arXiv preprint arXiv:2407.20750},212 year={2024}213}214```215