terrierteam/splade3

1### Putting it all together2 3When you use the document encoder in an indexing pipeline, the rewritten document contents are indexed:4 5<div class="pipeline">6  <div class="df" title="Document Frame">D</div>7  <div class="transformer attn" title="SPLADE Indexing Transformer">SPLADE</div>8  <div class="df" title="Document Frame">D</div>9  <div class="transformer" title="Indexer">Indexer</div>10  <div class="artefact" title="SPLADE Index">IDX</div>11</div>12 13```python14import pyterrier as pt15import pyt_splade16 17dataset = pt.get_dataset('irds:msmarco-passage')18splade = pyt_splade.Splade()19 20indexer = pt.IterDictIndexer('./msmarco_psg', pretokenised=True)21 22indxer_pipe = splade.doc_encoder() >> indexer23indxer_pipe.index(dataset.get_corpus_iter())24```25 26Once you built an index, you can build a retrieval pipeline that first encodes the query,27and then performs retrieval:28 29<div class="pipeline">30  <div class="df" title="Query Frame">Q</div>31  <div class="transformer attn" title="SPLADE Query Transformer">SPLADE</div>32  <div class="df" title="Query Frame">Q</div>33  <div class="transformer" title="Term Frequency Transformer">TF Retriever <div class="artefact" title="SPLADE Index">IDX</div></div>34  <div class="df" title="Result Frame">R</div>35</div>36 37```python38splade_retr = splade.query_encoder() >> pt.terrier.Retriever('./msmarco_psg', wmodel='Tf')39```40 41### References & Credits42 43This package uses [Naver's SPLADE repository](https://github.com/naver/splade).44 45 - Thibault Formal, Benjamin Piwowarski, Stéphane Clinchant. [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://arxiv.org/abs/2107.05720). SIGIR 2021.46 - Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, Iadh Ounis. [PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval](https://dl.acm.org/doi/abs/10.1145/3459637.3482013). CIKM 2021.47