Amdestya/cat-minilm-l12

cat-minilm-l12 — MiniLM cross-encoder reranker (no score injection)

A MiniLM-L12 cross-encoder passage reranker trained on MS MARCO by knowledge distillation.

This is the matched baseline for `Amdestya/bm25cat-minilm-l12`, built to reproduce the CE_CAT / CE_BM25CAT comparison in Askari et al., *"Injecting the BM25 Score as Text Improves BERT-Based Re-rankers"*, ECIR 2023.

The two models share identical training data (the same 5,000,000 pairs), steps, optimizer, learning rate, batch size and sequence length. The only difference is the input construction: this one receives no injected BM25 score.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("Amdestya/cat-minilm-l12", max_length=256)

scores = model.predict([
    ["what is a cat", "A cat is a small domesticated carnivorous mammal."],
    ["what is a cat", "The 1998 federal budget deficit was revised upward."],
])

Input format

text_a = query
text_b = passage
# tokenises to: [CLS] query [SEP] passage [SEP]

Do not inject a BM25 score. This model was trained without one and has never seen a numeric token in that position. For the injected variant use `Amdestya/bm25cat-minilm-l12`, whose input format differs — see its card.

Outputs are raw logits (identity activation), unbounded, higher = more relevant. They are comparable within a query, not across models.

Training

Note this is a knowledge-distillation setup, following the authors' released training notebook, rather than the cross-entropy objective described in the paper's §4.

Intended use

Built as a reference artifact for a dissertation study on reproducing IR papers with LLMs. Usable as a general MS MARCO passage reranker, but the community `cross-encoder/ms-marco-MiniLM-L-12-v2` is trained more thoroughly and is the better choice for production use.