BorisTM/loss-guided-static-multi

1---2license: apache-2.03library_name: sentence-transformers4pipeline_tag: sentence-similarity5language:6  - multilingual7tags:8  - sentence-transformers9  - sentence-similarity10  - static-embeddings11  - multilingual12---13 14# Loss-Guided Static Multilingual Sentence Embeddings15 16## Overview17 18A static multilingual sentence encoder: one embedding table, mean pooling, no Transformer at inference. Its vocabulary was extended with **loss-guided vocabulary discovery**: during contrastive training, adjacent token pairs are scored by how much the sentence objective would suffer without them, and the useful pairs receive their own embedding rows. The base tokenizer stays unchanged. The model is trained on 176 languages and is intended for CPU-efficient semantic similarity, retrieval, clustering, and classification.19 20## Model Specifications21 22- **Embedding dimension**: 1,02423- **Base vocabulary**: 257,914 rows (mmBERT tokenizer + 1,914 language markers)24- **Learned pair rows**: 56,07125- **Parameters**: 321.7M (embedding table only), FP32 safetensors, 1.29 GB26- **Training languages**: 17627- **Inference**: table lookup and mean pooling; requires a language marker at the start of each text28 29## Performance30 31MTEB(Multilingual, v2), 131 tasks, primary score (Mean over tasks) **50.13**, Mean over task types 43.45:32 33| Task type | Count | Mean score |34|-----------|-------|-----------|35| Bitext mining | 13 | 53.90 |36| Classification | 43 | 52.87 |37| Clustering | 16 | 38.91 |38| Instruction reranking | 3 | -1.04 |39| Multilabel classification | 5 | 17.64 |40| Pair classification | 11 | 72.69 |41| Reranking | 6 | 49.42 |42| Retrieval | 18 | 41.64 |43| STS | 16 | 64.99 |44 45Cross-lingual retrieval (macro F1 over language pairs): Tatoeba 57.26 (106 pairs), BUCC 98.12 (4 pairs), Flores 63.70 (193 pairs).46 47## Installation & Usage48 49Every input must start with a language marker `__<iso639-3>_<script>__`, for example `__eng_Latn__`, `__rus_Cyrl__`, `__arb_Arab__`, `__cmn_Hani__`. A text without a marker raises an error. The repository ships the custom embedding module, so `trust_remote_code=True` is required.50 51```python52pip install -U sentence-transformers53 54from sentence_transformers import SentenceTransformer55 56model = SentenceTransformer("BorisTM/loss-guided-static-multi", trust_remote_code=True)57 58sentences = [59    "__eng_Latn__ The cat sleeps on the sofa.",60    "__rus_Cyrl__ Кошка спит на диване.",61    "__deu_Latn__ Die Katze schläft auf dem Sofa.",62    "__eng_Latn__ The stock market fell sharply today.",63]64 65embeddings = model.encode(sentences, normalize_embeddings=True)66similarities = model.similarity(embeddings, embeddings)67# [[1.00, 0.44, 0.80, 0.05],68#  [0.44, 1.00, 0.41, 0.06],69#  [0.80, 0.41, 1.00, 0.08],70#  [0.05, 0.06, 0.08, 1.00]]71```72 73**License**: Apache 2.074