deepvk/USER2-small
1314k
1---2library_name: sentence-transformers3pipeline_tag: sentence-similarity4tags:5- sentence-transformers6- feature-extraction7- sentence-similarity8license: apache-2.09base_model:10- deepvk/RuModernBERT-small11datasets:12- deepvk/ru-HNP13- deepvk/ru-WANLI14- deepvk/cultura_ru_ed15- Shitao/bge-m3-data16- CarlBrendt/Summ_Dialog_News17- IlyaGusev/gazeta18- its5Q/habr_qna19- wikimedia/wikipedia20- RussianNLP/wikiomnia21language:22- ru23---24 25# USER2-small26 27**USER2** is a new generation of the **U**niversal **S**entence **E**ncoder for **R**ussian, designed for sentence representation with long-context support of up to 8,192 tokens.28 29The models are built on top of the [`RuModernBERT`](https://huggingface.co/collections/deepvk/rumodernbert-67b5e82fbc707d7ed3857743) encoders and are fine-tuned for retrieval and semantic tasks. 30They also support [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) — a technique that enables reducing embedding size with minimal loss in representation quality.31 32This is a small model with 34 million parameters.33 34| Model | Size | Context Length | Hidden Dim | MRL Dims |35|-----------------------------------------------------------------------:|:----:|:--------------:|:----------:|:-----------------------:|36| `deepvk/USER2-small` | 34M | 8192 | 384 | [32, 64, 128, 256, 384] |37| [`deepvk/USER2-base`](https://huggingface.co/deepvk/USER2-base) | 149M | 8192 | 768 | [32, 64, 128, 256, 384, 512, 768] |38 39## Performance40 41To evaluate the model, we measure quality on the `MTEB-rus` benchmark.42Additionally, to measure long-context retrieval, we run Russian subset of MultiLongDocRetrieval (MLDR) task.43 44**MTEB-rus**45 46| Model | Size | Hidden Dim | Context Length | MRL support | Mean(task) | Mean(taskType) | Classification | Clustering | MultiLabelClassification | PairClassification | Reranking | Retrieval | STS |47|----------------------------------------------------------------------------------------------:|:-----:|:----------:|:--------------:|:-----------:|:----------:|:--------------:|:-------------:|:----------:|:------------------------:|:-----------------:|:---------:|:---------:|:-----:|48| `USER-base` | 124M | 768 | 512 | ❌ | 58.11 | 56.67 | 59.89 | 53.26 | 37.72 | 59.76 | 55.58 | 56.14 | 74.35 |49| `USER-bge-m3` | 359M | 1024 | 8192 | ❌ | 62.80 | 62.28 | 61.92 | 53.66 | 36.18 | 65.07 | 68.72 | 73.63 | 76.76 |50| `multilingual-e5-base` | 278M | 768 | 512 | ❌ | 58.34 | 57.24 | 58.25 | 50.27 | 33.65 | 54.98 | 66.24 | 67.14 | 70.16 |51| `multilingual-e5-large-instruct` | 560M | 1024 | 512 | ❌ | 65.00 | 63.36 | 66.28 | 63.13 | 41.15 | 63.89 | 64.35 | 68.23 | 76.48 | 52| `jina-embeddings-v3` | 572M | 1024 | 8192 | ✅ | 63.45 | 60.93 | 65.24 | 60.90 | 39.24 | 59.22 | 53.86 | 71.99 | 76.04 |53| `ru-en-RoSBERTa` | 404M | 1024 | 512 | ❌ | 61.71 | 60.40 | 62.56 | 56.06 | 38.88 | 60.79 | 63.89 | 66.52 | 74.13 |54| `USER2-small` | 34M | 384 | 8192 | ✅ | 58.32 | 56.68 | 59.76 | 57.06 | 33.56 | 54.02 | 58.26 | 61.87 | 72.25 |55| `USER2-base` | 149M | 768 | 8192 | ✅ | 61.12 | 59.59 | 61.67 | 59.22 | 36.61 | 56.39 | 62.06 | 66.90 | 74.28 |56 57**MLDR-rus**58 59| Model | Size | nDCG@10 ↑ |60|-------------------:|:---------:|:---------:|61| `USER-bge-m3` | 359M | 58.53 | 62| `KaLM-v1.5` | 494M | 53.75 | 63| `jina-embeddings-v3` | 572M | 49.67 |64| `E5-mistral-7b` | 7.11B | 52.40 |65| `USER2-small` | 34M | 51.69 |66| `USER2-base` | 149M | 54.17 |67 68We compare only model with context length of 8192.69 70## Matryoshka71 72To evaluate MRL capabilities, we also use `MTEB-rus`, applying dimensionality cropping to the embeddings to match the selected size.73 74<img src="assets/mrl.png" alt="MRL" width="600"/>75 76## Usage77 78### Prefixes79 80This model is trained similarly to [Nomic Embed](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5#task-instruction-prefixes) and expects task-specific prefixes to be added to the input. The choice of prefix depends on the specific task. We follow a few general guidelines when selecting a prefix:81- "classification: " is the default and most universal prefix, often performing well across a variety of tasks.82- "clustering: " is recommended for clustering applications: group texts into clusters, discover shared topics, or remove semantic duplicates.83- "search_query: " and "search_document: " are intended for retrieval and reranking tasks. Also, in some classification tasks, especially with shorter texts, "search_query" shows superior performance to other prefixes. On the other hand, "search_document" can be beneficial for long-context sentence similarity tasks.84 85However, we encourage users to experiment with different prefixes, as certain domains may benefit from specific ones.86 87### Sentence Transformers88 89```python90from sentence_transformers import SentenceTransformer91 92model = SentenceTransformer("deepvk/USER2-small")93 94query_embeddings = model.encode(["Когда был спущен на воду первый миноносец «Спокойный»?"], prompt_name="search_query")95document_embeddings = model.encode(["Спокойный (эсминец)\nЗачислен в списки ВМФ СССР 19 августа 1952 года."], prompt_name="search_document")96 97similarities = model.similarity(query_embeddings, document_embeddings)98```99 100To truncate the embedding dimension, simply pass the new value to the model initialization:101```python102model = SentenceTransformer("deepvk/USER2-small", truncate_dim=128)103```104This model was trained with dimensions `[32, 64, 128, 256, 384]`, so it’s recommended to use one of these for best performance.105 106### Transformers107 108```python109import torch110import torch.nn.functional as F111from transformers import AutoTokenizer, AutoModel112 113 114def mean_pooling(model_output, attention_mask):115 token_embeddings = model_output[0]116 input_mask_expanded = (117 attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()118 )119 return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(120 input_mask_expanded.sum(1), min=1e-9121 )122 123 124queries = ["search_query: Когда был спущен на воду первый миноносец «Спокойный»?"]125documents = ["search_document: Спокойный (эсминец)\nЗачислен в списки ВМФ СССР 19 августа 1952 года."]126 127tokenizer = AutoTokenizer.from_pretrained("deepvk/USER2-small")128model = AutoModel.from_pretrained("deepvk/USER2-small")129 130encoded_queries = tokenizer(queries, padding=True, truncation=True, return_tensors="pt")131encoded_documents = tokenizer(documents, padding=True, truncation=True, return_tensors="pt")132 133with torch.no_grad():134 queries_outputs = model(**encoded_queries)135 documents_outputs = model(**encoded_documents)136 137query_embeddings = mean_pooling(queries_outputs, encoded_queries["attention_mask"])138query_embeddings = F.normalize(query_embeddings, p=2, dim=1)139doc_embeddings = mean_pooling(documents_outputs, encoded_documents["attention_mask"])140doc_embeddings = F.normalize(doc_embeddings, p=2, dim=1)141 142similarities = query_embeddings @ doc_embeddings.T143```144 145To truncate the embedding dimension, select the first values:146```python147query_embeddings = mean_pooling(queries_outputs, encoded_queries["attention_mask"])148query_embeddings = query_embeddings[:, :truncate_dim]149query_embeddings = F.normalize(query_embeddings, p=2, dim=1)150```151 152## Training details153 154This is the small version with 34 million parameters, based on [`RuModernBERT-small`](https://huggingface.co/deepvk/RuModernBERT-small). 155It was fine-tuned in three stages: RetroMAE, Weakly Supervised Fine-Tuning, and Supervised Fine-Tuning.156 157Following the *bge-m3* training strategy, we use RetroMAE as a retrieval-oriented continuous pretraining step. 158Leveraging data from the final stage of RuModernBERT training, RetroMAE enhances retrieval quality—particularly for long-context inputs.159 160To follow best practices for building a state-of-the-art encoder, we rely on large-scale training with weakly related text pairs. 161However, such datasets are not publicly available for Russian, unlike for English or Chinese. 162To overcome this, we apply two complementary strategies:163 164- **Cross-lingual transfer**: We train on both English and Russian data, leveraging English resources (`nomic-unsupervised`) alongside our in-house English-Russian parallel corpora. 165- **Unsupervised pair mining**: From the [`deepvk/cultura_ru_edu`](https://huggingface.co/datasets/deepvk/cultura_ru_edu) corpus, we extract 50M pairs using a simple heuristic—selecting non-overlapping text blocks that are not substrings of one another.166 167This approach has shown promising results, allowing us to train high-performing models with minimal target-language pairs—especially when compared to pipelines used for other languages.168 169The table below shows the datasets used and the number of times each was upsampled.170 171| Dataset | Size | Upsample |172|----------------------------:|:----:|:-------:|173| [nomic-en](https://github.com/nomic-ai/nomic) | 235M | 1 |174| [nomic-ru](https://github.com/nomic-ai/nomic) | 39M | 3 |175| in-house En-Ru parallel | 250M | 1 |176| [cultura-sampled](https://huggingface.co/datasets/deepvk/cultura_ru_edu) | 50M | 1 |177| **Total** | 652M | |178 179For the third stage, we switch to cleaner, task-specific datasets. 180In some cases, additional filtering was applied using a cross-encoder. 181For all retrieval datasets, we mine hard negatives.182 183| Dataset | Examples | Notes |184|-------------------------------------------------------------------------------------------------------------------------------------------------:|:--------:|:------------------------------------------|185| [Nomic-en-supervised](https://huggingface.co/datasets/nomic-ai/nomic-embed-supervised-data) | 1.7 M | Unmodified |186| AllNLI | 200 K | Translated SNLI/MNLI/ANLI to Russian |187| [fishkinet-posts](https://huggingface.co/datasets/nyuuzyou/fishkinet-posts) | 93 K | Title–content pairs |188| [gazeta](https://huggingface.co/datasets/IlyaGusev/gazeta) | 55 K | Title–text pairs |189| [habr_qna](https://huggingface.co/datasets/its5Q/habr_qna) | 100 K | Title–description pairs |190| [lenta](https://huggingface.co/datasets/zloelias/lenta-ru) | 100 K | Title–news pairs |191| [miracl_ru](https://huggingface.co/datasets/Shitao/bge-m3-data) | 10 K | One positive per anchor |192| [mldr_ru](https://huggingface.co/datasets/Shitao/bge-m3-data) | 1.8 K | Unmodified |193| [mr-tydi_ru](https://huggingface.co/datasets/Shitao/bge-m3-data) | 5.3 K | Unmodified |194| [mmarco_ru](https://huggingface.co/datasets/unicamp-dl/mmarco) | 500 K | Unmodified |195| [ru-HNP](https://huggingface.co/datasets/deepvk/ru-HNP) | 100 K | One pos + one neg per anchor |196| ru‑queries | 199 K | In-house (generated as in [arXiv:2401.00368](https://arxiv.org/abs/2401.00368)) |197| [ru‑WaNLI](https://huggingface.co/datasets/deepvk/ru-WANLI) | 35 K | Entailment -> pos, contradiction -> neg |198| [sampled_wiki](https://huggingface.co/datasets/wikimedia/wikipedia) | 1 M | Sampled text blocks from Wikipedia |199| [summ_dialog_news](https://huggingface.co/datasets/CarlBrendt/Summ_Dialog_News) | 37 K | Summary–info pairs |200| [wikiomnia_qna](https://huggingface.co/datasets/RussianNLP/wikiomnia) | 100 K | QA pairs (T5-generated) |201| [yandex_q](https://huggingface.co/datasets/its5Q/yandex-q) | 83 K | Q+desc-answer pairs |202| **Total** | 4.3 M | |203 204 205### Ablation206 207Alongside the final model, we also release all intermediate training steps. 208Both the **retromae** and **weakly_sft** models are available under the specified revisions in this repository. 209We hope these additional models prove useful for your experiments.210 211Below is a comparison of all training stages on a subset of `MTEB-rus`.212 213<img src="assets/training_stages.png" alt="training_stages" width="600"/>214 215## Citations216 217```218@misc{deepvk2025user,219 title={USER2},220 author={Malashenko, Boris and Spirin, Egor and Sokolov Andrey},221 url={https://huggingface.co/deepvk/USER2-small},222 publisher={Hugging Face}223 year={2025},224}225```