deepvk/RuModernBERT-base

1---2library_name: transformers3license: apache-2.04datasets:5- deepvk/cultura_ru_edu6- HuggingFaceFW/fineweb-27- HuggingFaceFW/fineweb8language:9- ru10- en11pipeline_tag: fill-mask12---13 14# RuModernBERT-base15 16The Russian version of the modernized bidirectional encoder-only Transformer model, [ModernBERT](https://arxiv.org/abs/2412.13663).17RuModernBERT was pre-trained on approximately 2 trillion tokens of Russian, English, and code data with a context length of up to 8,192 tokens, using data from the internet, books, scientific sources, and social media.18 19|                                                                               | Model Size | Hidden Dim | Num Layers | Vocab Size | Context Length |    Task   |20|------------------------------------------------------------------------------:|:----------:|:----------:|:----------:|:----------:|:--------------:|:---------:|21| [deepvk/RuModernBERT-small](https://huggingface.co/deepvk/RuModernBERT-small) |     35M    |     384    |     12     |    50368   |      8192      | Masked LM |22|                                               deepvk/RuModernBERT-base [this] |    150M    |     768    |     22     |    50368   |      8192      | Masked LM |23 24## Notice ⚠️25 26The patched tokenizer is provided under the [patched-tokenizer](https://huggingface.co/deepvk/RuModernBERT-base/tree/patched-tokenizer) revision.27 28<details>29<summary>Details</summary>30 31We observed that several Russian lowercase letters were split into multiple subword tokens. This can be problematic for tasks like Named Entity Recognition (NER), where it is important that the first token of a word is a semantically meaningful unit.32 33To address this, we release a patched revision of the tokenizer with minimal but targeted change. Six common Russian lowercase letters *(а, е, и, н, о, т)* are now encoded as single tokens. These tokens were assigned to [unusedX] slots in the vocabulary. Corresponding BPE merges were added to ensure proper single-token encoding during inference. To preserve compatibility with the pretrained model each new token was initialized with the embedding of its uppercase counterpart both in tok_embedding and lm_head. To prevent duplicate vectors and maintain robustness, a small amount of Gaussian noise was added during initialization with gamma 1e-3.34 35We evaluated the patched model on 20 tasks from the RuMTEB benchmark and did not observe any statistically significant differences in performance compared to the original version. If your task is sensitive to tokenization granularity, such as in NER, we recommend using this updated version.36 37Usage example:38 39```python40from transformers import AutoTokenizer, AutoModelForMaskedLM41 42model_id = "deepvk/RuModernBERT-base"43 44# You can specify revision45revision = "patched-tokenizer"46tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)47model = AutoModelForMaskedLM.from_pretrained(model_id, revision=revision, attn_implementation="flash_attention_2")48```49 50</details>51 52## Usage53 54Don't forget to update `transformers` and install `flash-attn` if your GPU supports it.55 56```python57from transformers import AutoTokenizer, AutoModelForMaskedLM58 59# Prepare model60model_id = "deepvk/RuModernBERT-base"61tokenizer = AutoTokenizer.from_pretrained(model_id)62model = AutoModelForMaskedLM.from_pretrained(model_id, attn_implementation="flash_attention_2")63model = model.eval()64 65# Prepare input66text = "Лимончелло это настойка из [MASK]."67inputs = tokenizer(text, return_tensors="pt")68masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)69 70# Make prediction71outputs = model(**inputs)72 73# Show prediction74predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)75predicted_token = tokenizer.decode(predicted_token_id)76print("Predicted token:", predicted_token)77# Predicted token:  лимона78```79 80## Training Details81 82This is the base version with 150 million parameters and the same configuration as in [`ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base).83The crucial difference lies in the data we used to pre-train this model.84 85### Tokenizer86 87We trained a new tokenizer following the original configuration.88We maintained the size of the vocabulary and added the same special tokens.89The tokenizer was trained on a mixture of Russian and English from FineWeb.90 91### Dataset92 93Pre-training includes three main stages: massive pre-training, context extension, and cooldown.94Unlike the original model, we did not use the same data for all stages.95For the second and third stages, we used cleaner data sources.96 97|           Data Source | Stage 1  | Stage 2 | Stage 3  |98|----------------------:|:--------:|:-------:|:--------:|99|       FineWeb (En+Ru) |    ✅    |    ❌    |    ❌    |100|  CulturaX-Ru-Edu (Ru) |    ❌    |    ✅    |    ❌    |101|          Wiki (En+Ru) |    ✅    |    ✅    |    ✅    |102|            ArXiv (En) |    ✅    |    ✅    |    ✅    |103|          Book (En+Ru) |    ✅    |    ✅    |    ✅    |104|                  Code |    ✅    |    ✅    |    ✅    |105| StackExchange (En+Ru) |    ✅    |    ✅    |    ✅    |106|           Social (Ru) |    ✅    |    ✅    |    ✅    |107|      **Total Tokens** |   1.7T   |   250B  |    50B   |108 109 110### Context length111 112In the first stage, the model was trained with a context length of `1,024`.113In the second and third stages, it was extended to `8,192`.114 115## Evaluation116 117To evaluate the model, we measure quality on the [`encodechka`](https://github.com/avidale/encodechka) and [`Russian Super Glue (RSG)`](https://russiansuperglue.com/) benchmarks.118For RSG, we perform a grid search for optimal hyperparameters and report metrics from the **dev** split.119 120For a fair comparison, we compare the RuModernBERT model only with raw encoders that were not trained on retrieval or sentence embedding tasks.121 122### Russian Super Glue123 124<img src="./rsg.jpg">125 126| Model                                                                          | RCB       |  PARus | MuSeRC  | TERRa | RUSSE   | RWSD    | DaNetQA | Score     |127|-------------------------------------------------------------------------------:|:---------:|:------:|:-------:|:-----:|:-------:|:-------:|:-------:|:---------:|128| [deepvk/deberta-v1-distill](https://huggingface.co/deepvk/deberta-v1-distill)  | 0.433     |  0.56  | 0.625   | 0.590 | 0.943   | 0.569   | 0.726   | 0.635     |129| [deepvk/deberta-v1-base](https://huggingface.co/deepvk/deberta-v1-base)        | 0.450     |  0.61  | 0.722   | 0.704 | 0.948   | 0.578   | **0.760**   | 0.682     |130| [ai-forever/ruBert-base](https://huggingface.co/ai-forever/ruBert-base)        | 0.491     |  0.61  | 0.663   | 0.769 | 0.962   | 0.574   | 0.678   | 0.678     |131| [deepvk/RuModernBERT-small](https://huggingface.co/deepvk/RuModernBERT-small)  | 0.555     |  **0.64**  | 0.746   | 0.593 | 0.930   | 0.574   | 0.743   | 0.683     |132| deepvk/RuModernBERT-base [this]                                                | **0.556** |  0.61  | **0.857**   | **0.818** | **0.977**   | **0.583**   | 0.758   | **0.737**     |133 134### Encodechka135 136|                                                                                     | Model Size |   STS-B  | Paraphraser |   XNLI   | Sentiment | Toxicity | Inappropriateness |  Intents | IntentsX |  FactRu  |  RuDReC  |   Avg. S   |  Avg. S+W |137|------------------------------------------------------------------------------------:|:----------:|:--------:|:-----------:|:--------:|:---------:|:--------:|:-----------------:|:--------:|:--------:|:--------:|:--------:|:----------:|:---------:|138|         [cointegrated/rubert-tiny](https://huggingface.co/cointegrated/rubert-tiny) |    11.9M   |   0.66   |     0.53    | **0.40** |    0.71   |   0.89   |        0.68       |   0.70   | **0.58** |   0.24   |   0.34   |    0.645   |   0.575   |139|       [deepvk/deberta-v1-distill](https://huggingface.co/deepvk/deberta-v1-distill) |    81.5M   | **0.70** |   **0.57**  |   0.38   |  **0.77** | **0.98** |        0.79       |   0.77   |   0.36   |   0.36   | **0.44** |    0.665   | **0.612** |140|             [deepvk/deberta-v1-base](https://huggingface.co/deepvk/deberta-v1-base) |    124M    |   0.68   |     0.54    |   0.38   |    0.76   | **0.98** |      **0.80**     | **0.78** |   0.29   |   0.29   |   0.40   |    0.653   |   0.591   |141|   [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) |    150M    |   0.50   |     0.29    |   0.36   |    0.64   |   0.79   |        0.62       |   0.59   |   0.10   |   0.22   |   0.20   |    0.486   |   0.431   |142|             [ai-forever/ruBert-base](https://huggingface.co/ai-forever/ruBert-base) |    178M    |   0.67   |     0.53    |   0.39   |  **0.77** | **0.98** |        0.78       |   0.77   |   0.38   |    🥴    |    🥴    |    0.659   |    🥴    |143| [DeepPavlov/rubert-base-cased](https://huggingface.co/DeepPavlov/rubert-base-cased) |    180M    |   0.63   |     0.50    |   0.38   |    0.73   |   0.94   |        0.74       |   0.74   |   0.31   |    🥴    |    🥴    |    0.621   |    🥴    |144|       [deepvk/RuModernBERT-small](https://huggingface.co/deepvk/RuModernBERT-small) |     35M    |   0.64   |     0.50    |   0.36   |    0.72   |   0.95   |        0.73       |   0.72   |   0.47   |   0.28   |   0.26   |    0.636   |   0.563   |145|                                                     deepvk/RuModernBERT-base [this] |    150M    |   0.67   |     0.54    |   0.35   |    0.75   |   0.97   |        0.76       |   0.76   | **0.58** | **0.37** |   0.36   | **0.673**  |   0.611   |146 147## Citation148 149```150@misc{deepvk2025rumodernbert,151    title={RuModernBERT: Modernized BERT for Russian},152    author={Spirin, Egor and Malashenko, Boris and Sokolov Andrey},153    url={https://huggingface.co/deepvk/rumodernbert-base},154    publisher={Hugging Face}155    year={2025},156}157```