feynmanzhao/chinese-modernbert-large-wwm

Chinese ModernBERT

Model Description

Chinese ModernBERT is an encoder-only transformer model pre-trained from scratch on 1.2 trillion Chinese tokens. It is designed to be a modern, efficient, and powerful foundation model for Chinese NLP tasks.

The model incorporates several key architectural and training advancements:

• —Modern Architecture: Based on the ModernBERT structure, it features a 28-layer, 1024-hidden dimension pre-norm Transformer with RMSNorm, GeGLU activations, and bias-free linear layers.
• —Efficient Long Context: The model supports a native context window of 8,192 tokens. This is achieved through a two-stage pre-training process and an alternating local/global attention mechanism coupled with RoPE, making it highly efficient for long sequence tasks.
• —Chinese-Centric Tokenizer: It uses a custom 32k BPE vocabulary trained on high-quality Chinese text. This improves tokenization efficiency (chars/token) and reduces the embedding parameter budget, allocating more capacity to the compute core.
• —Advanced Masking Strategy: The model was pre-trained using Whole-Word Masking (WWM) with a dynamic masking curriculum. The masking rate starts high (30%) to encourage global reasoning and gradually decreases to 15% for local refinement.

Intended Uses & Limitations

Intended Use

The model is primarily intended to be used as a text encoder for a variety of downstream tasks, especially where latency and long-context capabilities are important.

• —Text Embeddings: Generating embeddings for retrieval, ranking, and semantic search.
• —Sentence Similarity: Calculating the similarity between pairs of sentences.
• —Fine-tuning: Serving as a base model for classification, clustering, and other Chinese NLP tasks.

Limitations

• —The model was pre-trained exclusively on Chinese text. Its performance on other languages, code, or mathematical reasoning is limited.
• —The reported STS benchmark results were achieved by fine-tuning on open-source contrastive datasets. Performance may vary depending on the fine-tuning data and procedure.

How to Use

Training Data

The model was pre-trained on approximately 1.2 trillion Chinese tokens from a mixture of high-quality, deduplicated sources: | Dataset | Ratio | |-----------------------------|-------| | Baike (custom) | 0.03 | | CCI3-HQ | 0.57 | | CCI4 Chinese (FinWeb 34) | 0.10 | | CCI4 Chinese (FinWeb 45) | 0.10 | | OpenCSG Cosmopedia-Chinese | 0.20 |

Evaluation Results

CLUE Benchmark

The model shows competitive performance on the CLUE (Chinese Language Understanding Evaluation) benchmark dev set. Fine-tuning protocol: 3 epochs, peak LR 3e-5, linear decay.

Inference Throughput

The model demonstrates excellent throughput, especially for long sequences, due to its alternating local/global attention and efficient architecture. Benchmark run on a single NVIDIA A100 GPU with bfloat16.

Semantic Textual Similarity (SimCLUE)

The model was fine-tuned on open contrastive datasets to evaluate its potential for semantic similarity tasks. It achieves strong results and surpasses comparable models like Qwen-0.6B-embedding in this open-data setting.

Model Architecture

How to Get Started With the Model

from transformers import AutoTokenizer,AutoModel
tokenizer = AutoTokenizer.from_pretrained("feynmanzhao/chinese-modernbert-large-wwm", trust_remote_code=True)
model = AutoModel.from_pretrained("feynmanzhao/chinese-modernbert-large-wwm")

Citation

@misc{zhao2025chinesemodernbert,
      title={Chinese ModernBERT with Whole-Word Masking}, 
      author={Zeyu Zhao and Ningtao Wang and Xing Fu and Yu Cheng},
      year={2025},
      eprint={2510.12285},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}