dots-studio/dots.llm1.base

1---2license: mit3license_link: https://huggingface.co/rednote-hilab/dots.llm1.base/blob/main/LICENSE4library_name: transformers5language:6- en7- zh8---9 10# dots111 12<p align="center">13    <img src="figures/new_logo2.png" width="300"/>14<p>15 16<p align="center">17    &nbsp&nbsp🤗 <a href="https://huggingface.co/rednote-hilab">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp 📑 <a href="https://www.arxiv.org/abs/2506.05767">Paper</a> &nbsp&nbsp 18<br>19🖥️ <a href="https://huggingface.co/spaces/rednote-hilab/dots-demo">Demo</a>&nbsp&nbsp | &nbsp&nbsp💬 <a href="figures/wechat.png">WeChat (微信)</a>&nbsp&nbsp | &nbsp&nbsp📕 <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c">rednote</a>&nbsp&nbsp20</p>21 22 23Visit our Hugging Face (click links above), search checkpoints with names starting with `dots.llm1` or visit the [dots1 collection](https://huggingface.co/collections/rednote-hilab/dotsllm1-68246aaaaba3363374a8aa7c), and you will find all you need! Enjoy!24 25 26## News27 28- 2025.06.06: We released the `dots.llm1` series. Check our [report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf) for more details!29 30 31## 1. Introduction32 33 34The `dots.llm1` model is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models. 35Leveraging our meticulously crafted and efficient data processing pipeline, `dots.llm1` achieves performance comparable to Qwen2.5-72B after pretrained on high-quality corpus without synthetic data. To foster further research, we open-source intermediate training checkpoints spanning the entire training process, providing valuable insights into the learning dynamics of large language models.36 37 38<p align="center">39  <img width="90%" src="./figures/performance.png">40</p>41 42## 2. Model Summary43 44**This repo contains the base and instruction-tuned `dots.llm1` model**. which has the following features:45 46- Type: A MoE model with 14B activated and 142B total parameters trained on high-quality corpus.47- Training Stages: Pretraining and SFT.48- Architecture: Multi-head Attention with QK-Norm in attention Layer, fine-grained MoE utilizing top-6 out of 128 routed experts, plus 2 shared experts.49- Number of Layers: 6250- Number of Attention Heads: 3251- Supported Languages: English, Chinese52- Context Length: 32,768 tokens53- License: MIT54 55The highlights from `dots.llm1` include:56 57- **Enhanced Data Processing**: We propose a scalable and fine-grained *three-stage* data processing framework designed to generate large-scale, high-quality and diverse data for pretraining.58- **No Synthetic Data during Pretraining**: High-quality non-synthetic tokens was used in base model pretraining.59- **Performance and Cost Efficiency**: `dots.llm1` is an open-source model that activates only *14B* parameters at inference, delivering both comprehensive capabilities and high computational efficiency.60- **Infrastructure**: We introduce an innovative MoE all-to-all communication and computation overlapping recipe based on interleaved 1F1B pipeline scheduling and an efficient grouped GEMM implementation to boost computational efficiency.61- **Open Accessibility to Model Dynamics**: Intermediate model checkpoints are released spanning the entire training process, facilitating future research into the learning dynamics of large language models.62 63## 3. Example Usage64 65### Model Downloads66 67<div align="center">68 69| **Model** | **#Total Params** | **#Activated Params** | **Context Length** | **Download Link** |70| :------------: | :------------: | :------------: | :------------: | :------------: |71| dots.llm1.base | 142B | 14B | 32K   | [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.base)   |72| dots.llm1.inst  | 142B | 14B |  32K   | [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.inst)   |73 74</div>75 76### Docker (recommended)77 78 79The docker images are available on [Docker Hub](https://hub.docker.com/repository/docker/rednotehilab/dots1/tags), based on the official images.80 81You can start a server via vllm.82 83```shell84docker run --gpus all \85    -v ~/.cache/huggingface:/root/.cache/huggingface \86    -p 8000:8000 \87    --ipc=host \88    rednotehilab/dots1:vllm-openai-v0.9.0.1 \89    --model rednote-hilab/dots.llm1.inst \90    --tensor-parallel-size 8 \91    --trust-remote-code \92    --served-model-name dots193```94 95Then you can verify whether the model is running successfully in the following way.96 97```shell98curl http://localhost:8000/v1/chat/completions \99    -H "Content-Type: application/json" \100    -d '{101        "model": "dots1",102        "messages": [103            {"role": "system", "content": "You are a helpful assistant."},104            {"role": "user", "content": "Who won the world series in 2020?"}105        ],106        "max_tokens": 32,107        "temperature": 0108    }'109```110 111 112### Inference with huggingface113 114We are working to merge it into Transformers ([PR #38143](https://github.com/huggingface/transformers/pull/38143)).115 116#### Text Completion117 118```python119import torch120from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig121 122model_name = "rednote-hilab/dots.llm1.base"123tokenizer = AutoTokenizer.from_pretrained(model_name)124 125model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)126 127text = "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is"128inputs = tokenizer(text, return_tensors="pt")129outputs = model.generate(**inputs.to(model.device), max_new_tokens=100)130result = tokenizer.decode(outputs[0], skip_special_tokens=True)131print(result)132```133 134#### Chat Completion135 136```python137import torch138from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig139 140model_name = "rednote-hilab/dots.llm1.inst"141tokenizer = AutoTokenizer.from_pretrained(model_name)142 143model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)144 145messages = [146    {"role": "user", "content": "Write a piece of quicksort code in C++"}147]148input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")149outputs = model.generate(input_tensor.to(model.device), max_new_tokens=200)150 151result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)152print(result)153```154 155### Inference with vllm156 157[vLLM](https://github.com/vllm-project/vllm) is a high-throughput and memory-efficient inference and serving engine for LLMs. Official support for this feature is covered in [PR #18254](https://github.com/vllm-project/vllm/pull/18254).158 159```shell160vllm serve dots.llm1.inst --port 8000 --tensor-parallel-size 8161```162 163An OpenAI-compatible API will be available at `http://localhost:8000/v1`.164 165### Inference with sglang166 167[SGLang](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. SGLang could be used to launch a server with OpenAI-compatible API service. Official support for this feature is covered in [PR #6471](https://github.com/sgl-project/sglang/pull/6471).168 169Getting started is as simple as running:170 171```shell172python -m sglang.launch_server --model-path dots.llm1.inst --tp 8 --host 0.0.0.0 --port 8000173```174 175An OpenAI-compatible API will be available at `http://localhost:8000/v1`.176 177## 4. Evaluation Results178 179Detailed evaluation results are reported in this [📑 report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf).180 181## Citation182 183If you find `dots.llm1` is useful or want to use in your projects, please kindly cite our paper:184 185```186@misc{huo2025dotsllm1technicalreport,187      title={dots.llm1 Technical Report}, 188      author={Bi Huo and Bin Tu and Cheng Qin and Da Zheng and Debing Zhang and Dongjie Zhang and En Li and Fu Guo and Jian Yao and Jie Lou and Junfeng Tian and Li Hu and Ran Zhu and Shengdong Chen and Shuo Liu and Su Guang and Te Wo and Weijun Zhang and Xiaoming Shi and Xinxin Peng and Xing Wu and Yawen Liu and Yuqiu Ji and Ze Wen and Zhenhai Liu and Zichao Li and Zilong Liao},189      year={2025},190      eprint={2506.05767},191      archivePrefix={arXiv},192      primaryClass={cs.CL},193      url={https://arxiv.org/abs/2506.05767}, 194}195```