CofeAI/Tele-FLM

1---2license: apache-2.03---4 5 6# Tele-FLM7Tele-FLM (aka FLM-2) is a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgement capabilities. 8Built upon the decoder-only transformer architecture, it has been trained on approximately 2T tokens.9Tele-FLM demonstrates superior performances at its scale, and sometimes surpass larger models.10In addition to sharing the model weights, we provide the core designs, engineering practices, and training details, anticipating their benefits for both academic and industrial communities.11 12## Model Details13 14- **Developed by:** BAAI & TeleAI15- **Language(s):** English; Chinese; Other languages16- **License:** Apache 2.017 18## Technical Report19 20[Tele-FLM Technical Report](https://arxiv.org/pdf/2404.16645)21 22 23## Bias, Risks, and Limitations24 25Although we've made extensive efforts to thoroughly clean and filter the training corpus for the model, due to the open nature of the dataset, the model may still have picked up on some unsafe examples. Consequently, the model may still generate unexpected content, including but not limited to discrimination, bias, or offensive language. We would like to strongly advise users not to spread any unsafe content generated by the model. The project developers cannot be held responsible for any repercussions stemming from the dissemination of harmful information.26 27 28## Quick Start29 30Use the code below to get started with Tele-FLM.31 32```python33import torch34from transformers import AutoTokenizer, AutoModelForCausalLM35tokenizer = AutoTokenizer.from_pretrained('CofeAI/Tele-FLM', trust_remote_code=True)36model = AutoModelForCausalLM.from_pretrained('CofeAI/Tele-FLM', torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto", trust_remote_code=True)37inputs = tokenizer('Beijing is the capital of China.', return_tensors='pt').to(model.device)38generated = model.generate(**inputs, max_new_tokens=128, repetition_penalty=1.03)39print(tokenizer.decode(generated.cpu()[0], skip_special_tokens=True))40```41 42## Training Details43 44### Training Data45Our training dataset comprises a variety of domains, as detailed in the table below.46The total amount of data is roughly 2 trillion, with English and Chinese data in a ratio of about 2:1.47In line with the methodology of GPT-4, we collected some instruct data and incorporated it into our pre-training data after removing the test sets of common datasets using the strict n-gram-based method. We deliberately avoid “training on the test set” or any other benchmark-oriented trick.48|Domain |Language|Sampling Prop.  |Epochs   |Disk Size    |49|-------|:--------------:|:--------------:|:-------:|:-----------:|50| Webtext        |en, zh   | 75.21% | 1.0 | 5.9 TB   |51| Code            |code, zh     | 9.81%  | 1.0 | 528.1 GB |52| Book            |en, zh    | 7.17%  | 0.8 | 647.6 GB |53| WorldKnowledge |multi, en, zh     | 2.87%  | 2.5 | 67.5 GB  |54| QA              |en, zh     | 2.12%  | 1.0 | 159.2 GB |55| AcademicPaper   |en    | 0.99%  | 1.0 | 54.4 GB  |56| Profession-Law             |zh     | 1.04%  | 1.0 | 84.2 GB  |57| Profession-Math             |math    | 0.62%  | 2.0 | 6.1 GB   |58| Profession-Patent           |zh    | 0.14%  | 1.0 | 10.4 GB  |59| Profession-Medical           |zh   | 0.02%  | 1.0 | 1.2 GB   |60| ClassicalChinese |zh   | 0.02%  | 2.5 | 0.5 GB   |61 62 63### Model Architecture64We adopt the architecture of FLM-101B as the backbone for Tele-FLM, with several modifications:65- Rotary Positional Embedding (RoPE)66- RMSNorm for normalization67- SwiGLU for activation function68- Linear bias disabled69- Embedding and language model head untied70- Input and output multiplier71 72Consequently, Tele-FLM is largely compatible with Llama architecturally.73To maximize convenience for the community, we made minimal adjustments to Llama's code to adapt it to Tele-FLM and released it as open source.74 75In the pre-training stage, we employ μP for optimal hyperparameter search. The μP model (Tele-FLM_μP) is architecturally identical to Tele-FLM except for the model width.76The architecture of Tele-FLM and Tele-FLM_μP is listed below.77For more details of μP, please refer to our technical report and the original Tensor Program papers.78 79| Models | layer<br>number | attention<br>heads| hidden<br>size | ffn hidden<br>size| vocab<br>size | context<br>length | param size<br>(M) |80|--------|--------------|----------------|-------------|----------------|------------|----------------|----------------|81| Tele-FLM | 64         | 64             | 8,192       | 21,824         | 80,000     | 4,096          | 52,850         |82| Tele-FLM_μP | 64      | 4              | 512         | 1,344          | 80,000     | 4,096          | 283            |83 84 85 86 87### Training Hyperparameters88 89Due to the smaller size, Tele-FLM_μP allows for significantly more experimental runs within fixed time and resource constraints.90We searched seven hyperparameters for pretraining. All the hyperparameters are shown below.91 92 93| Searched Hyperparameters                       ||| Non-Searched Hyperparameters    ||94|--------------------------------------------|-|-|-|----------------------------------|95| Learning Rate                     | 1.5e-4      || LR Schedule Type     | cosine    |96| Matrix Learning Rate              | 1.5e-4      || LR Schedule (tokens) | 2.5T      |97| Minimum Learning Rate             | 1.5e-5      || Warmup Step          | 2,000     |98| Standard Deviation                | 4e-3        || Clip Grad            | 1.0       |99| Matrix Standard Deviation         | 4.242e-3    || Weight Decay         | 0.0       |100| Input Mult                        | 1.0         || Batch Size (tokens)  | 5,505,024 |101| Output Mult                       | 3.125e-2    || RoPE Theta           | 10,000    |102 103 104### Training Loss105 106 107<p align="center" width="100%">108  <a><img src="figures/train_loss.png" alt="nexa-octopus" style="width: 90%; min-width: 500px; display: block; margin: auto;"></a>109</p>110 111 112#### Hardware113 114Tele-FLM is trained on a cluster of 112 A800 SXM4 GPU servers, each with 8 NVLink A800 GPUs and 2TB of RAM. 115The nodes have varied CPU configurations: 96 nodes with Intel 8358 (128x 2.60GHz) CPUs and 16 nodes with AMD 7643 (96x 2.30GHz) CPUs. 116All nodes are interconnected via InfiniBand (IB). The training process lasted around two months, including downtime due to unexpected factors.117 118#### Software119 120Tele-FLM utilizes 3D parallel training, combining the prevailing methodologies: data parallelism, tensor parallelism, and pipeline parallelism.121The parallel training setup for Tele-FLM is configured as follows: tensor parallel=4, pipeline parallel=2, and data parallel=112.122 123 124 125 126## Evaluation127 128### English129 130#### Open LLM Leaderboard131| Model      | Average |   ARC   | HellaSwag |  MMLU  | TruthfulQA | WinoGrade |  GSM8K | HumanEval |   BBH  |132|------------|:-------:|:-------:|:---------:|:------:|:----------:|:---------:|:------:|:---------:|:------:|133|            |         | 25-shot |  10-shot  | 5-shot |  zero-shot |   5-shot  | 5-shot | zero-shot | 3-shot |134| LLAMA2-70B |  63.39  |  67.32  |   87.33   |  69.83 |    44.92   |   83.74   |  54.06 |   46.95   |  52.94 |135| LLAMA2-13B |  50.29  |  59.39  |   82.13   |  55.77 |    37.38   |   76.64   |  22.82 |   28.66   |  39.52 |136| LLAMA-65B  |  56.98  |  63.48  |   86.09   |  63.93 |    43.43   |   82.56   |  37.23 |   33.54   |  45.54 |137| LLAMA-13B  |  46.20  |  56.23  |   80.93   |  47.67 |    39.48   |   76.24   |  7.58  |   23.78   |  37.72 |138| Tele-FLM   |  56.60  |  59.47  |   82.25   |  64.00 |    43.09   |   79.40   |  45.19 |   34.76   |  44.60 |139 140### Chinese141 142#### OpenCompass143| Model        | Average | C-Eval | CMMLU | C3    | CHID  | CSL   |144|--------------|:-------:|:------:|:-----:|:-----:|:-----:|:-----:|145| GPT-4        | 76.64   | 69.90  | 71.00 | 95.10 | 82.20 | 65.00 |146| GPT-3.5      | 61.86   | 52.50  | 53.90 | 85.60 | 60.40 | 56.90 |147| Qwen1.5-72B  | 80.45   | 83.72  | 83.09 | 81.86 | 91.09 | 62.50 |148| Qwen-72B     | 83.00   | 83.30  | 83.60 | 95.80 | 91.10 | 61.20 |149| DeepSeek-67B | 73.46   | 66.90  | 70.40 | 77.80 | 89.10 | 63.10 |150| Tele-FLM     | 71.13   | 65.48  | 66.98 | 66.25 | 92.57 | 64.38 |151 152 153## Acknowledgements154This work was supported by the National Science and Technology Major Project (No. 2022ZD0116314).155 156 157## Citation158If you find our work helpful, please consider citing it.159```160@article{tele-flm-2024,161  author       = {Xiang Li and Yiqun Yao and Xin Jiang and Xuezhi Fang and Chao Wang and Xinzhang Liu and Zihan Wang and Yu Zhao and Xin Wang and Yuyao Huang and Shuangyong Song and Yongxiang Li and Zheng Zhang and Bo Zhao and Aixin Sun and Yequan Wang and Zhongjiang He and Zhongyuan Wang and Xuelong Li and Tiejun Huang},162  title        = {Tele-FLM Technical Report},163  journal      = {CoRR},164  volume       = {abs/2404.16645},165  year         = {2024},166  url          = {https://doi.org/10.48550/arXiv.2404.16645},167  doi          = {10.48550/ARXIV.2404.16645},168  eprinttype    = {arXiv},169  eprint       = {2404.16645},170}171```172