openkg/knowlm-13b-diff

1---2license: apache-2.03language:4- en5- zh6pipeline_tag: text-generation7tags:8- code9---10 11 12<p align="center" width="100%">13<a href="" target="_blank"><img src="https://github.com/zjunlp/KnowLM/blob/main/assets/KnowLM.png?raw=true" alt="ZJU-KnowLM" style="width: 40%; min-width: 40px; display: block; margin: auto;"></a>14</p>15 16 17> This is the result of the weight difference between `Llama 13B` and `ZhiXi-13B`. You can click [here](https://github.com/zjunlp/KnowLM) to learn more.18 19 20# Knowledgable Large Language Model Framework.21 22With the rapid development of deep learning technology, large language models such as ChatGPT have made substantial strides in the realm of natural language processing. However, these expansive models still encounter several challenges in acquiring and comprehending knowledge, including the difficulty of updating knowledge and potential knowledge discrepancies and biases, collectively known as knowledge fallacies. The KnowLM project endeavors to tackle these issues by launching an open-source large-scale knowledgable language model framework and releasing corresponding models. 23 24The project's `initial phase` introduced a knowledge extraction LLM based on LLaMA, dubbed **ZhiXi** (**智析**, which means intelligent analysis of data for knowledge extraction). To integrate the capacity of Chinese understanding into the language models without compromising their inherent knowledge, we firstly <b>(1) use Chinese corpora for the full-scale pre-training with LLaMA (13B), augment the language model's understanding of Chinese and improve its knowledge richness while retaining its original English and code capacities;</b> Then <b>(2) we fine-tune the model obtained from the first step with an instruction dataset, thus bolstering the language model's understanding of human instructions for knowledge extraction.</b>25- ❗Please note that this project is still undergoing optimization, and the model weights will be regularly updated to support new features and models!26 27**The features of this project are as follows:**28 29- Centered on knowledge and large models, a **full-scale pre-training** of the large model, such as LLaMA, is conducted using the built Chinese&English pre-training corpus.30- Based on the technology of **KG2Instructions**, the knowledge extraction tasks, including NER, RE, and IE, are optimized and can be completed using human instructions.31- Using the built Chinese instruction dataset (approximately 1400K), LoRA fine-tuning is used to enhance the model's understanding of human instructions.32- The weights of the pre-training model and LoRA's instruction fine-tuning are open-sourced.33- The **full-scale pre-training code** (providing conversion, construction, and loading of large corpora) and **LoRA instruction fine-tuning code** are open-sourced (support multi-machine multi-GPU).34 35 36All weights have been uploaded to HuggingFace🤗. It should be noted that all the following effects are based on `ZhiXi-13B-Diff`. If you have downloaded `ZhiXi-13B-Diff-fp16`, there may be some variations in the effects.37| Model Name       | Train Method    | Weight Type          | Size     | Download Link                           | Notes                                                         |38| -------------- | ------------ | --------------------- | -------- | ---------------------------------- | ------------------------------------------------------------ |39| ZhiXi-13B-Diff | Full Pretraining   | Differential Weights | 48GB     | [HuggingFace](https://huggingface.co/zjunlp/zhixi-13b-diff) <br/> [GoogleDrive](https://drive.google.com/drive/folders/1PZDqZNaBJYQYeON1-9aFBtagktEWAtUK?usp=drive_link)| Restoring the pre-trained weights (i.e. **ZhiXi-13B**) needs to match the weights of `LLaMA-13B`, please refer to [here](#2-2) for specific instructions. |40| ZhiXi-13B-Diff-fp16 | Full Pretraining   | Differential Weights(fp16) | 24GB     | [HuggingFace](https://huggingface.co/zjunlp/zhixi-13b-diff-fp16) <br/> [Google Drive](https://drive.google.com/drive/folders/1LYm-HUSSQ5Rl8nqZcswdiSpcP9xYTXaO?usp=sharing) | The main difference with `ZhiXi-13B-Diff` is the adoption of the `fp16` format for storage, which reduces memory usage. However, it may result in slight differences in the weights obtained from our actual training, which can slightly impact performance. For specific usage instructions, please refer to [here](#2-2) for specific instructions. |41| ZhiXi-13B-LoRA | LoRA Instruction-tuning | LoRA Weights              | 251MB    | [HuggingFace](https://huggingface.co/zjunlp/zhixi-13b-lora) <br/>  [GoogleDrive](https://drive.google.com/drive/folders/1GLyaWIyDIayudrQhb_tJYoNPAUk1xByS?usp=drive_link) | It needs to be used with **ZhiXi-13B**. For specific instructions, please refer to [here](#2-4).          |42| ZhiXi-7B Series   | Coming soon     | Coming soon            | Coming soon | Coming soon                           | Coming soon                                                 |43 44 45## NEWS46- \[**June 2023**\] The project name has been changed from CaMA to KnowLM.47- \[**June 2023**\] Release the first version of pre-trained weights and the LoRA weights.48 49## Contents50 51- [Quick Start](#1)52  - [Environment Configuration](#1-1)53  - [Model Weight(Pretrain and LoRA)](#1-2)54  - [Model Usage Guide](#1-4)55  - [Information Extraction Prompt](#1-5)56- [Training Details](#2)57  - [Pertraining data and Pretraining scripts](#2-1)58  - [Instruction data and Instruction-tuning scripts](#2-3)59- [Limitations](#3)60- [TODO List](#4)61- [FAQ](#5)62- [Acknowledgments/Contributors/Citations](#6)63 64 65<h2 id="1">1. Quick Start</h2>66 67<h3 id="1-1">1.1 Environment Configuration</h3>68 69```shell70conda create -n knowlm python=3.9 -y71conda activate knowlm72pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu11673pip install -r requirements.txt74```75 76 77<h3 id="1-2">1.2 Pretraining model weight acquisition and restoration</h3>78 79❗❗❗ Note that in terms of hardware, performing step `2.2`, which involves merging LLaMA-13B with KnowLM-13B-Diff, requires approximately **100GB** of RAM, with no demand for VRAM (this is due to the memory overhead caused by our merging strategy. For your convenience, we have provided the fp16 weights at this link: https://huggingface.co/zjunlp/zhixi-13b-diff-fp16. **fp16 weights require less memory but may slightly impact performance**. We will improve our merging approach in future updates, and we are currently developing a 7B model as well, so stay tuned). For step `2.4`, which involves inference using `ZhiXi`, a minimum of **26GB** of VRAM is required.80 81**1. Download LLaMA 13B and KnowLM-13B-Diff**82 83Please click [here](https://forms.gle/jk851eBVbX1m5TAv5) to apply for the official pre-training weights of LLaMA from `meta`. In this case, we are using the `13B` version of the model, so you only need to download the `13B` version. Once downloaded, the file directory will be as follows:84 85```shell86|-- 13B87|	|-- checklist.chk88|	|-- consolidated.00.pth89|	|-- consolidated.01.pth90|	|-- params.json91|-- llama.sh92|-- tokenizer.model93|-- tokenizer_checklist.chk94```95 96You can use the following command to download the `KnowLM-13B-Diff` file (assuming it is saved in the `./knowlm-diff` folder):97```shell98python tools/download.py --specify --repo_name openkg/knowlm-13b-diff --download_path ./knowlm-diff99```100 101> :exclamation:Noted. If the download is interrupted, please repeat the command mentioned above. HuggingFace provides the functionality of resumable downloads, allowing you to resume the download from where it was interrupted.102 103**2. Use the conversion script provided by huggingface**104 105To convert the original LLaMA-13B model into the HuggingFace format, you can use the provided script file by HuggingFace, which can be found [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). Below is the command to run the script (assuming the downloaded original files(LLaMA-13B) are located in `./` and you want the converted files to be stored in `./converted`):106 107```shell108python convert_llama_weights_to_hf.py --input_dir ./ --model_size 13B --output_dir ./converted109```110 111**3. Restore KnowLM 13B**112 113Use the script we provided, located at `./tools/weight_diff.py`, execute the following command, and you will get the complete `KnowLM` weight:114 115```shell116python tools/weight_diff.py recover --path_raw ./converted --path_diff ./knowlm-diff --path_tuned ./knowlm --check_integrity_naively False117```118 119The final complete KnowLM weights are saved in the `./knowlm` folder.120 121 122<h3 id="1-3">1.3 Instruction tuning LoRA weight acquisition</h3>123 124Use the script file we provided, located at `./tools/download.py`, execute the following command to get the LoRA weight (assuming the saved path is located at `./LoRA`):125 126```shell127python tools/download.py --download_path ./lora --specify --repo_name openkg/knowlm-13b-lora128```129 130The final complete weights are saved in the `./lora` folder.131 132 133 134<h3 id="1-4">1.4 Model Usage Guide</h3>135 136**1. Usage of Pretraining Model**137 138We offer two methods: the first one is **command-line interaction**, and the second one is **web-based interaction**, which provides greater flexibility.139 1401. Use the following command to enter **command-line interaction**:141 142   ```shell143   python examples/generate_finetune.py --base_model ./knowlm --interactive144   ```145 146   The disadvantage is the inability to dynamically change decoding parameters.147 1482. Use the following command to enter **web-based interaction**:149 150   ```shell151   python examples/generate_finetune_web.py --base_model ./knowlm152   ```153   Here is a screenshot of the web-based interaction:154   <p align="center" width="100%">155   <a href="" target="_blank"><img src="https://github.com/zjunlp/KnowLM/blob/main/assets/finetune_web.jpg?raw=true" alt="finetune-web" style="width: 100%; min-width: 100px; display: block; margin: auto;"></a>156   </p>157 158 159**2. Usage of Instruction tuning Model**160 161Here, we provide a web-based interaction method. Use the following command to access the web:162 163```shell164python examples/generate_lora_web.py --base_model ./knowlm --lora_weights ./lora165```166 167Here is a screenshot of the web-based interaction:168<p align="center" width="100%">169<a href="" target="_blank"><img src="https://github.com/zjunlp/KnowLM/blob/main/assets/lora_web.png?raw=true" alt="finetune-web" style="width: 100%; min-width: 100px; display: block; margin: auto;"></a>170</p>171 172The `instruction` is a required parameter, while `input` is an optional parameter. For general tasks (such as the examples provided in section `1.3`), you can directly enter the input in the `instruction` field. For information extraction tasks (as shown in the example in section `1.2`), please enter the instruction in the `instruction` field and the sentence to be extracted in the `input` field. We provide an information extraction prompt in section `2.5`. 173 174If you want to perform batch testing, please modify the `examples/generate_lora.py` file and update the examples and hyperparameters in the variable `cases`.175 176 177 178<h3 id="1-5">1.5 Information Extraction Prompt</h3>179 180For information extraction tasks such as named entity recognition (NER), event extraction (EE), and relation extraction (RE), we provide some prompts for ease of use. You can refer to this [link](https://github.com/zjunlp/KnowLM/blob/main/examples/ie_prompt.py) for examples. Of course, you can also try using your own prompts.181 182Here is a [case](https://github.com/zjunlp/DeepKE/blob/main/example/llm/InstructKGC/README.md) where KnowLM-13B-LoRA is used to accomplish the instruction-based knowledge graph construction task in CCKS2023.183 184 185<h2 id="2">2. Training Details</h2>186 187> The following figures illustrates the entire training process and dataset construction. The training process is divided into two stages:188>189>  (1) Full pre-training stage. The purpose of this stage is to enhance the model's Chinese language proficiency and knowledge base. 190>191>  (2) Instruction tuning stage using LoRA. This stage enables the model to understand human instructions and generate appropriate responses.192 193![](https://github.com/zjunlp/KnowLM/blob/main/assets/main_new.jpg?raw=true)194 195<h3 id="2-1">2.1 Dataset Construction (Pretraining)</h3>196 197In order to enhance the model's understanding of Chinese while preserving its original code and English language capabilities, we did not expand the vocabulary. Instead, we collected Chinese corpora, English corpora, and code corpora. The Chinese corpora were sourced from Baidu Baike, Wudao, and Chinese Wikipedia. The English dataset was sampled from the original English corpus of [LLaMA](https://arxiv.org/pdf/2302.13971.pdf), with the exception of the Wikipedia data. The original paper's English Wikipedia data was up until August 2022, and **we additionally crawled data from September 2022 to February 2023, covering a total of six months.** As for the code dataset, due to the low-quality code in the `Pile` dataset, we crawled code data from GitHub and LeetCode. A portion of the data was used for pre-training, while another portion was used for fine-tuning with instructions.198 199For the crawled datasets mentioned above, we employed a heuristic approach to filter out harmful content. Additionally, we removed duplicate data.200 201<h3 id="2-2">2.2 Training Process (Pretraining)</h3>202 203Detailed data processing code, training code, complete training scripts, and detailed training results can be found in [./pretrain](https://github.com/zjunlp/KnowLM/blob/main/pretrain).204 205Before training, we need to tokenize the data. We set the maximum length of a single sample to `1024`, while most documents are much longer than this. Therefore, we need to partition these documents. **We designed a greedy algorithm to split the documents, with the goal of ensuring that each sample consists of complete sentences and minimizing the number of segments while maximizing the length of each sample.** Additionally, due to the diversity of data sources, we developed a comprehensive data preprocessing tool that can process and merge data from various sources. Finally, considering the large amount of data, loading it directly into memory would impose excessive hardware pressure. Therefore, we referred to [DeepSpeed-Megatron](https://github.com/bigscience-workshop/Megatron-DeepSpeed/tree/main/tools) and used the `mmap` method to process and load the data. This involves loading the indices into memory and accessing the corresponding data on disk when needed.206 207Finally, we performed pre-training on 5.5 million Chinese samples, 1.5 million English samples, and 0.9 million code samples. We utilized the transformers' `Trainer` in conjunction with Deepspeed ZeRO3 (it was observed that strategy ZeRO2 had slower speeds in a multi-node, multi-GPU setup). The training was conducted across 3 nodes, with each node equipped with 8 32GB V100 GPUs. The table below showcases our training speeds:208 209| Parameter                                         | Values         |210| ------------------------------------------------- | -------------- |211| micro batch size                                  | 20             |212| gradient accumulation                             | 3              |213| global batch size                                 | 20\*3\*24=1440 |214| Time-consuming of a step                          | 260s           |215 216 217 218<h3 id="2-3">2.3 Dataset Construction (Instruction tuning)</h3>219 220In addition to incorporating general capabilities such as reasoning and coding, we have also introduced additional information extraction abilities, including NER (Named Entity Recognition), IE (Information Extraction), and EE (Event Extraction), into the current homogeneous models. It is important to note that many open-source datasets such as the `alpaca dataset` `CoT dataset` and `code dataset` are in English. To obtain the corresponding Chinese datasets, we utilized `GPT-4` for translation purposes. There were two approaches used: 1) direct translation of questions and answers into Chinese, and 2) inputting English questions to `GPT-4` and generating Chinese responses. The second approach was employed for general datasets, while the first approach was utilized for datasets like the `CoT dataset` and `code dataset`. These datasets are readily available online.221 222 223For information extraction datasets, we used open-source datasets such as `CoNLL`, `ACE`, `CASIS`, and others to construct corresponding English instructions for generating the required training format. For the Chinese part, for NER and EE tasks, we utilized open-source datasets such as `DualEE`, `PEOPLE DAILY`, and others, and then created corresponding Chinese instructions to synthesize the required training format. As for the RE task, we built a dataset called [KG2Instruction](https://arxiv.org/abs/2305.11527). Specifically, we used Chinese Wikipedia data and BERT for Chinese entity recognition. We then aligned the recognized entities with the Wikipedia index. Due to potential ambiguity (i.e., a Chinese entity may have multiple indexes, such as `apple` referring to both a fruit and a company), we devised a strategy to disambiguate the entities. Subsequently, we used a distantly supervised method to generate possible triplets and applied predefined rules to filter out illegal or incorrect triplets. Finally, with the help of crowdsourcing, we refined the obtained triplets. Following that, we constructed corresponding Chinese instructions to generate the required training format.224 225In addition, we manually constructed a general Chinese dataset and translated it into English using the second approach. Finally, our data distribution is as follows:226 227| Dataset           | Number |228| -------------------- | ---- |229| COT Datasets (Chinese, English)        |   202333   |230| General Datasets (Chinese, English) |   105216   |231| Code Datasets (Chinese, English) |   44688   |232| Information Extraction Datasets (English)   |   537429   |233| Information Extraction Datasets (Chinese)   |   486768   |234 235**Flow diagram of KG2Instruction and other instruction fine-tuning datasets** 236<p align="center" width="100%">237<a href="" target="_blank"><img src="https://github.com/zjunlp/KnowLM/blob/main/assets/kg2instructions-en.png?raw=true"style="width: 90%; min-width: 90px; display: block; margin: auto;"></a>238</p>239 240<h3 id="2-4">2.4 Training Process (Instruction tuning)</h3>241 242Currently, most instruction tuning scripts using LoRA are based on [alpaca-lora](https://github.com/tloen/alpaca-lora/), so we will not go into detail here. Detailed instruction tuning parameters and training scripts can be found in [./finetune/lora](./finetune/lora).243 244 245 246<h2 id="3">3. Limitations</h2>247 248Due to time constraints, hardware limitations, and technical reasons, our model has limitations, including but not limited to:249 250- Our intruction tuning process does not involve full tuning. Instead, we use the LoRA approach for instruction tuning.251- Our model does not currently support multi-turn conversations.252- While we strive to ensure the usefulness, reasonableness, and harmlessness of the model's outputs, toxic outputs may still occur in some scenarios.253 254- The pretraining is not exhaustive. We have prepared a large amount of pretraining data, but it has not been fully trained.255 256- ······257 258  259 260<h2 id="4">4. TODO List</h2>261 262- Instruction tuning using full tuning instead of LoRA version is being trained and will be released soon. 263- New instruction tuning weights using LoRA will be updated shortly.264- New models (Llama-7b, Falcon-7b) are being trained (We have limited GPUs!).265- New abilities such as molecule and protein generation with [Mol-Instructions](https://github.com/zjunlp/Mol-Instructions), a large-scale biomolecules instruction dataset for large language models.266- supporting llama.cpp267- ......268 269 270 271<h2 id="5">5. FAQ</h2>272 273- Question: What should I do if the model encounters � during decoding?274 275  Answer: If this symbol appears in the middle of the decoded sentence, we recommend changing the input. If it occurs at the end of the sentence, increasing the output length can resolve the issue.276 277- Question: Why do I get different results with the same decoding parameters?278 279  Answer: It is possible that you have enabled `do_sample=True`. It could also be due to the order of execution. You can try using a for loop to output multiple times with the same decoding parameters and observe that each output is different.280  281- Question: Why is the extraction or answer quality not good?282 283  Answer: Please try changing the decoding parameters.284 285<h2 id="7">6. Others</h2>286 287<h3 id="7-1">6.1 Contributors</h3>288 289Ningyu Zhang, Haofen Wang, Xiang Chen, Jintian Zhang, Xiaozhuan Liang, Zhen Bi, Honghao Gui, Jing Chen, Runnan Fang, Xiaohan Wang, Shengyu Mao, Shuofei Qiao, Yixin Ou, Lei Li, Yunzhi Yao, Peng Wang, Siyuan Cheng, Bozhong Tian, Mengru Wang, Zhoubo Li, Yinuo Jiang, Yuqi Zhu, Hongbin Ye, Zekun Xi, Xinrong Li, Huajun Chen290 291 292 293<h3 id="7-2">6.2 Citation</h3>294 295If you use our repository, please cite the following related papers:296 297```bibtex298@article{deepke-llm,299  author = {Ningyu Zhang, Jintian Zhang, Xiaohan Wang, Honghao Gui, Yinuo Jiang, Xiang Chen, Shengyu Mao, Shuofei Qiao, Zhen Bi, Jing Chen, Xiaozhuan Liang, Yixin Ou, Ruinan Fang, Zekun Xi, Xin Xu, Liankuan Tao, Lei Li, Peng Wang, Zhoubo Li, Guozhou Zheng, Huajun Chen},300  title = {DeepKE-LLM: A Large Language Model Based Knowledge Extraction Toolkit},301  year = {2023},302  publisher = {GitHub},303  journal = {GitHub repository},304  howpublished = {\url{https://github.com/}},305}306```307 308 309<h3 id="7-3">6.3 Acknowledgment</h3>310 311We are very grateful to the following open source projects for their help:312 313- [Meta AI LLaMA](https://arxiv.org/abs/2302.13971v1)314 315- [Huggingface Transformers Llama](https://github.com/huggingface/transformers/tree/main/src/transformers/models/llama)316 317- [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) and [Alpaca-LoRA](https://github.com/tloen/alpaca-lora)318 319- [Vicuna](https://vicuna.lmsys.org/)320 321- [Llama-X](https://github.com/AetherCortex/Llama-X)