AdaptLLM/finance-tasks

1---2configs:3- config_name: ConvFinQA4  data_files:5  - split: test6    path: ConviFinQA/test.json7- config_name: FiQA_SA8  data_files:9  - split: test10    path: FiQA_SA/test.json11- config_name: FPB12  data_files:13  - split: test14    path: FPB/test.json15- config_name: Headline16  data_files:17  - split: test18    path: Headline/test.json19- config_name: NER20  data_files:21  - split: test22    path: NER/test.json23task_categories:24- text-classification25- question-answering26- zero-shot-classification27language:28- en29tags:30- finance31---32 33# Adapting LLMs to Domains via Continual Pre-Training (ICLR 2024)34This repo contains the **evaluation datasets** for our paper [Adapting Large Language Models via Reading Comprehension](https://huggingface.co/papers/2309.09530).35 36We explore **continued pre-training on domain-specific corpora** for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to **transform large-scale pre-training corpora into reading comprehension texts**, consistently improving prompting performance across tasks in biomedicine, finance, and law domains. **Our 7B model competes with much larger domain-specific models like BloombergGPT-50B**. 37 38### [2024/11/29] 🤗 Introduce the multimodal version of AdaptLLM at [AdaMLLM](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains), for adapting MLLMs to domains 🤗39 40**************************** **Updates** ****************************41* 2024/11/29: Released [AdaMLLM](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains) for adapting MLLMs to domains42* 2024/9/20: Our [research paper for Instruction-Pretrain](https://huggingface.co/papers/2406.14491) has been accepted by EMNLP 202443* 2024/8/29: Updated [guidelines](https://huggingface.co/datasets/AdaptLLM/finance-tasks) on evaluating any 🤗Huggingface models on the domain-specific tasks44* 2024/6/22: Released the [benchmarking code](https://github.com/microsoft/LMOps/tree/main/adaptllm)45* 2024/6/21: Released the general version of AdaptLLM at [Instruction-Pretrain](https://huggingface.co/instruction-pretrain)46* 2024/4/2: Released the [raw data splits (train and test)](https://huggingface.co/datasets/AdaptLLM/ConvFinQA) of all the evaluation datasets47* 2024/1/16: Our [research paper for AdaptLLM](https://huggingface.co/papers/2309.09530) has been accepted by ICLR 202448* 2023/12/19: Released our [13B base models](https://huggingface.co/AdaptLLM/law-LLM-13B) developed from LLaMA-1-13B49* 2023/12/8: Released our [chat models](https://huggingface.co/AdaptLLM/law-chat) developed from LLaMA-2-Chat-7B50* 2023/9/18: Released our [paper](https://huggingface.co/papers/2309.09530), [code](https://github.com/microsoft/LMOps), [data](https://huggingface.co/datasets/AdaptLLM/law-tasks), and [base models](https://huggingface.co/AdaptLLM/law-LLM) developed from LLaMA-1-7B51 52 53## 1. Domain-Specific Models54### LLaMA-1-7B55In our paper, we develop three domain-specific models from LLaMA-1-7B, which are also available in Huggingface: [Biomedicine-LLM](https://huggingface.co/AdaptLLM/medicine-LLM), [Finance-LLM](https://huggingface.co/AdaptLLM/finance-LLM) and [Law-LLM](https://huggingface.co/AdaptLLM/law-LLM), the performances of our AdaptLLM compared to other domain-specific LLMs are:56 57<p align='center'>58    <img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/6efPwitFgy-pLTzvccdcP.png" width="700">59</p>60 61### LLaMA-1-13B62Moreover, we scale up our base model to LLaMA-1-13B to see if **our method is similarly effective for larger-scale models**, and the results are consistently positive too: [Biomedicine-LLM-13B](https://huggingface.co/AdaptLLM/medicine-LLM-13B), [Finance-LLM-13B](https://huggingface.co/AdaptLLM/finance-LLM-13B) and [Law-LLM-13B](https://huggingface.co/AdaptLLM/law-LLM-13B).63 64### LLaMA-2-Chat65Our method is also effective for aligned models! LLaMA-2-Chat requires a [specific data format](https://huggingface.co/blog/llama2#how-to-prompt-llama-2), and our **reading comprehension can perfectly fit the data format** by transforming the reading comprehension into a multi-turn conversation. We have also open-sourced chat models in different domains: [Biomedicine-Chat](https://huggingface.co/AdaptLLM/medicine-chat), [Finance-Chat](https://huggingface.co/AdaptLLM/finance-chat) and [Law-Chat](https://huggingface.co/AdaptLLM/law-chat).66 67### LLaMA-3-8B (💡New!)68In our recent research on [Instruction-Pretrain](https://huggingface.co/papers/2406.14491), we developed a context-based instruction synthesizer to augment the raw corpora with instruction-response pairs, **enabling Llama3-8B to be comparable to or even outperform Llama3-70B**: [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B), [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B).69 70## 2. Domain-Specific Tasks71 72### Pre-templatized Testing Splits73To easily reproduce our prompting results, we have uploaded the filled-in zero/few-shot input instructions and output completions of the test each domain-specific task: [biomedicine-tasks](https://huggingface.co/datasets/AdaptLLM/medicine-tasks), [finance-tasks](https://huggingface.co/datasets/AdaptLLM/finance-tasks), and [law-tasks](https://huggingface.co/datasets/AdaptLLM/law-tasks).74 75Note: those filled-in instructions are specifically tailored for models before alignment and do NOT fit for the specific data format required for chat models.76 77### Evaluating Any Huggingface LMs on Domain-Specific Tasks (💡New!)78You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).79 801). **Set Up Dependencies**81   ```bash82   git clone https://github.com/microsoft/LMOps83   cd LMOps/adaptllm84   pip install -r requirements.txt85   ```86 872). **Evaluate the Model**88   ```bash89   # Select the domain from ['biomedicine', 'finance', 'law']90   DOMAIN='finance'91  92   # Specify any Huggingface model name (Not applicable to chat models)93   MODEL='instruction-pretrain/finance-Llama3-8B'94  95   # Model parallelization:96   # - Set MODEL_PARALLEL=False if the model fits on a single GPU. 97   #   We observe that LMs smaller than 10B always meet this requirement.98   # - Set MODEL_PARALLEL=True if the model is too large and encounters OOM on a single GPU.99   MODEL_PARALLEL=False100  101   # Choose the number of GPUs from [1, 2, 4, 8]102   N_GPU=1103  104   # Whether to add a BOS token at the beginning of the prompt input:105   # - Set to False for AdaptLLM.106   # - Set to True for instruction-pretrain models.107   # If unsure, we recommend setting it to False, as this is suitable for most LMs.108   add_bos_token=True109 110   # Run the evaluation script111   bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}112   ```113 114### Raw Datasets115We have also uploaded the raw training and testing splits, for facilitating fine-tuning or other usages: [ChemProt](https://huggingface.co/datasets/AdaptLLM/ChemProt), [RCT](https://huggingface.co/datasets/AdaptLLM/RCT), [ConvFinQA](https://huggingface.co/datasets/AdaptLLM/ConvFinQA), [FiQA_SA](https://huggingface.co/datasets/AdaptLLM/FiQA_SA), [Headline](https://huggingface.co/datasets/AdaptLLM/Headline), [NER](https://huggingface.co/datasets/AdaptLLM/NER), [FPB](https://huggingface.co/datasets/AdaptLLM/FPB)116 117### Domain Knowledge Probing118Our pre-processed knowledge probing datasets are available at: [med_knowledge_prob](https://huggingface.co/datasets/AdaptLLM/med_knowledge_prob) and [law_knowledge_prob](https://huggingface.co/datasets/AdaptLLM/law_knowledge_prob)119 120## Citation121If you find our work helpful, please cite us:122```bibtex123@inproceedings{124cheng2024adapting,125title={Adapting Large Language Models via Reading Comprehension},126author={Daixuan Cheng and Shaohan Huang and Furu Wei},127booktitle={The Twelfth International Conference on Learning Representations},128year={2024},129url={https://openreview.net/forum?id=y886UXPEZ0}130}131```