raidium/MQG
625
1---2library_name: transformers3license: apache-2.04datasets:5- raidium/ECNQA_generated_questions6- raidium/ECN-QA7language:8- en9metrics:10- accuracy11tags:12- medical13base_model: stanford-crfm/BioMedLM14---15 16 17# Model Card for Raidium MQG model18 19 20The model is introduced in the paper "Efficient Medical Question Answering with Knowledge-Augmented Question Generation".21 22Paper: [https://arxiv.org/abs/2405.14654](https://arxiv.org/abs/2405.14654)23 24MQG is is a transformer language model pre-trained on a series of medical textbooks, and medical questions generated by GPT-4. The weights are initialized with 25[BioMedLM](https://huggingface.co/stanford-crfm/BioMedLM), then further pre-trained on those datasets.26 27The questions have been generated from prompt containing medical data from the textbooks. 28They are available here: [ECNQA_generated_questions](https://huggingface.co/datasets/raidium/ECNQA_generated_questions).29 30MQG is designed to be fine-tuned for Medical Question Answering tasks.31 32## Model Details33 34### Model Description35 3637 38In the expanding field of language model applications, medical knowledge representation remains a significant challenge due to the specialized nature of the domain. 39Large language models, such as GPT-4, obtain reasonable scores on medical question answering tasks, but smaller models are far behind. 40In this work, we introduce a method to improve the proficiency of a small language model in the medical domain by employing a two-fold approach. 41We first fine-tune the model on a corpus of medical textbooks. Then, we use GPT-4 to generate questions similar to the downstream task, prompted with textbook knowledge, and use them to fine-tune the model. 42We show the benefits of our training strategy on a medical answering question dataset.43 44 45### Using the model46 47```python48from transformers import AutoTokenizer, AutoModelForCausalLM49 50tokenizer = AutoTokenizer.from_pretrained("raidium/MQG")51model = AutoModelForCausalLM.from_pretrained("raidium/MQG") 52```53 54 55- **Developed by:** Raidium56- **Model type:** Transformer57- **License:** Aopache 2.058- **Finetuned from model:** [BioMedLM](https://huggingface.co/stanford-crfm/BioMedLM)59 60### Model Sources [optional]61 62<!-- Provide the basic links for the model. -->63 64- **Repository:** [https://github.com/raidium-med/MQG]65- **Paper:** [https://arxiv.org/abs/2405.14654](https://arxiv.org/abs/2405.14654)66 67## Uses68 69### Direct Use70 71MQG is trained using next-token-prediction on generated questions. 72Therefore, it can be used out-of-the-box to generate potential answers for medical question answering tasks.73However, the generated questions might contain some errors, so it is advised to fine-tune the model on your dataset, and use the models to rank the potential answers.74 75### Downstream Use76 77MQG can be fine-tuned for Medical Question Answering tasks. 78For multiple choice questions, a classification head should be appended at the end of the model, to rank different proposed answers.79 80### Out-of-Scope Use81 82This model should not be used for datasets outside medical tasks.83 84## Bias, Risks, and Limitations85 86There is no guarantee that the model answers medical questions correctly. It should only be used for academic purposes, and not in clinical care.87 88## Training Details89 90### Training Data91 92The model is trained on a corpus of medical textbooks, and further pre-trained on generated questions: [ECNQA_generated_questions](https://huggingface.co/datasets/raidium/ECNQA_generated_questions).93 94### Training Procedure95 96MGQ is trained using next-token-prediction on both datasets.97 98#### Training Hyperparameters99 100- **Training regime:** fp16 mixed-precision training. <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->101 102## Evaluation103 104### Testing Data, Factors & Metrics105 106#### Testing Data107 108We tested the model on a medical question answering dataset, ECN-QA, based on the french medical residency examination. 109It is composed of "single" and "progressive" questions (i.e a serie of multiple related questions).110It is a multiple-choice question dataset, containing 5 propositions for each question.111 112#### Metrics113 114We use the accuracy to evaluate the model on Medical Question Answering.115 116### Results117 118See paper: [https://arxiv.org/abs/2405.14654](https://arxiv.org/abs/2405.14654)119 120### Model Architecture and Objective121 122The model is based on BioMedLM's architecture, which is modified from GPT-2 architecture.123 124### Compute Infrastructure125 126#### Hardware127 128The model was trained on the Jean-Zay supercomputer, on multiple nodes with 4 A100 gpus. 129 130#### Software131 132Pytorch, DeepSpeed133 134## Citation135 136 137**BibTeX:**138```139@article{khlaut2024efficient,140 title={Efficient Medical Question Answering with Knowledge-Augmented Question Generation},141 author={Khlaut, Julien and Dancette, Corentin and Ferreres, Elodie and Bennani, Alaedine and H{\'e}rent, Paul and Manceron, Pierre},142 journal={Clinical NLP Workshop, NAACL 2024},143 year={2024}144}145```146 147## Model Card Contact148 149julien.khlaut at raidium.fr