CoolFace
Modelpublic

FreedomIntelligence/Apollo-MoE-7B

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face
10likes177downloads
Model Card

Democratizing Medical LLMs For Much More Languages

Covering 12 Major Languages including English, Chinese, French, Hindi, Spanish, Arabic, Russian, Japanese, Korean, German, Italian, Portuguese and 38 Minor Languages So far.

<p align="center"> πŸ“ƒ <a href="https://arxiv.org/abs/2410.10626" target="blank">Paper</a> β€’ 🌐 <a href="" target="blank">Demo</a> β€’ πŸ€— <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEDataset" target="blank">ApolloMoEDataset</a> β€’ πŸ€— <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEBench" target="blank">ApolloMoEBench</a> β€’ πŸ€— <a href="https://huggingface.co/collections/FreedomIntelligence/apollomoe-and-apollo2-670ddebe3bb1ba1aebabbf2c" target="blank">Models</a> β€’πŸŒ <a href="https://github.com/FreedomIntelligence/Apollo" target="blank">Apollo</a> β€’ 🌐 <a href="https://github.com/FreedomIntelligence/ApolloMoE" target="_blank">ApolloMoE</a> </p>

[image]

🌈 Update

  • β€”[2024.10.15] ApolloMoE repo is publishedοΌπŸŽ‰

Languages Coverage

12 Major Languages and 38 Minor Languages

<details> <summary>Click to view the Languages Coverage</summary>

[image]

</details>

Architecture

<details> <summary>Click to view the MoE routing image</summary>

[image]

</details>

Results

Dense

πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo2-0.5B" target="blank">Apollo2-0.5B</a> β€’ πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo2-1.5B" target="blank">Apollo2-1.5B</a> β€’ πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo2-2B" target="_blank">Apollo2-2B</a>

πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo2-3.8B" target="blank">Apollo2-3.8B</a> β€’ πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo2-7B" target="blank">Apollo2-7B</a> β€’ πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo2-9B" target="_blank">Apollo2-9B</a>

<details> <summary>Click to view the Dense Models Results</summary>

[image]

</details>

Post-MoE

πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo-MoE-0.5B" target="blank">Apollo-MoE-0.5B</a> β€’ πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo-MoE-1.5B" target="blank">Apollo-MoE-1.5B</a> β€’ πŸ€— <a href="https://huggingface.co/FreedomIntelligence/Apollo-MoE-7B" target="_blank">Apollo-MoE-7B</a>

<details> <summary>Click to view the Post-MoE Models Results</summary>

[image]

</details>

Usage Format

Apollo2
  • β€”0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>
  • β€”2B, 9B: User:{query}\nAssistant:{response}\<eos\>
  • β€”3.8B: <|user|>\n{query}<|end|><|assisitant|>\n{response}<|end|>
Apollo-MoE
  • β€”0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>

Dataset & Evaluation

  • β€”Dataset πŸ€— <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEDataset" target="_blank">ApolloMoEDataset</a>

<details><summary>Click to expand</summary>

[image]

</details>

  • β€”Evaluation πŸ€— <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEBench" target="_blank">ApolloMoEBench</a>

<details><summary>Click to expand</summary>

  • β€”EN:
  • β€”MedQA-USMLE
  • β€”MedMCQA
  • β€”PubMedQA: Because the results fluctuated too much, they were not used in the paper.
  • β€”MMLU-Medical
  • β€”Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • β€”ZH:
  • β€”MedQA-MCMLE
  • β€”CMB-single: Not used in the paper
  • β€”Randomly sample 2,000 multiple-choice questions with single answer.
  • β€”CMMLU-Medical
  • β€”Anatomy, Clinicalknowledge, Collegemedicine, Genetics, Nutrition, Traditionalchinesemedicine, Virology
  • β€”CExam: Not used in the paper
  • β€”Randomly sample 2,000 multiple-choice questions
  • β€”ES: Head_qa
  • β€”FR:
  • β€”Frenchmedmcqa
  • β€”[MMLU_FR]
  • β€”Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • β€”HI: MMLU_HI
  • β€”Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • β€”AR: MMLU_AR
  • β€”Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • β€”JA: IgakuQA
  • β€”KO: KorMedMCQA
  • β€”IT:
  • β€”MedExpQA
  • β€”[MMLU_IT]
  • β€”Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • β€”DE: BioInstructQA: German part
  • β€”PT: BioInstructQA: Portuguese part
  • β€”RU: RuMedBench

</details>

Model Download and Inference

We take Apollo-MoE-0.5B as an example

  1. 1.Login Huggingface
       huggingface-cli login --token $HUGGINGFACE_TOKEN
  1. 1.Download model to local dir
python
       from huggingface_hub import snapshot_download
       import os

       local_model_dir=os.path.join('/path/to/models/dir','Apollo-MoE-0.5B')
       snapshot_download(repo_id="FreedomIntelligence/Apollo-MoE-0.5B", local_dir=local_model_dir)
  1. 1.Inference Example
python
      from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
      import os
      
      local_model_dir=os.path.join('/path/to/models/dir','Apollo-MoE-0.5B')
      
      model=AutoModelForCausalLM.from_pretrained(local_model_dir,trust_remote_code=True)
      tokenizer = AutoTokenizer.from_pretrained(local_model_dir,trust_remote_code=True)
      generation_config = GenerationConfig.from_pretrained(local_model_dir, pad_token_id=tokenizer.pad_token_id, num_return_sequences=1, max_new_tokens=7, min_new_tokens=2, do_sample=False, temperature=1.0, top_k=50, top_p=1.0)
      
      inputs = tokenizer('Answer direclty.\nThe capital of Mongolia is Ulaanbaatar.\nThe capital of Iceland is Reykjavik.\nThe capital of Australia is', return_tensors='pt')
      inputs = inputs.to(model.device)
      pred = model.generate(**inputs,generation_config=generation_config)
      print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Results reproduction

<details><summary>Click to expand</summary>

We take Apollo2-7B or Apollo-MoE-0.5B as example

  1. 1.Download Dataset for project:
      bash 0.download_data.shΒ 
  1. 1.Prepare test and dev data for specific model:
  • β€”Create test data for with special token
       bash 1.data_process_test&dev.sh
  1. 1.Prepare train data for specific model (Create tokenized data in advance):
  • β€”You can adjust data Training order and Training Epoch in this step
       bash 2.data_process_train.sh
  1. 1.Train the model
  • β€”If you want to train in Multi Nodes please refer to ./src/sft/trainingconfig/zeromulti.yaml
       bash 3.single_node_train.sh
  1. 1.Evaluate your model: Generate score for benchmark
         bash 4.eval.sh

</details>

Citation

Please use the following citation if you intend to use our dataset for training or evaluation:

@misc{zheng2024efficientlydemocratizingmedicalllms,
      title={Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts}, 
      author={Guorui Zheng and Xidong Wang and Juhao Liang and Nuo Chen and Yuping Zheng and Benyou Wang},
      year={2024},
      eprint={2410.10626},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.10626}, 
}