FuseAI/OpenChat-3.5-7B-Mixtral
1040
1---2license: apache-2.03language:4- en5base_model: openchat/openchat_3.56datasets:7- FuseAI/FuseChat-Mixture8pipeline_tag: text-generation9tags:10- mistral11- mixtral12- solar13- model-fusion14- fusechat15library_name: transformers16model-index:17- name: OpenChat-3.5-7B-Mixtral18 results:19 - task:20 type: text-generation21 name: Text Generation22 dataset:23 name: MT-Bench24 type: unknown25 metrics:26 - type: unknown27 value: 8.0828 name: score29 source:30 url: https://huggingface.co/spaces/lmsys/mt-bench31 - task:32 type: text-generation33 name: Text Generation34 dataset:35 name: AI2 Reasoning Challenge (25-Shot)36 type: ai2_arc37 config: ARC-Challenge38 split: test39 args:40 num_few_shot: 2541 metrics:42 - type: acc_norm43 value: 62.844 name: normalized accuracy45 source:46 url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/OpenChat-3.5-7B-Mixtral47 name: Open LLM Leaderboard48 - task:49 type: text-generation50 name: Text Generation51 dataset:52 name: HellaSwag (10-Shot)53 type: hellaswag54 split: validation55 args:56 num_few_shot: 1057 metrics:58 - type: acc_norm59 value: 84.2460 name: normalized accuracy61 source:62 url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/OpenChat-3.5-7B-Mixtral63 name: Open LLM Leaderboard64 - task:65 type: text-generation66 name: Text Generation67 dataset:68 name: MMLU (5-Shot)69 type: cais/mmlu70 config: all71 split: test72 args:73 num_few_shot: 574 metrics:75 - type: acc76 value: 63.9577 name: accuracy78 source:79 url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/OpenChat-3.5-7B-Mixtral80 name: Open LLM Leaderboard81 - task:82 type: text-generation83 name: Text Generation84 dataset:85 name: TruthfulQA (0-shot)86 type: truthful_qa87 config: multiple_choice88 split: validation89 args:90 num_few_shot: 091 metrics:92 - type: mc293 value: 45.6894 source:95 url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/OpenChat-3.5-7B-Mixtral96 name: Open LLM Leaderboard97 - task:98 type: text-generation99 name: Text Generation100 dataset:101 name: Winogrande (5-shot)102 type: winogrande103 config: winogrande_xl104 split: validation105 args:106 num_few_shot: 5107 metrics:108 - type: acc109 value: 79.64110 name: accuracy111 source:112 url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/OpenChat-3.5-7B-Mixtral113 name: Open LLM Leaderboard114 - task:115 type: text-generation116 name: Text Generation117 dataset:118 name: GSM8k (5-shot)119 type: gsm8k120 config: main121 split: test122 args:123 num_few_shot: 5124 metrics:125 - type: acc126 value: 62.09127 name: accuracy128 source:129 url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/OpenChat-3.5-7B-Mixtral130 name: Open LLM Leaderboard131---132<p align="center" width="100%">133</p>134 135<div id="top" align="center">136 137<p style="font-size: 30px; font-weight: bold;">FuseChat: Knowledge Fusion of Chat Models</p>138 139<p style="font-size: 24px; font-weight: bold;">[SOTA 7B LLM on MT-Bench]</p>140 141<h4> |<a href="https://arxiv.org/abs/2402.16107"> 📑 Paper </a> |142<a href="https://huggingface.co/FuseAI"> 🤗 HuggingFace Repo </a> |143<a href="https://github.com/fanqiwan/FuseLLM"> 🐱 GitHub Repo </a> |144</h4>145 146<!-- **Authors:** -->147 148_**Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, Wei Bi**_149 150 151<!-- **Affiliations:** -->152 153 154_Sun Yat-sen University_155 156<p align="center">157 <img src="./assets/fig_0.png" width="70%"> <br>158</p>159 160| Proprietary Models | #Params | MT-Bench | Open Source Models | #Params | MT-Bench |161|-----------------------------------------------------------------------|---------|----------|-----------------------------------------------------------------------|---------|----------|162| GPT-4-1106-preview | - | 9.32 | Qwen1.5-72B-Chat | 72B | 8.61 |163| GPT-4-0613 | - | 9.18 | Nous-Hermes-2-Mixtral-8x7B-DPO | 8x7B | 8.33 |164| GPT-4-0314 | - | 8.96 | Mixtral-8x7B-Instruct-v0.1 | 8x7B | 8.30 |165| Mistral Medium | - | 8.61 | 🤗 [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM) | 7B | 8.22 |166| GPT-3.5-Turbo-0613 | - | 8.39 | Starling-LM-7B-alpha | 7B | 8.09 |167| GPT-3.5-Turbo-1106 | - | 8.32 | Tulu-2-DPO-70B | 70B | 7.89 |168| 🤗 [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM) | 7B | 8.22 | OpenChat-3.5 | 7B | 7.81 |169| Claude-2.1 | - | 8.18 | OpenChat-3.5-0106 | 7B | 7.80 |170| Claude-2.0 | - | 8.06 | WizardLM-70B-v1.0 | 70B | 7.71 |171| GPT-3.5-Turbo-0314 | - | 7.94 | Yi-34B-Chat | 34B | 7.67 |172| Claude-1 | - | 7.90 | Nous-Hermes-2-SOLAR-10.7B | 10.7B | 7.66 |173 174 175</div>176 177 178## News179- **Feb 26, 2024:** 🔥🔥 We release [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM), which is the fusion of three prominent chat LLMs with diverse architectures and scales, namely [NH2-Mixtral-8x7B](https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO), [NH2-Solar-10.7B](https://huggingface.co/NousResearch/Nous-Hermes-2-SOLAR-10.7B), and [OpenChat-3.5-7B](https://huggingface.co/openchat/openchat_3.5). FuseChat-7B-VaRM achieves an average performance of **8.22** on MT-Bench, outperforming various powerful chat LLMs at 7B and 34B scales like [Starling-7B](https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha) and [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat), even surpassing [GPT-3.5 (March)](https://platform.openai.com/docs/models/gpt-3-5-turbo), [Claude-2.1](https://www.anthropic.com/news/claude-2-1), and approaching [Mixtral-8x7B-Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1). 180 181- **Feb 25, 2024:** 🔥 We release [FuseChat-Mixture](https://huggingface.co/datasets/FuseAI/FuseChat-Mixture), which is a comprehensive training dataset covers different styles and capabilities, featuring both human-written and model-generated, and spanning general instruction-following and specific skills.182 183## Contents184 185- [Overview](#overview)186- [Model Release](#model-release)187- [Quick Start](#quick-start)188- [Data Construction](#data-construction)189- [Pairwise Knowledge Fusion](#pairwise-knowledge-fusion)190- [Model Merging](#model-merging)191- [Evaluation](#evaluation)192- [Citation](#citation)193 194## Overview195 196In this work, we propose an extended framework of FuseLLM to integrate the collective knowledge and individual strengths of multiple structure and scale-varied chat LLMs into a more powerful chat LLM, resulting in FuseChat. FuseChat adopts a fuse-then-merge strategy with two main stages. Firstly, it undertakes pairwise knowledge fusion for source LLMs to derive multiple target LLMs of identical structure and size via lightweight fine-tuning. Then, these target LLMs are merged within the parameter space, wherein we propose a novel method VaRM for determining the merging weights based on the variation ratio of parameter matrices before and after fine-tuning. 197 198 199Moreover, we argue that the concept of knowledge fusion adopted by both FuseChat and FuseLLM shares a fundamentally similar purpose with other related topics, such as the recently popular topic of mixture of experts (MoEs), because they all aim to leverage the strengths of multiple models (experts). However, while MoEs require loading multiple experts during inference, which has higher memory requirements, knowledge fusion supports the integration of multiple LLMs with diverse architectures into a single LLM without any additional memory requirement, making it more memory-efficient. 200 201<p align="center">202 <img src="./assets/fig_1.png" width="95%"> <br>203</p>204 205 206## Model Release207 208We release [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM), which is the fusion of three prominent chat LLMs with diverse architectures and scales, namely [NH2-Mixtral-8x7B](https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO), [NH2-Solar-10.7B](https://huggingface.co/NousResearch/Nous-Hermes-2-SOLAR-10.7B), and [OpenChat-3.5-7B](https://huggingface.co/openchat/openchat_3.5). FuseChat-7B-VaRM achieves an average performance of **8.22** on MT-Bench, outperforming various powerful chat LLMs at 7B and 34B scales like [Starling-7B](https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha) and [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat), even surpassing [GPT-3.5 (March)](https://platform.openai.com/docs/models/gpt-3-5-turbo), [Claude-2.1](https://www.anthropic.com/news/claude-2-1), and approaching [Mixtral-8x7B-Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1).209 210To support a plug-and-play fusion of new source LLM, we release our target LLMs: [OpenChat-3.5-7B-Solar](https://huggingface.co/FuseAI/OpenChat-3.5-7B-Solar) and [OpenChat-3.5-7B-Mixtral](https://huggingface.co/FuseAI/OpenChat-3.5-7B-Mixtral), which are obtained from pair-wise knowledge fusion. Integrating a new source LLM at any scale requires only obtaining a target LLM from the new source LLM and merging it with the existing target LLMs.211 212We also release FuseChat with other merging methods: [FuseChat-7B-SLERP](https://huggingface.co/FuseAI/FuseChat-7B-SLERP) and [FuseChat-7B-TA](https://huggingface.co/FuseAI/FuseChat-7B-TA), which achieves an average performance of **8.19** and **8.20** on MT-Bench respectively.213 214Here are the evaluation results.215 216<p align="center">217 <img src="./assets/tab_1.png" width="95%"> <br>218</p>219 220## Quick Start221 222### Setup223 224We use `python 3.11` in this project.225 226Then, we have to install all the libraries listed in `requirements.txt`.227 228```bash229pip install -r requirements.txt230```231 232### Usage233 234Here's how you can run the model using the 🤗 Transformers:235 236```python237import transformers238tokenizer = transformers.AutoTokenizer.from_pretrained("FuseAI/FuseChat-7B-VaRM")239# Single-turn240tokens = tokenizer("GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant:").input_ids241assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]242# Multi-turn243tokens = tokenizer("GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant: Hi<|end_of_turn|>GPT4 Correct User: How are you today?<|end_of_turn|>GPT4 Correct Assistant:").input_ids244assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]245```246 247The GPT4 template is also available as the integrated `tokenizer.chat_template`, which can be used instead of manually specifying the template:248 249```python250messages = [251 {"role": "user", "content": "Hello"},252 {"role": "assistant", "content": "Hi"},253 {"role": "user", "content": "How are you today?"}254]255tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)256assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]257```258 259## Data Construction260 261We curated a comprehensive training dataset, [FuseChat-Mixture](https://huggingface.co/datasets/FuseAI/FuseChat-Mixture), from various sources. This dataset covers different styles and capabilities, featuring both human-written and model-generated, and spanning general instruction-following and specific skills. 262 263Here we show the scripts to obtain representations from multiple source LLMs for model fusion.264 2651. Get representations for each source LLM266 267```bash268# We split the dataset into 4 splits, then process each split on one or multiple GPU.269 270# OpenChat-3.5-7B271export CUDA_VISIBLE_DEVICES=0272for i in {0..3}; do273python /train/get_data_representation.py \274 --model_name_or_path "openchat/openchat_3.5" \275 --data_path "/data/fusechat_v1_clean_split_2048_filter_wrong.json" \276 --dataset_save_dir "<${i}_4_path_to_openchat_representation>" \277 --tknz_dataset_path "<${i}_4_path_to_openchat_tknz>" \278 --cache_dir "/.cache/huggingface/datasets" \279 --model_max_length 2048 \280 --load_in_half bf16 \281 --batch_size 32 \282 --top_k_logits 10 \283 --save_per_token_metric \284 --no_assert \285 --conv_temp "openchat" \286 --flash_attn_transformers \287 --mask_instruction \288 --dataset_split_num 4 \289 --dataset_index ${i}290done 291 292# NH2-Mixtral-8x7B293export CUDA_VISIBLE_DEVICES=0,1,2294for i in {0..3}; do295python /train/get_data_representation.py \296 --model_name_or_path "NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO" \297 --data_path "/data/fusechat_v1_clean_split_2048_filter_wrong.json" \298 --dataset_save_dir "<${i}_4_path_to_mixtral_representation>" \299 --tknz_dataset_path "<${i}_4_path_to_mixtral_tknz>" \300 --cache_dir "/.cache/huggingface/datasets" \301 --model_max_length 2048 \302 --load_in_half bf16 \303 --batch_size 4 \304 --top_k_logits 10 \305 --save_per_token_metric \306 --no_assert \307 --conv_temp "openchat" \308 --flash_attn_transformers \309 --mask_instruction \310 --device_map "auto" \311 --dataset_split_num 4 \312 --dataset_index ${i}313done 314 315# NH2-Solar-10.7B316export CUDA_VISIBLE_DEVICES=0317for i in {0..3}; do318python /train/get_data_representation.py \319 --model_name_or_path "NousResearch/Nous-Hermes-2-SOLAR-10.7B" \320 --data_path "/data/fusechat_v1_clean_split_2048_filter_wrong.json" \321 --dataset_save_dir "<${i}_4_path_to_solar_representation>" \322 --tknz_dataset_path "<${i}_4_path_to_solar_tknz>" \323 --cache_dir "/.cache/huggingface/datasets" \324 --model_max_length 2048 \325 --load_in_half bf16 \326 --batch_size 8 \327 --top_k_logits 10 \328 --save_per_token_metric \329 --no_assert \330 --conv_temp "openchat" \331 --flash_attn_transformers \332 --mask_instruction \333 --dataset_split_num 4 \334 --dataset_index ${i}335done 336```337 3382. Align representations from different source LLMs339 340```bash341# Since the tokenizers and vocabularies of these source LLMs are identical, we do not align.342 343# OpenChat-3.5-7B <-> NH2-Mixtral-8x7B344for i in {0..3}; do345python /train/replace_model.py \346 --dataset_dir "<${i}_4_path_to_openchat_representation>" \347 --replace_dataset_dir "<${i}_4_path_to_mixtral_representation>" \348 --dataset_save_dir "<${i}_4_path_to_openchat_mixtral_representation>" \349 --preprocessing_num_workers 64 \350 --batch_size 1000 \351 --replace_model model_0352done 353 354# OpenChat-3.5-7B <-> NH2-Solar-10.7B355for i in {0..3}; do356python /train/replace_model.py \357 --dataset_dir "<${i}_4_path_to_openchat_mixtral_representation>" \358 --replace_dataset_dir "<${i}_4_path_to_solar_representation>" \359 --dataset_save_dir "<${i}_4_path_to_openchat_mixtral_solar_representation>" \360 --preprocessing_num_workers 64 \361 --batch_size 1000 \362 --replace_model model_1363done364```365 3663. Filter instances with NaN loss in the dataset367 368```bash369for i in {0..3}; do370python /train/filter_nan.py \371 --input_data_dir "<${i}_4_path_to_openchat_mixtral_solar_representation>" \372 --output_data_dir "<${i}_4_path_to_openchat_mixtral_solar_representation_fnan>"373done374```375 376The final processed data is at `<${i}_4_path_to_openchat_mixtral_solar_representation_fnan>`.377 378## Pairwise Knowledge Fusion379 380We show the scripts for pairwise knowledge fusion.381 382```bash383# OpenChat-3.5-7B <-> NH2-Mixtral-8x7B384export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7385torchrun --nproc_per_node=8 --master_port=20001 /train/train.py \386 --model_name_or_path "openchat/openchat_3.5" \387 --data_path "<0_4_path_to_openchat_mixtral_solar_representation_fnan>,<1_4_path_to_openchat_mixtral_solar_representation_fnan>,<2_4_path_to_openchat_mixtral_solar_representation_fnan>,<3_4_path_to_openchat_mixtral_solar_representation_fnan>" \388 --bf16 True \389 --output_dir "<path_to_save_openchat_mixtral_ckpt>" \390 --num_train_epochs 3 \391 --per_device_train_batch_size 4 \392 --per_device_eval_batch_size 4 \393 --gradient_accumulation_steps 4 \394 --evaluation_strategy "no" \395 --save_strategy "epoch" \396 --save_steps 10000 \397 --save_total_limit 5 \398 --learning_rate 5e-6 \399 --weight_decay 0. \400 --warmup_ratio 0.03 \401 --lr_scheduler_type "cosine" \402 --logging_steps 1 \403 --fsdp "full_shard auto_wrap" \404 --fsdp_transformer_layer_cls_to_wrap 'MistralDecoderLayer' \405 --tf32 True \406 --model_max_length 2048 \407 --gradient_checkpointing True \408 --conv_temp "openchat" \409 --lazy_preprocess True \410 --flash_attn_transformers True \411 --do_train \412 --do_distill \413 --distill_with_ref_model True \414 --distill_with_aligned_model_0 True \415 --distill_with_aligned_model_1 False \416 --distill_loss_type "ce" \417 --distill_teacher_temperature 1.0 \418 --lm_loss_weight 0.9 \419 --distill_greater_as_gt True \420 --distill_greater_as_gt_type hard \421 --dataloader_num_workers 8 \422 --remove_unused_columns False423 424# OpenChat-3.5-7B <-> NH2-Solar-10.7B425export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7426torchrun --nproc_per_node=8 --master_port=20001 /train/train.py \427 --model_name_or_path "openchat/openchat_3.5" \428 --data_path "<0_4_path_to_openchat_mixtral_solar_representation_fnan>,<1_4_path_to_openchat_mixtral_solar_representation_fnan>,<2_4_path_to_openchat_mixtral_solar_representation_fnan>,<3_4_path_to_openchat_mixtral_solar_representation_fnan>" \429 --bf16 True \430 --output_dir "<path_to_save_openchat_solar_ckpt>" \431 --num_train_epochs 3 \432 --per_device_train_batch_size 4 \433 --per_device_eval_batch_size 4 \434 --gradient_accumulation_steps 4 \435 --evaluation_strategy "no" \436 --save_strategy "epoch" \437 --save_steps 10000 \438 --save_total_limit 5 \439 --learning_rate 5e-6 \440 --weight_decay 0. \441 --warmup_ratio 0.03 \442 --lr_scheduler_type "cosine" \443 --logging_steps 1 \444 --fsdp "full_shard auto_wrap" \445 --fsdp_transformer_layer_cls_to_wrap 'MistralDecoderLayer' \446 --tf32 True \447 --model_max_length 2048 \448 --gradient_checkpointing True \449 --conv_temp "openchat" \450 --lazy_preprocess True \451 --flash_attn_transformers True \452 --do_train \453 --do_distill \454 --distill_with_ref_model True \455 --distill_with_aligned_model_0 False \456 --distill_with_aligned_model_1 True \457 --distill_loss_type "ce" \458 --distill_teacher_temperature 1.0 \459 --lm_loss_weight 0.9 \460 --distill_greater_as_gt True \461 --distill_greater_as_gt_type hard \462 --dataloader_num_workers 8 \463 --remove_unused_columns False464```465 466## Model Merging467 468We show the scripts to obtain the final FuseChat using different merging methods.469 470```bash471# For "slerp", "ta", "ties", and "dare" methods (Please install "mergekit")472export CUDA_VISIBLE_DEVICES=0473mergekit-yaml merge/mergekit_configs/fusechat-slerp.yml "<path_to_save_fusechat_7b_slerp>"474mergekit-yaml merge/mergekit_configs/fusechat-ta.yml "<path_to_save_fusechat_7b_ta>"475mergekit-yaml merge/mergekit_configs/fusechat-ties.yml "<path_to_save_fusechat_7b_ties>"476mergekit-yaml merge/mergekit_configs/fusechat-dare.yml "<path_to_save_fusechat_7b_dare>"477 478# For "linear" method 479python merge/VaRM/merge.py \480 --merged_model_names "FuseAI/OpenChat-3.5-7B-Mixtral,FuseAI/OpenChat-3.5-7B-Solar" \481 --merged_model_save_dir "<path_to_save_fusechat_7b_linear>" \482 --merge_method "linear" \483 --linear_weights "1,2"484 485# For our "varm" method486python merge/VaRM/analysis.py \487 --model1_path "FuseAI/OpenChat-3.5-7B-Mixtral" \488 --model2_path "FuseAI/OpenChat-3.5-7B-Solar" \489 --save_path "<path_to_save_analysis_result>/analysis.json" \490 --merge_type "square"491 492python merge/VaRM/merge.py \493 --merged_model_names "FuseAI/OpenChat-3.5-7B-Mixtral,FuseAI/OpenChat-3.5-7B-Solar" \494 --analysis_result "<path_to_save_analysis_result>/analysis.json" \495 --merged_model_save_dir "<path_to_save_fusechat_7b_varm>" \496 --merge_method "avg_param" \497 --merge_type "square"498```499 500## Evaluation501 502We evaluate FuseChat on MT-Bench, which comprises 80 multi-turn dialogues spanning writing, roleplay, reasoning, math, coding, stem, and humanities domains. Please download the [official code](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) and follow the guidelines for evaluation. We provide the scripts for our evaluation. 503 504```bash505# Step 1. Generate model answers to MT-bench questions506export CUDA_VISIBLE_DEVICES=0,1507python gen_model_answer.py \508 --model-path "FuseAI/FuseChat-7B-VaRM" \509 --model-id "openchat_3.5_fusechat_7b_varm" \510 --num-gpus-per-model 1 \511 --num-gpus-total 2512 513# Step 2. Generate GPT-4 judgments514export OPENAI_API_KEY=XXXXXX # set the OpenAI API key515python gen_judgment.py \516 --parallel 2517 518# Step 3. Show MT-bench scores519python show_result.py520```521 522## Citation523 524If you find this work is relevant with your research or applications, please feel free to cite our work!525```526@article{wan2024fusechat,527 title={FuseChat: Knowledge Fusion of Chat Models},528 author={Fanqi Wan and Ziyi Yang and Longguang Zhong and Xiaojun Quan and Xinting Huang and Wei Bi},529 journal={arXiv preprint arXiv:2402.16107},530 year={2024}531}532```533 534# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)535Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_FuseAI__OpenChat-3.5-7B-Mixtral)536 537| Metric |Value|538|---------------------------------|----:|539|Avg. |66.40|540|AI2 Reasoning Challenge (25-Shot)|62.80|541|HellaSwag (10-Shot) |84.24|542|MMLU (5-Shot) |63.95|543|TruthfulQA (0-shot) |45.68|544|Winogrande (5-shot) |79.64|545|GSM8k (5-shot) |62.09|546