FunAudioLLM/Fun-Audio-Chat-8B

1---2language:3- en4- zh5license: apache-2.06library_name: transformers7tags:8- audio-language-model9- speech-to-speech10pipeline_tag: any-to-any11---12 13# Fun-Audio-Chat-8B14 15<p align="right">16  <a href="Fun-Audio-Chat-8B/blob/main/README.md">English</a> | <a href="Fun-Audio-Chat-8B/blob/main/README_zh.md">中文</a>17</p>18 19<div align="center">20 21<img src="https://github.com/FunAudioLLM/Fun-Audio-Chat/blob/main/assets/TONGYI Fun.png?raw=true" alt="TONGYI Fun" height="80">22 23**Fun-Audio-Chat** is a Large Audio Language Model built for natural, low-latency voice interactions.24 25[![arXiv](https://img.shields.io/badge/arXiv-2512.20156-red)](https://arxiv.org/pdf/2512.20156)26[![GitHub](https://img.shields.io/badge/GitHub-Code-blue)](https://github.com/FunAudioLLM/Fun-Audio-Chat)27[![Demo](https://img.shields.io/badge/Demo-Page-green)](https://funaudiollm.github.io/funaudiochat)28 29</div>30 31## Model Description32 33Fun-Audio-Chat is a Large Audio Language Model built for natural, low-latency voice interactions. It introduces **Dual-Resolution Speech Representations** (an efficient 5Hz shared backbone + a 25Hz refined head) to cut compute while keeping high speech quality, and **Core-Cocktail training** to preserve strong text LLM capabilities. It delivers top-tier results on spoken QA, audio understanding, speech function calling, speech instruction-following and voice empathy benchmarks.34 35<p align="center">36  <img width="95%" src="https://github.com/FunAudioLLM/Fun-Audio-Chat/blob/main/assets/Results.png?raw=true">37</p>38 39### Key Features40 41- **Dual-Resolution Speech Representations**: Efficient 5Hz frame rate (vs. 12.5Hz or 25Hz for other models), reducing GPU hours by nearly 50% while maintaining high speech quality42- **State-of-the-Art Performance**: Ranks Top among models of the same size (around-8B parameters) on OpenAudioBench, VoiceBench, UltraEval-Audio, MMAU, MMAU-Pro, MMSU, Speech-ACEBench, Speech-BFCL, Speech-SmartInteract, VStyle43- **Comprehensive Capabilities**: Supports spoken QA, audio understanding, speech function calling, speech instruction-following, voice empathy44 45<p align="center">46  <img width="95%" src="https://github.com/FunAudioLLM/Fun-Audio-Chat/blob/main/assets/Architecture.png?raw=true">47</p>48 49## Model Details50 51| Attribute | Value |52|-----------|-------|53| Model Size | ~8B parameters |54| Architecture | Dual-Resolution Speech Representations |55| Languages | English, Chinese |56| License | Apache 2.0 |57 58## Requirements59 60- Python == 3.1261- PyTorch == 2.8.062- ffmpeg63- GPU Memory: ~24GB for inference, 4×80GB for training64 65## Installation66 67```bash68git clone --recurse-submodules https://github.com/FunAudioLLM/Fun-Audio-Chat69cd Fun-Audio-Chat70 71apt install ffmpeg72conda create -n FunAudioChat python=3.12 -y73conda activate FunAudioChat74pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu12875pip install -r requirements.txt76```77 78## Quick Start79 80### Download Models81 82**Using HuggingFace:**83```bash84pip install huggingface-hub85hf download FunAudioLLM/Fun-Audio-Chat-8B --local-dir ./pretrained_models/Fun-Audio-Chat-8B86hf download FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --local-dir ./pretrained_models/Fun-CosyVoice3-0.5B-251287```88 89**Or using ModelScope:**90```bash91modelscope download --model FunAudioLLM/Fun-Audio-Chat-8B --local_dir pretrained_models/Fun-Audio-Chat-8B92modelscope download --model FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --local_dir pretrained_models/Fun-CosyVoice3-0.5B-251293```94 95### Inference96 97```bash98export PYTHONPATH=`pwd`99# Speech-to-Text100python examples/infer_s2t.py101# Speech-to-Speech102python examples/infer_s2s.py103```104 105## Evaluation106 107| Benchmark | Category |108|-----------|----------|109| OpenAudioBench | Spoken QA |110| VoiceBench | Spoken QA |111| UltraEval-Audio | Speech-to-Speech |112| MMAU, MMAU-Pro, MMSU | Audio Understanding |113| Speech-ACEBench, Speech-BFCL, Speech-SmartInteract | Speech Function Calling |114| VStyle | Speech Instruction-Following |115 116For detailed evaluation instructions, please refer to the [GitHub repository](https://github.com/FunAudioLLM/Fun-Audio-Chat).117 118## Citation119 120If you find this model useful, please cite our paper:121 122```bibtex123@article{funaudiochat2025,124  title={Fun-Audio-Chat Technical Report},125  author={Qian Chen and Luyao Cheng and Chong Deng and Xiangang Li and Jiaqing Liu and Chao-Hong Tan and Wen Wang and Junhao Xu and Jieping Ye and Qinglin Zhang and Qiquan Zhang and Jingren Zhou},126  year={2025},127  eprint={2512.20156},128  archivePrefix={arXiv},129  primaryClass={cs.CL},130  url={https://arxiv.org/abs/2512.20156},131}132 133 134@misc{tan2025drvoiceparallelspeechtextvoice,135  title={DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations}, 136  author={Chao-Hong Tan and Qian Chen and Wen Wang and Chong Deng and Qinglin Zhang and Luyao Cheng and Hai Yu and Xin Zhang and Xiang Lv and Tianyu Zhao and Chong Zhang and Yukun Ma and Yafeng Chen and Hui Wang and Jiaqing Liu and Xiangang Li and Jieping Ye},137  year={2025},138  eprint={2506.09349},139  archivePrefix={arXiv},140  primaryClass={cs.CL},141  url={https://arxiv.org/abs/2506.09349}, 142}143```144 145## License146 147This model is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).148 149## Acknowledgments150 151This project is based on the following excellent open-source projects:152 153- [Transformers](https://github.com/huggingface/transformers)154- [LlamaFactory](https://github.com/hiyouga/LLaMA-Factory)155- [Moshi](https://github.com/kyutai-labs/moshi)156- [CosyVoice](https://github.com/FunAudioLLM/CosyVoice)157 158## Contact159 160- 🐛 Submit an [Issue](https://github.com/FunAudioLLM/Fun-Audio-Chat/issues)161- 💡 Submit a [Pull Request](https://github.com/FunAudioLLM/Fun-Audio-Chat/pulls)