FireRedTeam/FireRedASR

1<div align="center">2<h1>FireRedASR: Open-Source Industrial-Grade3<br>4Automatic Speech Recognition Models</h1>5 6</div>7 8[[Paper]](https://arxiv.org/pdf/2501.14350)9[[Model]](https://huggingface.co/fireredteam)10[[Blog]](https://fireredteam.github.io/demos/firered_asr/)11 12FireRedASR is a family of open-source industrial-grade automatic speech recognition (ASR) models supporting Mandarin, Chinese dialects and English, achieving a new state-of-the-art (SOTA) on public Mandarin ASR benchmarks, while also offering outstanding singing lyrics recognition capability.13 14 15## 🔥 News16- [2025/02/17] We release [FireRedASR-LLM-L](https://huggingface.co/fireredteam/FireRedASR-LLM-L/tree/main) model weights.17- [2025/01/24] We release [technical report](https://arxiv.org/pdf/2501.14350), [blog](https://fireredteam.github.io/demos/firered_asr/), and [FireRedASR-AED-L](https://huggingface.co/fireredteam/FireRedASR-AED-L/tree/main) model weights.18 19 20## Method21 22FireRedASR is designed to meet diverse requirements in superior performance and optimal efficiency across various applications. It comprises two variants:23- FireRedASR-LLM: Designed to achieve state-of-the-art (SOTA) performance and to enable seamless end-to-end speech interaction. It adopts an Encoder-Adapter-LLM framework leveraging large language model (LLM) capabilities.24- FireRedASR-AED: Designed to balance high performance and computational efficiency and to serve as an effective speech representation module in LLM-based speech models. It utilizes an Attention-based Encoder-Decoder (AED) architecture.25 26![Model](/assets/FireRedASR_model.png)27 28 29## Evaluation30Results are reported in Character Error Rate (CER%) for Chinese and Word Error Rate (WER%) for English.31 32### Evaluation on Public Mandarin ASR Benchmarks33| Model            | #Params | aishell1 | aishell2 | ws\_net  | ws\_meeting | Average-4 |34|:----------------:|:-------:|:--------:|:--------:|:--------:|:-----------:|:---------:|35| FireRedASR-LLM   | 8.3B | 0.76 | 2.15 | 4.60 | 4.67 | 3.05 |36| FireRedASR-AED   | 1.1B | 0.55 | 2.52 | 4.88 | 4.76 | 3.18 |37| Seed-ASR         | 12B+ | 0.68 | 2.27 | 4.66 | 5.69 | 3.33 |38| Qwen-Audio       | 8.4B | 1.30 | 3.10 | 9.50 | 10.87 | 6.19 |39| SenseVoice-L     | 1.6B | 2.09 | 3.04 | 6.01 | 6.73 | 4.47 |40| Whisper-Large-v3 | 1.6B | 5.14 | 4.96 | 10.48 | 18.87 | 9.86 |41| Paraformer-Large | 0.2B | 1.68 | 2.85 | 6.74 | 6.97 | 4.56 |42 43`ws` means WenetSpeech.44 45### Evaluation on Public Chinese Dialect and English ASR Benchmarks46|Test Set       | KeSpeech | LibriSpeech test-clean | LibriSpeech test-other  |47| :------------:| :------: | :--------------------: | :----------------------:|48|FireRedASR-LLM | 3.56 | 1.73 | 3.67 |49|FireRedASR-AED | 4.48 | 1.93 | 4.44 |50|Previous SOTA Results | 6.70 | 1.82 | 3.50 |51 52 53## Usage54Download model files from [huggingface](https://huggingface.co/fireredteam) and place them in the folder `pretrained_models`.55 56If you want to use `FireRedASR-LLM-L`, you also need to download [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct) and place it in the folder `pretrained_models`. Then, go to folder `FireRedASR-LLM-L` and run `$ ln -s ../Qwen2-7B-Instruct`57 58 59### Setup60Create a Python environment and install dependencies61```bash62$ git clone https://github.com/FireRedTeam/FireRedASR.git63$ conda create --name fireredasr python=3.1064$ pip install -r requirements.txt65```66 67Set up Linux PATH and PYTHONPATH68```69$ export PATH=$PWD/fireredasr/:$PWD/fireredasr/utils/:$PATH70$ export PYTHONPATH=$PWD/:$PYTHONPATH71```72 73Convert audio to 16kHz 16-bit PCM format74```75ffmpeg -i input_audio -ar 16000 -ac 1 -acodec pcm_s16le -f wav output.wav76```77 78### Quick Start79```bash80$ cd examples81$ bash inference_fireredasr_aed.sh82$ bash inference_fireredasr_llm.sh83```84 85### Command-line Usage86```bash87$ speech2text.py --help88$ speech2text.py --wav_path examples/wav/BAC009S0764W0121.wav --asr_type "aed" --model_dir pretrained_models/FireRedASR-AED-L89$ speech2text.py --wav_path examples/wav/BAC009S0764W0121.wav --asr_type "llm" --model_dir pretrained_models/FireRedASR-LLM-L90```91 92### Python Usage93```python94from fireredasr.models.fireredasr import FireRedAsr95 96batch_uttid = ["BAC009S0764W0121"]97batch_wav_path = ["examples/wav/BAC009S0764W0121.wav"]98 99# FireRedASR-AED100model = FireRedAsr.from_pretrained("aed", "pretrained_models/FireRedASR-AED-L")101results = model.transcribe(102    batch_uttid,103    batch_wav_path,104    {105        "use_gpu": 1,106        "beam_size": 3,107        "nbest": 1,108        "decode_max_len": 0,109        "softmax_smoothing": 1.25,110        "aed_length_penalty": 0.6,111        "eos_penalty": 1.0112    }113)114print(results)115 116 117# FireRedASR-LLM118model = FireRedAsr.from_pretrained("llm", "pretrained_models/FireRedASR-LLM-L")119results = model.transcribe(120    batch_uttid,121    batch_wav_path,122    {123        "use_gpu": 1,124        "beam_size": 3,125        "decode_max_len": 0,126        "decode_min_len": 0,127        "repetition_penalty": 3.0,128        "llm_length_penalty": 1.0,129        "temperature": 1.0130    }131)132print(results)133```134 135## Usage Tips136### Batch Beam Search137- When performing batch beam search with FireRedASR-LLM, please ensure that the input lengths of the utterances are similar. If there are significant differences in utterance lengths, shorter utterances may experience repetition issues. You can either sort your dataset by length or set `batch_size` to 1 to avoid the repetition issue.138 139### Input Length Limitations140- FireRedASR-AED supports audio input up to 60s. Input longer than 60s may cause hallucination issues, and input exceeding 200s will trigger positional encoding errors.141- FireRedASR-LLM supports audio input up to 30s. The behavior for longer input is currently unknown.142 143 144## Acknowledgements145Thanks to the following open-source works:146- [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct)147- [icefall/ASR_LLM](https://github.com/k2-fsa/icefall/tree/master/egs/speech_llm/ASR_LLM)148- [WeNet](https://github.com/wenet-e2e/wenet)149- [Speech-Transformer](https://github.com/kaituoxu/Speech-Transformer)150 151 152## Citation153```bibtex154@article{xu2025fireredasr,155  title={FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration},156  author={Xu, Kai-Tuo and Xie, Feng-Long and Tang, Xu and Hu, Yao},157  journal={arXiv preprint arXiv:2501.14350},158  year={2025}159}160```161