0120Kiya/icefall-aishell-experiment-full

<div align="center"> <img src="https://raw.githubusercontent.com/k2-fsa/icefall/master/docs/source/_static/logo.png" width=168> </div>

Introduction

The icefall project contains speech-related recipes for various datasets using k2-fsa and lhotse.

You can use sherpa, sherpa-ncnn or sherpa-onnx for deployment with models in icefall; these frameworks also support models not included in icefall; please refer to respective documents for more details.

You can try pre-trained models from within your browser without the need to download or install anything by visiting this huggingface space. Please refer to document for more details.

Installation

Please refer to document for installation.

Recipes

Please refer to document for more details.

ASR: Automatic Speech Recognition

Supported Datasets

• —[yesno][yesno]

• —[Aidatatang200zh][aidatatang200zh]
• —[Aishell][aishell]
• —[Aishell2][aishell2]
• —[Aishell4][aishell4]
• —[Alimeeting][alimeeting]
• —[AMI][ami]
• —[CommonVoice][commonvoice]
• —[Corpus of Spontaneous Japanese][csj]
• —[GigaSpeech][gigaspeech]
• —[LibriCSS][libricss]
• —[LibriSpeech][librispeech]
• —[Libriheavy][libriheavy]
• —[Multi-Dialect Broadcast News Arabic Speech Recognition][mgb2]
• —[SPGISpeech][spgispeech]
• —[Switchboard][swbd]
• —[TIMIT][timit]
• —[TED-LIUM3][tedlium3]
• —[TALCSASR][talcsasr]
• —[Voxpopuli][voxpopuli]
• —[XBMU-AMDO31][xbmu-amdo31]
• —[WenetSpeech][wenetspeech]

More datasets will be added in the future.

Supported Models

The [LibriSpeech][librispeech] recipe supports the most comprehensive set of models, you are welcome to try them out.

CTC

• —TDNN LSTM CTC
• —Conformer CTC
• —Zipformer CTC

MMI

• —Conformer MMI
• —Zipformer MMI

Transducer

• —Conformer-based Encoder
• —LSTM-based Encoder
• —Zipformer-based Encoder
• —LSTM-based Predictor
• —Stateless Predictor

Whisper

• —OpenAi Whisper (We support fine-tuning on AiShell-1.)

If you are willing to contribute to icefall, please refer to contributing for more details.

We would like to highlight the performance of some of the recipes here.

[yesno][yesno]

This is the simplest ASR recipe in icefall and can be run on CPU. Training takes less than 30 seconds and gives you the following WER:

[test_set] %WER 0.42% [1 / 240, 0 ins, 1 del, 0 sub ]

We provide a Colab notebook for this recipe: 

[LibriSpeech][librispeech]

Please see RESULTS.md for the latest results.

Conformer CTC

We provide a Colab notebook to test the pre-trained model: 

TDNN LSTM CTC

We provide a Colab notebook to test the pre-trained model: 

Transducer (Conformer Encoder + LSTM Predictor)

We provide a Colab notebook to test the pre-trained model: 

Transducer (Conformer Encoder + Stateless Predictor)

We provide a Colab notebook to test the pre-trained model: 

Transducer (Zipformer Encoder + Stateless Predictor)

WER (modifiedbeamsearch beam_size=4 unless further stated)

1. 1.LibriSpeech-960hr

1. 1.LibriSpeech-960hr + GigaSpeech

1. 1.LibriSpeech-960hr + GigaSpeech + CommonVoice

[GigaSpeech][gigaspeech]

Conformer CTC

Transducer (pruned_transducer_stateless2)

Conformer Encoder + Stateless Predictor + k2 Pruned RNN-T Loss

Transducer (Zipformer Encoder + Stateless Predictor)

[Aishell][aishell]

TDNN LSTM CTC

We provide a Colab notebook to test the pre-trained model: 

Transducer (Conformer Encoder + Stateless Predictor)

We provide a Colab notebook to test the pre-trained model: 

Transducer (Zipformer Encoder + Stateless Predictor)

WER (modifiedbeamsearch beam_size=4)

[Aishell4][aishell4]

Transducer (pruned_transducer_stateless5)

1 Trained with all subsets: | | test | |-----|------------| | CER | 29.08 |

We provide a Colab notebook to test the pre-trained model: 

[TIMIT][timit]

TDNN LSTM CTC

We provide a Colab notebook to test the pre-trained model: 

TDNN LiGRU CTC

We provide a Colab notebook to test the pre-trained model: 

[TED-LIUM3][tedlium3]

Transducer (Conformer Encoder + Stateless Predictor)

We provide a Colab notebook to test the pre-trained model: 

Transducer (pruned_transducer_stateless)

We provide a Colab notebook to test the pre-trained model: 

[Aidatatang200zh][aidatatang200zh]

Transducer (pruned_transducer_stateless2)

We provide a Colab notebook to test the pre-trained model: 

[WenetSpeech][wenetspeech]

Transducer (pruned_transducer_stateless2)

We provide a Colab notebook to test the pre-trained model: 

Transducer **Streaming** (pruned_transducer_stateless5)

[Alimeeting][alimeeting]

Transducer (pruned_transducer_stateless2)

We provide a Colab notebook to test the pre-trained model: 

[TALCSASR][talcsasr]

Transducer (pruned_transducer_stateless5)

The best results for Chinese CER(%) and English WER(%) respectively (zh: Chinese, en: English): |decoding-method | dev | devzh | deven | test | testzh | testen | |--|--|--|--|--|--|--| |greedysearch| 7.30 | 6.48 | 19.19 |7.39| 6.66 | 19.13| |fastbeamsearch| 7.18 | 6.39| 18.90 | 7.27| 6.55 | 18.77| |modifiedbeam_search| 7.15 | 6.35 | 18.95 | 7.22| 6.50 | 18.70 |

We provide a Colab notebook to test the pre-trained model: 

TTS: Text-to-Speech

Supported Datasets

• —[LJSpeech][ljspeech]
• —[VCTK][vctk]
• —[LibriTTS][libritts_tts]

Supported Models

• —VITS

Deployment with C++

Once you have trained a model in icefall, you may want to deploy it with C++ without Python dependencies.

Please refer to

• —https://k2-fsa.github.io/icefall/model-export/export-with-torch-jit-script.html
• —https://k2-fsa.github.io/icefall/model-export/export-onnx.html
• —https://k2-fsa.github.io/icefall/model-export/export-ncnn.html

for how to do this.

We also provide a Colab notebook, showing you how to run a torch scripted model in [k2][k2] with C++. Please see: 

[yesno]: egs/yesno/ASR [librispeech]: egs/librispeech/ASR [aishell]: egs/aishell/ASR [aishell2]: egs/aishell2/ASR [aishell4]: egs/aishell4/ASR [timit]: egs/timit/ASR [tedlium3]: egs/tedlium3/ASR [gigaspeech]: egs/gigaspeech/ASR [aidatatang200zh]: egs/aidatatang200zh/ASR [wenetspeech]: egs/wenetspeech/ASR [alimeeting]: egs/alimeeting/ASR [talcsasr]: egs/talcsasr/ASR [ami]: egs/ami [swbd]: egs/swbd/ASR [k2]: https://github.com/k2-fsa/k2 [commonvoice]: egs/commonvoice/ASR [csj]: egs/csj/ASR [libricss]: egs/libricss/SURT [libritts_asr]: egs/libritts/ASR [libriheavy]: egs/libriheavy/ASR [mgb2]: egs/mgb2/ASR [spgispeech]: egs/spgispeech/ASR [voxpopuli]: egs/voxpopuli/ASR [xbmu-amdo31]: egs/xbmu-amdo31/ASR

[vctk]: egs/vctk/TTS [ljspeech]: egs/ljspeech/TTS [libritts_tts]: egs/libritts/TTS

Acknowledgements

Some contributors to this project were supported by Xiaomi Corporation. Others were supported by National Science Foundation CCRI award 2120435. This is not an exhaustive list of sources of support.