espnet/tedlium2_streaming_transformer
ESPnet2 streaming ASR model
espnet/tedlium2_streaming_transformer
A streaming Transformer: it decodes while the audio is still arriving, a block at a time, rather than waiting for the end of the utterance. Trained on TED-LIUM 2 with the egs2/tedlium2/asr1 recipe and conf/train_asr_streaming_transformer.yaml, with a BPE-500 language model beside it.
Trained and originally published by [Keqi Deng](https://huggingface.co/D-Keqi), as `D-Keqi/espnet_asr_train_asr_streaming_transformer_raw_en_bpe500_sp_valid.acc.ave`. This copy is here so that the demo notebooks and the documentation point at a model the organization keeps alive; the weights are the same ones.
Usage
import soundfile as sf
from espnet2.bin.asr_inference_streaming import Speech2TextStreaming
s2t = Speech2TextStreaming.from_pretrained(
"espnet/tedlium2_streaming_transformer",
device="cpu",
beam_size=20,
ctc_weight=0.5,
penalty=0.0,
nbest=1,
disable_repetition_detection=True,
)
speech, rate = sf.read("audio.wav", dtype="float32") # 16 kHz, one channel
# 640 samples is 40 ms: the hypothesis comes back after each slice, and grows
chunk = 640
slices = len(speech) // chunk
for i in range(slices):
results = s2t(speech=speech[i * chunk : (i + 1) * chunk], is_final=False)
if results:
print(results[0][0])
results = s2t(speech=speech[slices * chunk :], is_final=True)
print(results[0][0])There is a notebook that runs this end to end: `Demos/asr_streaming_demo.ipynb`.
For non-streaming recognition, OWSM-CTC is far more accurate and covers 150+ languages.
Results
TED-LIUM 2, decoded with the language model in this repository.
The full tables are in `exp/asr_train_asr_streaming_transformer_raw_en_bpe500_sp/RESULTS.md`.
Trained with espnet 0.9.8 and pytorch 1.5.1, in November 2021, and loaded by current ESPnet through the meta.yaml in this repository.
Citing ESPnet
@inproceedings{watanabe2018espnet,
author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson {Enrique Yalta Soplin} and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
title={{ESPnet}: End-to-End Speech Processing Toolkit},
year={2018},
booktitle={Proceedings of Interspeech},
pages={2207--2211},
doi={10.21437/Interspeech.2018-1456},
url={http://dx.doi.org/10.21437/Interspeech.2018-1456}
}