espnet/Turn_taking_prediction_SWBD

1---2tags:3- espnet4- audio5language: en6datasets:7- swbd8license: cc-by-4.09pipeline_tag: automatic-speech-recognition10---11 12## Usage13 14```python15import librosa16from espnet2.bin.asr_inference import Speech2Text17 18speech2text = Speech2Text.from_pretrained(model_tag="espnet/Turn_taking_prediction_SWBD")19# librosa resamples and mixes to one channel, so any file works; 16000 is20# what nearly every espnet recogniser is trained on - check this model's21# config if its audio is not 16 kHz22speech, rate = librosa.load("audio.wav", sr=16000, mono=True)23text, *_ = speech2text(speech)[0]24print(text)25```26 27## ESPnet2 Turn taking model28 29### `espnet/Turn_taking_prediction_SWBD`30 31This model was trained by “siddhu001” using swbd recipe in [espnet](https://github.com/espnet/espnet/).32 33### Demo: How to use in ESPnet234 35Follow the [ESPnet installation instructions](https://espnet.github.io/espnet/installation.html)36if you haven't done that already.37 38```bash39cd espnet40git checkout cea64abdeea5fa4f3da1a898be396e8c95c6e3ae41pip install -e .42cd egs2/swbd/asr143./run.sh --skip_data_prep false --skip_train true --download_model espnet/Turn_taking_prediction_SWBD44```45 46Use the following Python code to run inference and obtain the probability of a turn-taking event every 40 milliseconds.47```python48import soundfile49import os50import sys51from espnet2.bin.asr_inference import Speech2Text52speech2text = Speech2Text("exp/asr_train_asr_whisper_turn_taking_raw_en_word/config.yaml", "exp/asr_train_asr_whisper_turn_taking_raw_en_word/valid.loss.ave.pth",device="cuda", run_chunk=True)53audio, rate = soundfile.read(key)54print(speech2text(audio)[0][0])55```56 57# RESULTS58 59## asr_train_asr_whisper_turn_taking_target_raw_en_word60### ROC_AUC61 62|dataset|Continuation|Backchannel|Turn change|Interruption|Silence|Overall|63|---|---|---|---|---|---|---|64|decode_asr_chunk_asr_model_valid.loss.ave/test|93.3|89.4|90.8|91.3|95.1|92.0|65 66## ASR config67 68<details><summary>expand</summary>69 70```71config: conf/train_asr_whisper_3_uselast.yaml72print_config: false73log_level: INFO74drop_last_iter: false75dry_run: false76iterator_type: sequence77valid_iterator_type: null78output_dir: exp/asr_train_asr_whisper_3_uselast_raw_en_word79ngpu: 180seed: 081num_workers: 182num_att_plot: 383dist_backend: nccl84dist_init_method: env://85dist_world_size: 886dist_rank: 087local_rank: 088dist_master_addr: localhost89dist_master_port: 3342990dist_launcher: null91multiprocessing_distributed: true92unused_parameters: true93sharded_ddp: false94cudnn_enabled: true95cudnn_benchmark: false96cudnn_deterministic: true97collect_stats: false98write_collected_feats: false99max_epoch: 32100patience: null101val_scheduler_criterion:102- valid103- loss104early_stopping_criterion:105- valid106- loss107- min108best_model_criterion:109-   - valid110    - loss111    - min112keep_nbest_models: 10113nbest_averaging_interval: 0114grad_clip: 5.0115grad_clip_type: 2.0116grad_noise: false117accum_grad: 1118no_forward_run: false119resume: true120train_dtype: float32121use_amp: false122log_interval: null123use_matplotlib: true124use_tensorboard: true125create_graph_in_tensorboard: false126use_wandb: false127wandb_project: null128wandb_id: null129wandb_entity: null130wandb_name: null131wandb_model_log_interval: -1132detect_anomaly: false133use_adapter: false134adapter: lora135save_strategy: all136adapter_conf: {}137pretrain_path: null138init_param: []139ignore_init_mismatch: false140freeze_param:141- encoder142num_iters_per_epoch: 750143batch_size: 4000144valid_batch_size: null145batch_bins: 1000000146valid_batch_bins: null147train_shape_file:148- exp/asr_stats_raw_en_word/train/speech_shape149- exp/asr_stats_raw_en_word/train/text_shape.word150valid_shape_file:151- exp/asr_stats_raw_en_word/valid/speech_shape152- exp/asr_stats_raw_en_word/valid/text_shape.word153batch_type: folded154valid_batch_type: null155fold_length:156- 80000157- 150158sort_in_batch: descending159shuffle_within_batch: false160sort_batch: descending161multiple_iterator: false162chunk_length: 500163chunk_shift_ratio: 0.5164num_cache_chunks: 1024165chunk_excluded_key_prefixes: []166chunk_default_fs: null167train_data_path_and_name_and_type:168-   - dump/raw/train/wav.scp169    - speech170    - kaldi_ark171-   - dump/raw/train/text172    - text173    - text174valid_data_path_and_name_and_type:175-   - dump/raw/valid/wav.scp176    - speech177    - kaldi_ark178-   - dump/raw/valid/text179    - text180    - text181allow_variable_data_keys: false182max_cache_size: 0.0183max_cache_fd: 32184allow_multi_rates: false185valid_max_cache_size: null186exclude_weight_decay: false187exclude_weight_decay_conf: {}188optim: adam189optim_conf:190    lr: 0.0005191scheduler: warmuplr192scheduler_conf:193    warmup_steps: 500194token_list:195- <blank>196- <unk>197- C198- NA199- I200- BC201- T202- <sos/eos>203init: null204input_size: 1205ctc_conf:206    dropout_rate: 0.0207    ctc_type: builtin208    reduce: true209    ignore_nan_grad: null210    zero_infinity: true211    brctc_risk_strategy: exp212    brctc_group_strategy: end213    brctc_risk_factor: 0.0214joint_net_conf: null215use_preprocessor: true216use_lang_prompt: false217use_nlp_prompt: false218token_type: word219bpemodel: null220non_linguistic_symbols: null221cleaner: null222g2p: null223speech_volume_normalize: null224rir_scp: null225rir_apply_prob: 1.0226noise_scp: null227noise_apply_prob: 1.0228noise_db_range: '13_15'229short_noise_thres: 0.5230aux_ctc_tasks: []231frontend: null232frontend_conf: {}233specaug: null234specaug_conf: {}235normalize: null236normalize_conf: {}237model: espnet238model_conf:239    ctc_weight: 0.0240    lsm_weight: 0.1241    length_normalized_loss: false242    superb_setup: true243    num_class: 5244    ssl_input_size: 1024245    extract_feats_in_collect_stats: false246    use_only_last_correct: true247preencoder: null248preencoder_conf: {}249encoder: whisper250encoder_conf:251    whisper_model: medium252    dropout_rate: 0.0253    use_specaug: false254    specaug_conf:255        apply_time_warp: true256        time_warp_window: 5257        time_warp_mode: bicubic258        apply_freq_mask: true259        freq_mask_width_range:260        - 0261        - 40262        num_freq_mask: 2263        apply_time_mask: true264        time_mask_width_ratio_range:265        - 0.0266        - 0.12267        num_time_mask: 5268postencoder: null269postencoder_conf: {}270decoder: null271decoder_conf: {}272preprocessor: default273preprocessor_conf: {}274required:275- output_dir276- token_list277version: '202402'278distributed: true279```280 281</details>282 283 284 285### Citing ESPnet286 287```BibTex288 289@inproceedings{290arora2025talking,291title={Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics},292author={Siddhant Arora and Zhiyun Lu and Chung-Cheng Chiu and Ruoming Pang and Shinji Watanabe},293booktitle={The Thirteenth International Conference on Learning Representations},294year={2025},295url={https://openreview.net/forum?id=2e4ECh0ikn}296}297 298@inproceedings{watanabe2018espnet,299  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},300  title={{ESPnet}: End-to-End Speech Processing Toolkit},301  year={2018},302  booktitle={Proceedings of Interspeech},303  pages={2207--2211},304  doi={10.21437/Interspeech.2018-1456},305  url={http://dx.doi.org/10.21437/Interspeech.2018-1456}306}307 308```309