chendl/compositional_test

1<!---2Copyright 2021 The HuggingFace Team. All rights reserved.3 4Licensed under the Apache License, Version 2.0 (the "License");5you may not use this file except in compliance with the License.6You may obtain a copy of the License at7 8    http://www.apache.org/licenses/LICENSE-2.09 10Unless required by applicable law or agreed to in writing, software11distributed under the License is distributed on an "AS IS" BASIS,12WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.13See the License for the specific language governing permissions and14limitations under the License.15-->16 17# Automatic Speech Recognition Examples18 19## Table of Contents20 21- [Automatic Speech Recognition with CTC](#connectionist-temporal-classification)22	- [Single GPU example](#single-gpu-ctc)23	- [Multi GPU example](#multi-gpu-ctc)24	- [Examples](#examples-ctc)25		- [TIMIT](#timit-ctc)26		- [Librispeech](#librispeech-ctc)27		- [Common Voice](#common-voice-ctc)28		- [Multilingual Librispeech](#multilingual-librispeech-ctc)29- [Automatic Speech Recognition with Sequence-to-Sequence](#sequence-to-sequence)30	- [Whisper Model](#whisper-model)31	- [Speech-Encoder-Decoder Model](#warm-started-speech-encoder-decoder-model)32	- [Examples](#examples-seq2seq)33		- [Librispeech](#librispeech-seq2seq)34 35## Connectionist Temporal Classification36 37The script [`run_speech_recognition_ctc.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py) can be used to fine-tune any pretrained [Connectionist Temporal Classification Model](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForCTC) for automatic speech 38recognition on one of the [official speech recognition datasets](https://huggingface.co/datasets?task_ids=task_ids:automatic-speech-recognition) or a custom dataset.39 40Speech recognition models that have been pretrained in unsupervised fashion on audio data alone, *e.g.* [Wav2Vec2](https://huggingface.co/transformers/main/model_doc/wav2vec2.html), [HuBERT](https://huggingface.co/transformers/main/model_doc/hubert.html), [XLSR-Wav2Vec2](https://huggingface.co/transformers/main/model_doc/xlsr_wav2vec2.html), have shown to require only 41very little annotated data to yield good performance on automatic speech recognition datasets.42 43In the script [`run_speech_recognition_ctc`], we first create a vocabulary from all unique characters of both the training data and evaluation data. Then, we preprocesses the speech recognition dataset, which includes correct resampling, normalization and padding. Finally, the pretrained speech recognition model is fine-tuned on the annotated speech recognition datasets using CTC loss.44 45---46**NOTE**47 48If you encounter problems with data preprocessing by setting `--preprocessing_num_workers` > 1, 49you might want to set the environment variable `OMP_NUM_THREADS` to 1 as follows:50 51```bash52OMP_NUM_THREADS=1 python run_speech_recognition_ctc ...53```54 55If the environment variable is not set, the training script might freeze, *i.e.* see: https://github.com/pytorch/audio/issues/1021#issuecomment-72691523956 57---58 59### Single GPU CTC60 61The following command shows how to fine-tune [XLSR-Wav2Vec2](https://huggingface.co/transformers/main/model_doc/xlsr_wav2vec2.html) on [Common Voice](https://huggingface.co/datasets/common_voice) using a single GPU in half-precision.62 63```bash64python run_speech_recognition_ctc.py \65	--dataset_name="common_voice" \66	--model_name_or_path="facebook/wav2vec2-large-xlsr-53" \67	--dataset_config_name="tr" \68	--output_dir="./wav2vec2-common_voice-tr-demo" \69	--overwrite_output_dir \70	--num_train_epochs="15" \71	--per_device_train_batch_size="16" \72	--gradient_accumulation_steps="2" \73	--learning_rate="3e-4" \74	--warmup_steps="500" \75	--evaluation_strategy="steps" \76	--text_column_name="sentence" \77	--length_column_name="input_length" \78	--save_steps="400" \79	--eval_steps="100" \80	--layerdrop="0.0" \81	--save_total_limit="3" \82	--freeze_feature_encoder \83	--gradient_checkpointing \84	--chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \85	--fp16 \86	--group_by_length \87	--push_to_hub \88	--do_train --do_eval 89```90 91On a single V100 GPU, this script should run in *ca.* 1 hour 20 minutes and yield a CTC loss of **0.39** and word error rate92of **0.35**.93 94### Multi GPU CTC95 96The following command shows how to fine-tune [XLSR-Wav2Vec2](https://huggingface.co/transformers/main/model_doc/xlsr_wav2vec2.html) on [Common Voice](https://huggingface.co/datasets/common_voice) using 8 GPUs in half-precision.97 98```bash99python -m torch.distributed.launch \100	--nproc_per_node 8 run_speech_recognition_ctc.py \101	--dataset_name="common_voice" \102	--model_name_or_path="facebook/wav2vec2-large-xlsr-53" \103	--dataset_config_name="tr" \104	--output_dir="./wav2vec2-common_voice-tr-demo-dist" \105	--overwrite_output_dir \106	--num_train_epochs="15" \107	--per_device_train_batch_size="4" \108	--learning_rate="3e-4" \109	--warmup_steps="500" \110	--evaluation_strategy="steps" \111	--text_column_name="sentence" \112	--length_column_name="input_length" \113	--save_steps="400" \114	--eval_steps="100" \115	--logging_steps="1" \116	--layerdrop="0.0" \117	--save_total_limit="3" \118	--freeze_feature_encoder \119	--gradient_checkpointing \120	--chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \121	--fp16 \122	--group_by_length \123	--push_to_hub \124	--do_train --do_eval125```126 127On 8 V100 GPUs, this script should run in *ca.* 18 minutes and yield a CTC loss of **0.39** and word error rate128of **0.36**.129 130 131### Multi GPU CTC with Dataset Streaming132 133The following command shows how to use [Dataset Streaming mode](https://huggingface.co/docs/datasets/dataset_streaming.html)134to fine-tune [XLS-R](https://huggingface.co/transformers/main/model_doc/xls_r.html) 135on [Common Voice](https://huggingface.co/datasets/common_voice) using 4 GPUs in half-precision.136 137Streaming mode imposes several constraints on training:1381. We need to construct a tokenizer beforehand and define it via `--tokenizer_name_or_path`.1392. `--num_train_epochs` has to be replaced by `--max_steps`. Similarly, all other epoch-based arguments have to be 140replaced by step-based ones.1413. Full dataset shuffling on each epoch is not possible, since we don't have the whole dataset available at once. 142However, the `--shuffle_buffer_size` argument controls how many examples we can pre-download before shuffling them.143 144 145```bash146**python -m torch.distributed.launch \147	--nproc_per_node 4 run_speech_recognition_ctc_streaming.py \148	--dataset_name="common_voice" \149	--model_name_or_path="facebook/wav2vec2-xls-r-300m" \150	--tokenizer_name_or_path="anton-l/wav2vec2-tokenizer-turkish" \151	--dataset_config_name="tr" \152	--train_split_name="train+validation" \153	--eval_split_name="test" \154	--output_dir="wav2vec2-xls-r-common_voice-tr-ft" \155	--overwrite_output_dir \156	--max_steps="5000" \157	--per_device_train_batch_size="8" \158	--gradient_accumulation_steps="2" \159	--learning_rate="5e-4" \160	--warmup_steps="500" \161	--evaluation_strategy="steps" \162	--text_column_name="sentence" \163	--save_steps="500" \164	--eval_steps="500" \165	--logging_steps="1" \166	--layerdrop="0.0" \167	--eval_metrics wer cer \168	--save_total_limit="1" \169	--mask_time_prob="0.3" \170	--mask_time_length="10" \171	--mask_feature_prob="0.1" \172	--mask_feature_length="64" \173	--freeze_feature_encoder \174	--chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \175	--max_duration_in_seconds="20" \176	--shuffle_buffer_size="500" \177	--fp16 \178	--push_to_hub \179	--do_train --do_eval \180	--gradient_checkpointing**181```182 183On 4 V100 GPUs, this script should run in *ca.* 3h 31min and yield a CTC loss of **0.35** and word error rate184of **0.29**.185 186### Examples CTC187 188The following tables present a couple of example runs on the most popular speech-recognition datasets. 189The presented performances are by no means optimal as no hyper-parameter tuning was done. Nevertheless, 190they can serve as a baseline to improve upon.191 192 193#### TIMIT CTC194 195- [TIMIT](https://huggingface.co/datasets/timit_asr)196 197| Dataset | Dataset Config | Pretrained Model | Word error rate on eval | Phoneme error rate on eval | GPU setup | Training time | Fine-tuned Model & Logs | Command to reproduce |198|-------|------------------------------|-------------|---------------|---------------|----------------------|-------------| -------------| ------- |199| [TIMIT](https://huggingface.co/datasets/timit_asr)| -  | [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) | 0.21 | - | 1 GPU TITAN RTX |  32min                      | [here](https://huggingface.co/patrickvonplaten/wav2vec2-base-timit-fine-tuned)  | [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-base-timit-fine-tuned/blob/main/run.sh) |200| [TIMIT](https://huggingface.co/datasets/timit_asr)| -  | [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) | 0.21 | - | 1 GPU TITAN RTX |  32min                      | [here](https://huggingface.co/patrickvonplaten/wav2vec2-base-timit-fine-tuned)  | [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-base-timit-fine-tuned/blob/main/run.sh) |201| [TIMIT](https://huggingface.co/datasets/timit_asr)| -  | [unispeech-large-1500h-cv](https://huggingface.co/microsoft/unispeech-large-1500h-cv) | 0.22 | - | 1 GPU TITAN RTX |  35min                      | [here](https://huggingface.co/patrickvonplaten/unispeech-large-1500h-cv-timit)  | [run.sh](https://huggingface.co/patrickvonplaten/unispeech-large-1500h-cv-timit/blob/main/run.sh) |202| [TIMIT](https://huggingface.co/datasets/timit_asr)| -  | [asapp/sew-mid-100k](https://huggingface.co/asapp/sew-mid-100k) | 0.30 | - | 1 GPU TITAN RTX |  28min                      | [here](https://huggingface.co/patrickvonplaten/sew-small-100k-timit)  | [run.sh](https://huggingface.co/patrickvonplaten/sew-small-100k-timit/blob/main/run.sh) |203| [TIMIT](https://huggingface.co/datasets/timit_asr)| -  | [ntu-spml/distilhubert](https://huggingface.co/ntu-spml/distilhubert) | 0.68 | - | 1 GPU TITAN RTX |  26min                      | [here](https://huggingface.co/patrickvonplaten/distilhubert-timit)  | [run.sh](https://huggingface.co/patrickvonplaten/distilhubert-timit/blob/main/run.sh) |204 205 206#### Librispeech CTC207 208- [Librispeech](https://huggingface.co/datasets/librispeech_asr)209 210| Dataset | Dataset Config | Pretrained Model | Word error rate on eval | Phoneme error rate on eval | GPU setup | Training time | Fine-tuned Model & Logs | Command to reproduce |211|-------|------------------------------|-------------|---------------|---------------|----------------------|-------------| -------------| ------- |212| [Librispeech](https://huggingface.co/datasets/librispeech_asr)| `"clean"` - `"train.100"` |  [microsoft/wavlm-large](https://huggingface.co/microsoft/wavlm-large) | 0.049 | - | 8 GPU V100 | 1h30min  | [here](https://huggingface.co/patrickvonplaten/wavlm-libri-clean-100h-large) | [run.sh](https://huggingface.co/patrickvonplaten/wavlm-libri-clean-100h-large/blob/main/run.sh) |213| [Librispeech](https://huggingface.co/datasets/librispeech_asr)| `"clean"` - `"train.100"` |  [microsoft/wavlm-base-plus](https://huggingface.co/microsoft/wavlm-base-plus) | 0.068 | - | 8 GPU V100 | 1h30min  | [here](https://huggingface.co/patrickvonplaten/wavlm-libri-clean-100h-base-plus) | [run.sh](https://huggingface.co/patrickvonplaten/wavlm-libri-clean-100h-base-plus/blob/main/run.sh) |214| [Librispeech](https://huggingface.co/datasets/librispeech_asr)| `"clean"` - `"train.100"` |  [facebook/wav2vec2-large-lv60](https://huggingface.co/facebook/wav2vec2-large-lv60) | 0.042 | - | 8 GPU V100 | 1h30min  | [here](https://huggingface.co/patrickvonplaten/wav2vec2-librispeech-clean-100h-demo-dist) | [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-librispeech-clean-100h-demo-dist/blob/main/run.sh) |215| [Librispeech](https://huggingface.co/datasets/librispeech_asr)| `"clean"` - `"train.100"` |  [facebook/wav2vec2-large-lv60](https://huggingface.co/facebook/wav2vec2-large-lv60) | 0.042 | - | 8 GPU V100 | 1h30min  | [here](https://huggingface.co/patrickvonplaten/wav2vec2-librispeech-clean-100h-demo-dist) | [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-librispeech-clean-100h-demo-dist/blob/main/run.sh) |216| [Librispeech](https://huggingface.co/datasets/librispeech_asr)| `"clean"` - `"train.100"` |  [facebook/hubert-large-ll60k](https://huggingface.co/facebook/hubert-large-ll60k) | 0.088 | - | 8 GPU V100 | 1h30min  | [here](https://huggingface.co/patrickvonplaten/hubert-librispeech-clean-100h-demo-dist) | [run.sh](https://huggingface.co/patrickvonplaten/hubert-librispeech-clean-100h-demo-dist/blob/main/run.sh) |217| [Librispeech](https://huggingface.co/datasets/librispeech_asr)| `"clean"` - `"train.100"` |  [asapp/sew-mid-100k](https://huggingface.co/asapp/sew-mid-100k) | 0.167 | | 8 GPU V100 | 54min  | [here](https://huggingface.co/patrickvonplaten/sew-mid-100k-librispeech-clean-100h-ft) | [run.sh](https://huggingface.co/patrickvonplaten/sew-mid-100k-librispeech-clean-100h-ft/blob/main/run.sh) |218 219 220#### Common Voice CTC221 222- [Common Voice](https://huggingface.co/datasets/common_voice)223 224| Dataset | Dataset Config | Pretrained Model | Word error rate on eval | Phoneme error rate on eval | GPU setup | Training time | Fine-tuned Model & Logs | Command to reproduce |225|-------|------------------------------|-------------|---------------|---------------|----------------------|-------------| -------------| ------- |226| [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_3_0)| `"tr"`  | [facebook/wav2vec2-large-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m)  | - |  0.099   | 8 GPU V100   |  23min                 | [here](https://huggingface.co/patrickvonplaten/xls-r-300m-tr-phoneme)      |  [run.sh](https://huggingface.co/patrickvonplaten/xls-r-300m-tr-phoneme/blob/main/run.sh) |227| [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_3_0)| `"it"`  | [facebook/wav2vec2-large-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m)  | - |  0.077   | 8 GPU V100   |  23min                 | [here](https://huggingface.co/patrickvonplaten/xls-r-300m-it-phoneme)      |  [run.sh](https://huggingface.co/patrickvonplaten/xls-r-300m-it-phoneme/blob/main/run.sh) |228| [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_3_0)| `"sv-SE"`  | [facebook/wav2vec2-large-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m)  | - |  0.099   | 8 GPU V100   |  23min                 | [here](https://huggingface.co/patrickvonplaten/xls-r-300m-sv-phoneme)      |  [run.sh](https://huggingface.co/patrickvonplaten/xls-r-300m-sv-phoneme/blob/main/run.sh) |229| [Common Voice](https://huggingface.co/datasets/common_voice)| `"tr"`  | [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53)  | 0.36 |  -      | 8 GPU V100   |  18min                 | [here](https://huggingface.co/patrickvonplaten/wav2vec2-common_voice-tr-demo-dist)      |  [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-common_voice-tr-demo-dist/blob/main/run_dist.sh) |230| [Common Voice](https://huggingface.co/datasets/common_voice)| `"tr"`  | [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53)  | 0.31  | -    | 8 GPU V100   |  1h05                 | [here](https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-common_voice-tr-ft)      |  [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-common_voice-tr-ft/blob/main/run.sh) |231| [Common Voice](https://huggingface.co/datasets/common_voice)| `"tr"`  | [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) | 0.35 | - | 1 GPU V100   |  1h20min                      | [here](https://huggingface.co/patrickvonplaten/wav2vec2-common_voice-tr-demo)  | [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-common_voice-tr-demo/blob/main/run.sh) |232| [Common Voice](https://huggingface.co/datasets/common_voice)| `"tr"`  | [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m)  | 0.31     | - | 8 GPU V100   |  1h05            | [here](https://huggingface.co/patrickvonplaten/wav2vec2-large-xls-r-300m-common_voice-tr-ft)      |  [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-large-xls-r-300m-common_voice-tr-ft/blob/main/run.sh) |233| [Common Voice](https://huggingface.co/datasets/common_voice)| `"tr"`  | [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b)  | 0.21 | -  | 2 GPU Titan 24 GB RAM   |  15h10            | [here](https://huggingface.co/patrickvonplaten/wav2vec2-xls-r-1b-common_voice-tr-ft)      |  [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-large-xls-r-1b-common_voice-tr-ft/blob/main/run.sh) |234| [Common Voice](https://huggingface.co/datasets/common_voice)| `"tr"` in streaming mode  | [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m)  | 0.29     | - | 4 GPU V100   |  3h31            | [here](https://huggingface.co/anton-l/wav2vec2-xls-r-common_voice-tr-ft-stream)      |  [run.sh](https://huggingface.co/anton-l/wav2vec2-xls-r-common_voice-tr-ft-stream/blob/main/run.sh) |235 236 237#### Multilingual Librispeech CTC238 239- [Multilingual Librispeech](https://huggingface.co/datasets/multilingual_librispeech)240 241| Dataset | Dataset Config | Pretrained Model | Word error rate on eval | Phoneme error rate on eval | GPU setup | Training time | Fine-tuned Model & Logs | Command to reproduce |242|-------|------------------------------|-------------|---------------|---------------|----------------------|-------------| -------------| ------- |243| [Multilingual Librispeech](https://huggingface.co/datasets/multilingual_librispeech)| `"german"`  | [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53)  | 0.13  | -     | 1 GPU Titan 24 GB RAM  |  15h04                 | [here](https://huggingface.co/patrickvonplaten/wav2vec2-xlsr-53-300m-mls-german-ft)      |  [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-xlsr-53-300m-mls-german-ft/blob/main/run.sh) |244| [Multilingual Librispeech](https://huggingface.co/datasets/multilingual_librispeech)| `"german"`  | [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m)  | 0.15 | -     | 1 GPU Titan 24 GB RAM  |  15h04                 | [here](https://huggingface.co/patrickvonplaten/wav2vec2-300m-mls-german-ft)      |  [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-300m-mls-german-ft/blob/main/run.sh) |245 246## Sequence to Sequence247 248The script [`run_speech_recognition_seq2seq.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py) can be used to fine-tune any [Speech Sequence-to-Sequence Model](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForSpeechSeq2Seq) for automatic speech 249recognition on one of the [official speech recognition datasets](https://huggingface.co/datasets?task_ids=task_ids:automatic-speech-recognition) or a custom dataset. This includes the Whisper model from OpenAI or a warm-started Speech-Encoder-Decoder Model, examples for which are included below.250 251### Whisper Model252We can load all components of the Whisper model directly from the pretrained checkpoint, including the pretrained model weights, feature extractor and tokenizer. We simply have to specify our fine-tuning dataset and training hyperparameters.253 254#### Single GPU Whisper Training255The following example shows how to fine-tune the [Whisper small](https://huggingface.co/openai/whisper-small) checkpoint on the Hindi subset of [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) using a single GPU device in half-precision:256```bash257python run_speech_recognition_seq2seq.py \258	--model_name_or_path="openai/whisper-small" \259	--dataset_name="mozilla-foundation/common_voice_11_0" \260	--dataset_config_name="hi" \261	--language="hindi" \262	--train_split_name="train+validation" \263	--eval_split_name="test" \264	--max_steps="5000" \265	--output_dir="./whisper-small-hi" \266	--per_device_train_batch_size="16" \267	--gradient_accumulation_steps="2" \268	--per_device_eval_batch_size="16" \269	--logging_steps="25" \270	--learning_rate="1e-5" \271	--warmup_steps="500" \272	--evaluation_strategy="steps" \273	--eval_steps="1000" \274	--save_strategy="steps" \275	--save_steps="1000" \276	--generation_max_length="225" \277	--preprocessing_num_workers="16" \278	--length_column_name="input_length" \279	--max_duration_in_seconds="30" \280	--text_column_name="sentence" \281	--freeze_feature_encoder="False" \282	--gradient_checkpointing \283	--group_by_length \284	--fp16 \285	--overwrite_output_dir \286	--do_train \287	--do_eval \288	--predict_with_generate \289	--use_auth_token290```291On a single V100, training should take approximately 8 hours, with a final cross-entropy loss of **1e-4** and word error rate of **32.6%**.292 293If training on a different language, you should be sure to change the `language` argument. The `language` argument should be omitted for English speech recognition.294 295#### Multi GPU Whisper Training296The following example shows how to fine-tune the [Whisper small](https://huggingface.co/openai/whisper-small) checkpoint on the Hindi subset of [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) using 2 GPU devices in half-precision:297```bash298python -m torch.distributed.launch \299 	--nproc_per_node 2 run_speech_recognition_seq2seq.py \300	--model_name_or_path="openai/whisper-small" \301	--dataset_name="mozilla-foundation/common_voice_11_0" \302	--dataset_config_name="hi" \303	--language="hindi" \304	--train_split_name="train+validation" \305	--eval_split_name="test" \306	--max_steps="5000" \307	--output_dir="./whisper-small-hi" \308	--per_device_train_batch_size="16" \309	--per_device_eval_batch_size="16" \310	--logging_steps="25" \311	--learning_rate="1e-5" \312	--warmup_steps="500" \313	--evaluation_strategy="steps" \314	--eval_steps="1000" \315	--save_strategy="steps" \316	--save_steps="1000" \317	--generation_max_length="225" \318	--preprocessing_num_workers="16" \319	--length_column_name="input_length" \320	--max_duration_in_seconds="30" \321	--text_column_name="sentence" \322	--freeze_feature_encoder="False" \323	--gradient_checkpointing \324	--group_by_length \325	--fp16 \326	--overwrite_output_dir \327	--do_train \328	--do_eval \329	--predict_with_generate \330	--use_auth_token331```332On two V100s, training should take approximately 4 hours, with a final cross-entropy loss of **1e-4** and word error rate of **32.6%**.333 334### Warm-Started Speech-Encoder-Decoder Model335A very common use case is to leverage a pretrained speech encoder model,336*e.g.* [Wav2Vec2](https://huggingface.co/transformers/main/model_doc/wav2vec2.html), [HuBERT](https://huggingface.co/transformers/main/model_doc/hubert.html) or [XLSR-Wav2Vec2](https://huggingface.co/transformers/main/model_doc/xlsr_wav2vec2.html), with a pretrained text decoder model, *e.g.* [BART](https://huggingface.co/docs/transformers/main/en/model_doc/bart#transformers.BartForCausalLM) or [GPT-2](https://huggingface.co/docs/transformers/main/en/model_doc/gpt2#transformers.GPT2ForCausalLM), to create a [Speech-Encoder-Decoder Model](https://huggingface.co/docs/transformers/main/en/model_doc/speech-encoder-decoder#speech-encoder-decoder-models).337 338By pairing a pretrained speech model with a pretrained text model, the warm-started model has prior knowledge of both the source audio and target text domains. However, the cross-attention weights between the encoder and decoder are randomly initialised. Thus, the model requires fine-tuning to learn the cross-attention weights and align the encoder mapping with that of the decoder. We can perform this very fine-tuning procedure using the example script.339 340As an example, let's instantiate a *Wav2Vec2-2-Bart* model with the `SpeechEnocderDecoderModel` framework. First create an empty repo on `hf.co`:341 342```bash343huggingface-cli repo create wav2vec2-2-bart-base344git clone https://huggingface.co/<your-user-name>/wav2vec2-2-bart-base345cd wav2vec2-2-bart-base346```347 348Next, run the following script **inside** the just cloned repo:349 350```python351from transformers import SpeechEncoderDecoderModel, AutoFeatureExtractor, AutoTokenizer, Wav2Vec2Processor352 353# checkpoints to leverage354encoder_id = "facebook/wav2vec2-base"355decoder_id = "facebook/bart-base"356 357# load and save speech-encoder-decoder model358# set some hyper-parameters for training and evaluation359model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(encoder_id, decoder_id, encoder_add_adapter=True, encoder_feat_proj_dropout=0.0, encoder_layerdrop=0.0, max_length=200, num_beams=5)360model.config.decoder_start_token_id = model.decoder.config.bos_token_id361model.config.pad_token_id = model.decoder.config.pad_token_id362model.config.eos_token_id = model.decoder.config.eos_token_id363model.save_pretrained("./")364 365# load and save processor366feature_extractor = AutoFeatureExtractor.from_pretrained(encoder_id)367tokenizer = AutoTokenizer.from_pretrained(decoder_id)368processor = Wav2Vec2Processor(feature_extractor, tokenizer)369processor.save_pretrained("./")370```371 372Finally, we can upload all files:373```bash374git lfs install375git add . && git commit -m "upload model files" && git push376```377 378and link the official `run_speech_recognition_seq2seq.py` script to the folder:379 380```bash381ln -s $(realpath <path/to/transformers>/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py) ./382```383 384Note that we have added a randomly initialized _adapter layer_ to `wav2vec2-base` with the argument385`encoder_add_adapter=True`. This adapter sub-samples the output sequence of 386`wav2vec2-base` along the time dimension. By default, a single387output vector of `wav2vec2-base` has a receptive field of *ca.* 25ms (*cf.* 388Section *4.2* of the [official Wav2Vec2 paper](https://arxiv.org/pdf/2006.11477.pdf)), which represents a little less a single character. On the other hand, BART389makes use of a sentence-piece tokenizer as an input processor, so that a single 390hidden vector of `bart-base` represents *ca.* 4 characters. To better align the 391receptive field of the *Wav2Vec2* output vectors with *BART*'s hidden-states in the cross-attention 392mechanism, we further subsample *Wav2Vec2*'s output by a factor of 8 by 393adding a convolution-based adapter.394 395Having warm-started the speech-encoder-decoder model under `<your-user-name>/wav2vec2-2-bart`, we can now fine-tune it on the task of speech recognition.396 397In the script [`run_speech_recognition_seq2seq`], we load the warm-started model, 398feature extractor, and tokenizer, process a speech recognition dataset, 399and subsequently make use of the [`Seq2SeqTrainer`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Seq2SeqTrainer) to train our system.400Note that it is important to align the target transcriptions with the decoder's vocabulary. For example, the [`Librispeech`](https://huggingface.co/datasets/librispeech_asr) dataset only contains captilized letters in the transcriptions,401whereas BART was pretrained mostly on normalized text. Thus, it is recommended to add the argument 402`--do_lower_case` to the fine-tuning script when using a warm-started `SpeechEncoderDecoderModel`. 403The model is fine-tuned on the standard cross-entropy language modeling404loss for sequence-to-sequence (just like *T5* or *BART* in natural language processing).405 406---407**NOTE**408 409If you encounter problems with data preprocessing by setting `--preprocessing_num_workers` > 1, 410you might want to set the environment variable `OMP_NUM_THREADS` to 1 as follows:411 412```bash413OMP_NUM_THREADS=1 python run_speech_recognition_ctc ...414```415 416If the environment variable is not set, the training script might freeze, *i.e.* see: https://github.com/pytorch/audio/issues/1021#issuecomment-726915239.417 418---419 420#### Single GPU Seq2Seq421 422The following command shows how to fine-tune [XLSR-Wav2Vec2](https://huggingface.co/transformers/main/model_doc/xlsr_wav2vec2.html) on [Common Voice](https://huggingface.co/datasets/common_voice) using a single GPU in half-precision.423 424```bash425python run_speech_recognition_seq2seq.py \426	--dataset_name="librispeech_asr" \427	--model_name_or_path="./" \428	--dataset_config_name="clean" \429	--train_split_name="train.100" \430	--eval_split_name="validation" \431	--output_dir="./" \432	--preprocessing_num_workers="16" \433	--length_column_name="input_length" \434	--overwrite_output_dir \435	--num_train_epochs="5" \436	--per_device_train_batch_size="8" \437	--per_device_eval_batch_size="8" \438	--gradient_accumulation_steps="8" \439	--learning_rate="3e-4" \440	--warmup_steps="400" \441	--evaluation_strategy="steps" \442	--text_column_name="text" \443	--save_steps="400" \444	--eval_steps="400" \445	--logging_steps="10" \446	--save_total_limit="1" \447	--freeze_feature_encoder \448	--gradient_checkpointing \449	--fp16 \450	--group_by_length \451	--predict_with_generate \452	--generation_max_length="40" \453	--generation_num_beams="1" \454	--do_train --do_eval \455	--do_lower_case456```457 458On a single V100 GPU, this script should run in *ca.* 5 hours and yield a 459cross-entropy loss of **0.405** and word error rate of **0.0728**.460 461#### Multi GPU Seq2Seq462 463The following command shows how to fine-tune [XLSR-Wav2Vec2](https://huggingface.co/transformers/main/model_doc/xlsr_wav2vec2.html) on [Common Voice](https://huggingface.co/datasets/common_voice) using 8 GPUs in half-precision.464 465```bash466python -m torch.distributed.launch \467 	--nproc_per_node 8 run_speech_recognition_seq2seq.py \468	--dataset_name="librispeech_asr" \469	--model_name_or_path="./" \470	--dataset_config_name="clean" \471	--train_split_name="train.100" \472	--eval_split_name="validation" \473	--output_dir="./" \474	--preprocessing_num_workers="16" \475	--length_column_name="input_length" \476	--overwrite_output_dir \477	--num_train_epochs="5" \478	--per_device_train_batch_size="8" \479	--per_device_eval_batch_size="8" \480	--gradient_accumulation_steps="1" \481	--learning_rate="3e-4" \482	--warmup_steps="400" \483	--evaluation_strategy="steps" \484	--text_column_name="text" \485	--save_steps="400" \486	--eval_steps="400" \487	--logging_steps="10" \488	--save_total_limit="1" \489	--freeze_feature_encoder \490	--gradient_checkpointing \491	--fp16 \492	--group_by_length \493	--predict_with_generate \494	--do_train --do_eval \495	--do_lower_case496```497 498On 8 V100 GPUs, this script should run in *ca.* 45 minutes and yield a cross-entropy loss of **0.405** and word error rate of **0.0728**499 500### Examples Seq2Seq501 502#### Librispeech Seq2Seq503 504- [Librispeech](https://huggingface.co/datasets/librispeech_asr)505 506| Dataset                                                        | Dataset Config            | Pretrained Model                                                                                                                                          | Word error rate on eval | Phoneme error rate on eval | GPU setup  | Training time | Fine-tuned Model & Logs                                               | Command to reproduce                                                                                                                                                                                              |507|----------------------------------------------------------------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|----------------------------|------------|---------------|-----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|508| [Librispeech](https://huggingface.co/datasets/librispeech_asr) | `"clean"` - `"train.100"` | [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) and [facebook/bart-base](https://huggingface.co/facebook/bart-base)               | 0.0728                  | -                          | 8 GPU V100 | 45min         | [here](https://huggingface.co/patrickvonplaten/wav2vec2-2-bart-base)  | [create_model.py](https://huggingface.co/patrickvonplaten/wav2vec2-2-bart-base/blob/main/create_model.py) & [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-2-bart-base/blob/main/run_librispeech.sh)   |509| [Librispeech](https://huggingface.co/datasets/librispeech_asr) | `"clean"` - `"train.100"` | [facebook/wav2vec2-large-lv60](https://huggingface.co/facebook/wav2vec2-large-lv60) and [facebook/bart-large](https://huggingface.co/facebook/bart-large) | 0.0486                  | -                          | 8 GPU V100 | 1h20min       | [here](https://huggingface.co/patrickvonplaten/wav2vec2-2-bart-large) | [create_model.py](https://huggingface.co/patrickvonplaten/wav2vec2-2-bart-large/blob/main/create_model.py) & [run.sh](https://huggingface.co/patrickvonplaten/wav2vec2-2-bart-large/blob/main/run_librispeech.sh) |510