Sangramsing/whisper-tiny

1---2language: 3- en4- zh5- de6- es7- ru8- ko9- fr10- ja11- pt12- tr13- pl14- ca15- nl16- ar17- sv18- it19- id20- hi21- fi22- vi23- he24- uk25- el26- ms27- cs28- ro29- da30- hu31- ta32- no33- th34- ur35- hr36- bg37- lt38- la39- mi40- ml41- cy42- sk43- te44- fa45- lv46- bn47- sr48- az49- sl50- kn51- et52- mk53- br54- eu55- is56- hy57- ne58- mn59- bs60- kk61- sq62- sw63- gl64- mr65- pa66- si67- km68- sn69- yo70- so71- af72- oc73- ka74- be75- tg76- sd77- gu78- am79- yi80- lo81- uz82- fo83- ht84- ps85- tk86- nn87- mt88- sa89- lb90- my91- bo92- tl93- mg94- as95- tt96- haw97- ln98- ha99- ba100- jw101- su102tags:103- audio104- automatic-speech-recognition105- hf-asr-leaderboard106widget:107- example_title: Librispeech sample 1108  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac109- example_title: Librispeech sample 2110  src: https://cdn-media.huggingface.co/speech_samples/sample2.flac111model-index:112- name: whisper-tiny113  results:114  - task:115      name: Automatic Speech Recognition116      type: automatic-speech-recognition117    dataset:118      name: LibriSpeech (clean)119      type: librispeech_asr120      config: clean121      split: test122      args: 123        language: en124    metrics:125    - name: Test WER126      type: wer127      value: 7.54128  - task:129      name: Automatic Speech Recognition130      type: automatic-speech-recognition131    dataset:132      name: LibriSpeech (other)133      type: librispeech_asr134      config: other135      split: test136      args: 137        language: en138    metrics:139    - name: Test WER140      type: wer141      value:  17.15142  - task:143      name: Automatic Speech Recognition144      type: automatic-speech-recognition145    dataset:146      name: Common Voice 11.0147      type: mozilla-foundation/common_voice_11_0148      config: hi149      split: test150      args:151        language: hi152    metrics:153    - name: Test WER154      type: wer155      value: 141156pipeline_tag: automatic-speech-recognition157license: apache-2.0158---159 160# Whisper161 162Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. Trained on 680k hours 163of labelled data, Whisper models demonstrate a strong ability to generalise to many datasets and domains **without** the need 164for fine-tuning.165 166Whisper was proposed in the paper [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356) 167by Alec Radford et al from OpenAI. The original code repository can be found [here](https://github.com/openai/whisper).168 169**Disclaimer**: Content for this model card has partly been written by the Hugging Face team, and parts of it were 170copied and pasted from the original model card.171 172## Model details173 174Whisper is a Transformer based encoder-decoder model, also referred to as a _sequence-to-sequence_ model. 175It was trained on 680k hours of labelled speech data annotated using large-scale weak supervision. 176 177The models were trained on either English-only data or multilingual data. The English-only models were trained 178on the task of speech recognition. The multilingual models were trained on both speech recognition and speech 179translation. For speech recognition, the model predicts transcriptions in the *same* language as the audio. 180For speech translation, the model predicts transcriptions to a *different* language to the audio.181 182Whisper checkpoints come in five configurations of varying model sizes.183The smallest four are trained on either English-only or multilingual data.184The largest checkpoints are multilingual only. All ten of the pre-trained checkpoints 185are available on the [Hugging Face Hub](https://huggingface.co/models?search=openai/whisper). The 186checkpoints are summarised in the following table with links to the models on the Hub:187 188| Size     | Parameters | English-only                                         | Multilingual                                        |189|----------|------------|------------------------------------------------------|-----------------------------------------------------|190| tiny     | 39 M       | [✓](https://huggingface.co/openai/whisper-tiny.en)   | [✓](https://huggingface.co/openai/whisper-tiny)     |191| base     | 74 M       | [✓](https://huggingface.co/openai/whisper-base.en)   | [✓](https://huggingface.co/openai/whisper-base)     |192| small    | 244 M      | [✓](https://huggingface.co/openai/whisper-small.en)  | [✓](https://huggingface.co/openai/whisper-small)    |193| medium   | 769 M      | [✓](https://huggingface.co/openai/whisper-medium.en) | [✓](https://huggingface.co/openai/whisper-medium)   |194| large    | 1550 M     | x                                                    | [✓](https://huggingface.co/openai/whisper-large)    |195| large-v2 | 1550 M     | x                                                    | [✓](https://huggingface.co/openai/whisper-large-v2) |196 197# Usage198 199To transcribe audio samples, the model has to be used alongside a [`WhisperProcessor`](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperProcessor).200 201The `WhisperProcessor` is used to:2021. Pre-process the audio inputs (converting them to log-Mel spectrograms for the model)2032. Post-process the model outputs (converting them from tokens to text)204 205The model is informed of which task to perform (transcription or translation) by passing the appropriate "context tokens". These context tokens 206are a sequence of tokens that are given to the decoder at the start of the decoding process, and take the following order:2071. The transcription always starts with the `<|startoftranscript|>` token2082. The second token is the language token (e.g. `<|en|>` for English)2093. The third token is the "task token". It can take one of two values: `<|transcribe|>` for speech recognition or `<|translate|>` for speech translation2104. In addition, a `<|notimestamps|>` token is added if the model should not include timestamp prediction211 212Thus, a typical sequence of context tokens might look as follows:213```214<|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|>215```216Which tells the model to decode in English, under the task of speech recognition, and not to predict timestamps.217 218These tokens can either be forced or un-forced. If they are forced, the model is made to predict each token at 219each position. This allows one to control the output language and task for the Whisper model. If they are un-forced, 220the Whisper model will automatically predict the output langauge and task itself.221 222The context tokens can be set accordingly:223 224```python225model.config.forced_decoder_ids = WhisperProcessor.get_decoder_prompt_ids(language="english", task="transcribe")226```227 228Which forces the model to predict in English under the task of speech recognition.229 230## Transcription231 232### English to English 233In this example, the context tokens are 'unforced', meaning the model automatically predicts the output language234(English) and task (transcribe).235 236```python237>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration238>>> from datasets import load_dataset239 240>>> # load model and processor241>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")242>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")243>>> model.config.forced_decoder_ids = None244 245>>> # load dummy dataset and read audio files246>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")247>>> sample = ds[0]["audio"]248>>> input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features 249 250>>> # generate token ids251>>> predicted_ids = model.generate(input_features)252>>> # decode token ids to text253>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)254['<|startoftranscript|><|en|><|transcribe|><|notimestamps|> Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.<|endoftext|>']255 256>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)257[' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.']258```259The context tokens can be removed from the start of the transcription by setting `skip_special_tokens=True`.260 261### French to French 262The following example demonstrates French to French transcription by setting the decoder ids appropriately. 263 264```python265>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration266>>> from datasets import Audio, load_dataset267 268>>> # load model and processor269>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")270>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")271>>> forced_decoder_ids = processor.get_decoder_prompt_ids(language="french", task="transcribe")272 273>>> # load streaming dataset and read first audio sample274>>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)275>>> ds = ds.cast_column("audio", Audio(sampling_rate=16_000))276>>> input_speech = next(iter(ds))["audio"]277>>> input_features = processor(input_speech["array"], sampling_rate=input_speech["sampling_rate"], return_tensors="pt").input_features278 279>>> # generate token ids280>>> predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids)281>>> # decode token ids to text282>>> transcription = processor.batch_decode(predicted_ids)283['<|startoftranscript|><|fr|><|transcribe|><|notimestamps|> Un vrai travail intéressant va enfin être mené sur ce sujet.<|endoftext|>']284 285>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)286[' Un vrai travail intéressant va enfin être mené sur ce sujet.']287```288 289## Translation 290Setting the task to "translate" forces the Whisper model to perform speech translation.291 292### French to English293 294```python295>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration296>>> from datasets import Audio, load_dataset297 298>>> # load model and processor299>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")300>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")301>>> forced_decoder_ids = processor.get_decoder_prompt_ids(language="french", task="translate")302 303>>> # load streaming dataset and read first audio sample304>>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)305>>> ds = ds.cast_column("audio", Audio(sampling_rate=16_000))306>>> input_speech = next(iter(ds))["audio"]307>>> input_features = processor(input_speech["array"], sampling_rate=input_speech["sampling_rate"], return_tensors="pt").input_features308 309>>> # generate token ids310>>> predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids)311>>> # decode token ids to text312>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)313[' A very interesting work, we will finally be given on this subject.']314```315 316## Evaluation317 318This code snippet shows how to evaluate Whisper Tiny on [LibriSpeech test-clean](https://huggingface.co/datasets/librispeech_asr):319 320```python321>>> from datasets import load_dataset322>>> from transformers import WhisperForConditionalGeneration, WhisperProcessor323>>> import torch324>>> from evaluate import load325 326>>> librispeech_test_clean = load_dataset("librispeech_asr", "clean", split="test")327 328>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")329>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny").to("cuda")330 331>>> def map_to_pred(batch):332>>>     audio = batch["audio"]333>>>     input_features = processor(audio["array"], sampling_rate=audio["sampling_rate"], return_tensors="pt").input_features334>>>     batch["reference"] = processor.tokenizer._normalize(batch['text'])335>>> 336>>>     with torch.no_grad():337>>>         predicted_ids = model.generate(input_features.to("cuda"))[0]338>>>     transcription = processor.decode(predicted_ids)339>>>     batch["prediction"] = processor.tokenizer._normalize(transcription)340>>>     return batch341 342>>> result = librispeech_test_clean.map(map_to_pred)343 344>>> wer = load("wer")345>>> print(100 * wer.compute(references=result["reference"], predictions=result["prediction"]))3467.547098647858638347```348 349## Long-Form Transcription350 351The Whisper model is intrinsically designed to work on audio samples of up to 30s in duration. However, by using a chunking 352algorithm, it can be used to transcribe audio samples of up to arbitrary length. This is possible through Transformers 353[`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) 354method. Chunking is enabled by setting `chunk_length_s=30` when instantiating the pipeline. It can also be extended to 355predict utterance level timestamps by passing `return_timestamps=True`:356 357```python358>>> import torch359>>> from transformers import pipeline360>>> from datasets import load_dataset361 362>>> device = "cuda:0" if torch.cuda.is_available() else "cpu"363 364>>> pipe = pipeline(365>>>   "automatic-speech-recognition",366>>>   model="openai/whisper-tiny",367>>>   chunk_length_s=30,368>>>   device=device,369>>> )370 371>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")372>>> sample = ds[0]["audio"]373 374>>> prediction = pipe(sample.copy())["text"]375" Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel."376 377>>> # we can also return timestamps for the predictions378>>> prediction = pipe(sample, return_timestamps=True)["chunks"]379[{'text': ' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.',380  'timestamp': (0.0, 5.44)}]381```382 383## Fine-Tuning384 385The pre-trained Whisper model demonstrates a strong ability to generalise to different datasets and domains. However, 386its predictive capabilities can be improved further for certain languages and tasks through *fine-tuning*. The blog 387post [Fine-Tune Whisper with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper) provides a step-by-step 388guide to fine-tuning the Whisper model with as little as 5 hours of labelled data.389 390### Evaluated Use391 392The primary intended users of these models are AI researchers studying robustness, generalization, capabilities, biases, and constraints of the current model. However, Whisper is also potentially quite useful as an ASR solution for developers, especially for English speech recognition. We recognize that once models are released, it is impossible to restrict access to only “intended” uses or to draw reasonable guidelines around what is or is not research.393 394The models are primarily trained and evaluated on ASR and speech translation to English tasks. They show strong ASR results in ~10 languages. They may exhibit additional capabilities, particularly if fine-tuned on certain tasks like voice activity detection, speaker classification, or speaker diarization but have not been robustly evaluated in these areas. We strongly recommend that users perform robust evaluations of the models in a particular context and domain before deploying them.395 396In particular, we caution against using Whisper models to transcribe recordings of individuals taken without their consent or purporting to use these models for any kind of subjective classification. We recommend against use in high-risk domains like decision-making contexts, where flaws in accuracy can lead to pronounced flaws in outcomes. The models are intended to transcribe and translate speech, use of the model for classification is not only not evaluated but also not appropriate, particularly to infer human attributes.397 398 399## Training Data400 401The models are trained on 680,000 hours of audio and the corresponding transcripts collected from the internet. 65% of this data (or 438,000 hours) represents English-language audio and matched English transcripts, roughly 18% (or 126,000 hours) represents non-English audio and English transcripts, while the final 17% (or 117,000 hours) represents non-English audio and the corresponding transcript. This non-English data represents 98 different languages. 402 403As discussed in [the accompanying paper](https://cdn.openai.com/papers/whisper.pdf), we see that performance on transcription in a given language is directly correlated with the amount of training data we employ in that language.404 405 406## Performance and Limitations407 408Our studies show that, over many existing ASR systems, the models exhibit improved robustness to accents, background noise, technical language, as well as zero shot translation from multiple languages into English; and that accuracy on speech recognition and translation is near the state-of-the-art level. 409 410However, because the models are trained in a weakly supervised manner using large-scale noisy data, the predictions may include texts that are not actually spoken in the audio input (i.e. hallucination). We hypothesize that this happens because, given their general knowledge of language, the models combine trying to predict the next word in audio with trying to transcribe the audio itself.411 412Our models perform unevenly across languages, and we observe lower accuracy on low-resource and/or low-discoverability languages or languages where we have less training data. The models also exhibit disparate performance on different accents and dialects of particular languages, which may include higher word error rate across speakers of different genders, races, ages, or other demographic criteria. Our full evaluation results are presented in [the paper accompanying this release](https://cdn.openai.com/papers/whisper.pdf). 413 414In addition, the sequence-to-sequence architecture of the model makes it prone to generating repetitive texts, which can be mitigated to some degree by beam search and temperature scheduling but not perfectly. Further analysis on these limitations are provided in [the paper](https://cdn.openai.com/papers/whisper.pdf). It is likely that this behavior and hallucinations may be worse on lower-resource and/or lower-discoverability languages.415 416 417## Broader Implications418 419We anticipate that Whisper models’ transcription capabilities may be used for improving accessibility tools. While Whisper models cannot be used for real-time transcription out of the box – their speed and size suggest that others may be able to build applications on top of them that allow for near-real-time speech recognition and translation. The real value of beneficial applications built on top of Whisper models suggests that the disparate performance of these models may have real economic implications.420 421There are also potential dual use concerns that come with releasing Whisper. While we hope the technology will be used primarily for beneficial purposes, making ASR technology more accessible could enable more actors to build capable surveillance technologies or scale up existing surveillance efforts, as the speed and accuracy allow for affordable automatic transcription and translation of large volumes of audio communication. Moreover, these models may have some capabilities to recognize specific individuals out of the box, which in turn presents safety concerns related both to dual use and disparate performance. In practice, we expect that the cost of transcription is not the limiting factor of scaling up surveillance projects.422 423 424### BibTeX entry and citation info425```bibtex426@misc{radford2022whisper,427  doi = {10.48550/ARXIV.2212.04356},428  url = {https://arxiv.org/abs/2212.04356},429  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},430  title = {Robust Speech Recognition via Large-Scale Weak Supervision},431  publisher = {arXiv},432  year = {2022},433  copyright = {arXiv.org perpetual, non-exclusive license}434}435```436