panjiariputra/multilingual-xls_r_300m-LARGE-5gram
<div align="center"> <h1>XLS-R Deep Learning Model for Multilingual ASR</h1> <p>A fine-tuned model for Automatic Speech Recognition on low-resource Southeast Asian languages: Indonesian, Javanese, and Sundanese.</p> </div>
<p align="center"> <a href="#how-to-use"><strong>How to Use</strong></a> · <a href="#evaluation"><strong>Evaluation Results</strong></a> · <a href="#citation"><strong>Citation</strong></a> · <a href="https://huggingface.co/spaces/panjiariputra/indonesian-java-sunda-speech-recognition" target="blank"><strong>Try on Spaces</strong></a> · <a href="http://www.icicelb.org/ellb/contents/2024/6/elb-15-06-02.pdf" target="blank"><strong>Read the Paper</strong></a> </p>
<br>
This is a fine-tuned version of the <strong><a href="https://huggingface.co/facebook/wav2vec2-xls-r-300m">facebook/wav2vec2-xls-r-300m</a></strong> model for Automatic Speech Recognition (ASR). It is the official model from the research paper "XLS-R DEEP LEARNING MODEL FOR MULTILINGUAL ASR ON LOW-RESOURCE LANGUAGES: INDONESIAN, JAVANESE, AND SUNDANESE".
The goal of this research is to improve ASR performance in converting spoken language into written text for Indonesian, Javanese, and Sundanese. The model's accuracy is significantly enhanced by integrating a 5-gram KenLM language model, which substantially reduces the Word Error Rate (WER).
<p align="center"> <img alt="Proposed Methodology" src="https://cdn-uploads.huggingface.co/production/uploads/6185e35858cb1f8c362f943c/_RN6E1iWfUgrKIrEPgKid.jpeg" width="800"> <br> <em>Proposed Methodology</em> </p>
<h3>Model Details</h3> <ul> <li><strong>Base Model:</strong> <code>facebook/wav2vec2-xls-r-300m</code> (a version with 300 million parameters was used).</li> <li><strong>Task:</strong> Multilingual Automatic Speech Recognition (ASR).</li> <li><strong>Languages:</strong> Indonesian, Javanese, Sundanese.</li> <li><strong>Library:</strong> Transformers.</li> <li><strong>Framework:</strong> Deep Learning, based on the wav2vec 2.0 and Transformer architecture.</li> </ul>
<h3>Authors</h3> <ul> <li>Panji Arisaputra </li> <li>Alif Tri Handoyo </li> <li>Amalia Zahra </li> </ul> <p><em>Computer Science Department, Bina Nusantara University, Jakarta, Indonesia. </em></p>
<h3>Datasets Used for Training</h3> <p> A total of seven datasets were combined for this study to create a robust multilingual corpus. </p> <h4>Speech Data for Fine-Tuning:</h4> <ul> <li><strong>Indonesian:</strong> <ul> <li><strong>TITML-IDN:</strong> A phonetically balanced collection of 343 sentences from 20 speakers, totaling 14.5 hours of audio.</li> <li><strong>Magic Data Corpus:</strong> 3.5 hours of scripted speeches from 10 Indonesian speakers.</li> <li><strong>Common Voice (Indonesian):</strong> Utilized the train, validation, and test subsets, comprising 5,809 instances from 170 individuals.</li> </ul> </li> <li><strong>Javanese & Sundanese:</strong> <ul> <li><strong>OpenSLR SLR35 & SLR36:</strong> Large ASR training datasets with speech recordings from native speakers. Due to computational limits, only the first three .zip files from each dataset were used.</li> <li><strong>OpenSLR SLR41 & SLR44:</strong> High-quality Text-to-Speech (TTS) data used for ASR training.</li> </ul> </li> </ul>
<h4>Text Data for Language Model:</h4> <ul> <li>In addition to the transcripts from the above datasets, the <strong>OSCAR corpus</strong> (unshuffled_deduplicated id subset) was used as an augmentation technique for building the KenLM language model. Only 6% of its 2.3 billion Indonesian words were used to avoid disproportionately influencing the model.</li> </ul>
<h4>Data Preprocessing</h4> <p>The datasets underwent a standardized preprocessing pipeline:</p> <ol> <li><strong>Data Splitting:</strong> Each dataset was split into 90% for training and 10% for testing. The training portion was further subdivided into 90% for the train set and 10% for the validation set.</li> <li><strong>Audio Standardization:</strong> Audio files were converted to WAV format with a single channel and resampled to a 16 kHz sampling rate.</li> <li><strong>Text Normalization:</strong> Transcriptions were cleaned by removing special characters and converting all text to lowercase.</li> </ol>
<h3 id="evaluation">Evaluation and Results</h3> <p> The model was evaluated against a previous model, XLSR-53, using the Word Error Rate (WER) metric. The integration of a <strong>5-gram KenLM language model</strong> demonstrated the best overall performance, achieving a significant reduction in WER and establishing a new benchmark for these languages. </p> <p>The XLS-R 300m model maintains a competitive edge by supporting Javanese and Sundanese in addition to Indonesian, which the previous XLSR-53 model did not.</p>
<table> <thead> <tr> <th rowspan="2">Model</th> <th rowspan="2">Data train & val</th> <th rowspan="2">KenLM</th> <th colspan="8">Data testing</th> </tr> <tr> <th>TITML-IDN</th> <th>Magic-Data</th> <th>Common Voice</th> <th>SLR 35</th> <th>SLR 36</th> <th>SLR 41</th> <th>SLR 44</th> <th>AVG</th> </tr> </thead> <tbody> <tr> <td rowspan="6">XLS-R 300m ASR multi-lingual model</td> <td rowspan="6"> <ul> <li>TITML-IDN</li> <li>Magic Data</li> <li>Common Voice</li> <li>SLR35</li> <li>SLR36</li> <li>SLR41</li> <li>SLR44</li> </ul> </td> <td>—</td> <td>7.73</td> <td>19.64</td> <td>15.30</td> <td>17.95</td> <td>2.39</td> <td>21.99</td> <td>7.10</td> <td>13.16</td> </tr> <tr> <td>2-gram</td> <td>1.79</td> <td>10.93</td> <td>6.55</td> <td>7.76</td> <td>1.20</td> <td>10.90</td> <td>3.58</td> <td>6.10</td> </tr> <tr> <td>3-gram</td> <td>1.39</td> <td><strong>10.38</strong></td> <td>5.63</td> <td>6.50</td> <td>1.15</td> <td>10.41</td> <td>3.47</td> <td>5.56</td> </tr> <tr> <td>4-gram</td> <td>1.37</td> <td><strong>10.38</strong></td> <td>5.11</td> <td><strong>6.38</strong></td> <td>1.15</td> <td>10.31</td> <td>3.47</td> <td>5.45</td> </tr> <tr> <td>5-gram</td> <td>1.37</td> <td><strong>10.38</strong></td> <td><strong>4.99</strong></td> <td>6.41</td> <td><strong>1.14</strong></td> <td><strong>10.25</strong></td> <td><strong>3.44</strong></td> <td><strong>5.43</strong></td> </tr> <tr> <td>6-gram</td> <td>1.37</td> <td><strong>10.38</strong></td> <td>5.01</td> <td>6.41</td> <td><strong>1.14</strong></td> <td>10.35</td> <td><strong>3.44</strong></td> <td>5.44</td> </tr> <tr> <td rowspan="5">XLSR-53 ASR model</td> <td rowspan="5"> <ul> <li>TITML-IDN</li> <li>Magic Data</li> <li>Common Voice</li> </ul> </td> <td>—</td> <td>2.17</td> <td>16.75</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>9.46</td> </tr> <tr> <td>2-gram</td> <td>0.77</td> <td>10.78</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>5.77</td> </tr> <tr> <td>3-gram</td> <td><strong>0.72</strong></td> <td>10.88</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>5.80</td> </tr> <tr> <td>4-gram</td> <td><strong>0.72</strong></td> <td>10.88</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>5.80</td> </tr> <tr> <td>5-gram</td> <td><strong>0.72</strong></td> <td>10.93</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>—</td> <td>5.82</td> </tr> </tbody> </table> <p style="font-size: smaller; text-align: center; margin-top: 5px;">*WER results extracted from Table 4 of the research paper.</p>
<h3 id="how-to-use">How to Use</h3>
<p> You can use this model with the <code>transformers</code> library pipeline. For optimal performance, as demonstrated in the research paper, we strongly recommend integrating the provided <strong>5-gram KenLM language model</strong>. </p>
<pre><code>pip install transformers torch torchaudio librosa
For decoding with the language model:
pip install pyctcdecode==0.4.0 kenlm </code></pre>
<pre><code class="language-python">from transformers import AutoProcessor, AutoModelForCTC, pipeline import torch, librosa
Load processor and model
processor = AutoProcessor.frompretrained("panjiariputra/multilingual-xlsr300m-LARGE-5gram") model = AutoModelForCTC.frompretrained("panjiariputra/multilingual-xlsr300m-LARGE-5gram")
Initialize ASR pipeline
asrpipeline = pipeline( "automatic-speech-recognition", model=model, tokenizer=processor.tokenizer, featureextractor=processor.featureextractor, device=0 if torch.cuda.isavailable() else -1 )
Load an audio file (must be 16kHz, mono)
audiopath = "path/to/your/audio.wav" speecharray, samplingrate = librosa.load(audiopath, sr=16000)
Run transcription
transcription = asrpipeline(speecharray) print(transcription) # {'text': 'your transcribed text here'} </code></pre>
<h4 id="with-language-model">With Language Model Integration (Recommended)</h4>
<p> For the best accuracy and lowest Word Error Rate (WER), use <code>pyctcdecode</code> with the 5-gram KenLM model (<code>5gram.arpa</code>) available in this repository. </p>
<pre><code class="language-python">from pyctcdecode import build_ctcdecoder import torch, librosa
Get vocabulary and build the decoder
vocabdict = processor.tokenizer.getvocab() sortedvocab = {k: v for k, v in sorted(vocabdict.items(), key=lambda item: item[1])}
decoder = buildctcdecoder( labels=list(sortedvocab.keys()), kenlmmodelpath="path/to/your/5gram.arpa" )
Load audio (16kHz, mono)
audiopath = "path/to/your/audio.wav" speecharray, = librosa.load(audiopath, sr=16000)
Get model logits
with torch.nograd(): inputs = processor(speecharray, samplingrate=16000, returntensors="pt", padding=True) logits = model(**inputs).logits.cpu().numpy()[0]
Decode using KenLM
lmtranscription = decoder.decode(logits) print({"text": lmtranscription}) # {'text': 'your more accurate transcribed text here'} </code></pre>
<h3 id="citation">Publication and Citation</h3> <p>This work was published in ICIC Express Letters, Part B: Applications. You can download the full paper <a href="http://www.icicelb.org/ellb/contents/2024/6/elb-15-06-02.pdf" target="_blank">here</a>. If you use this model or the findings from the paper in your research, please cite:</p> <pre><code>@article{Arisaputra2024XLS, author = {Panji Arisaputra and Alif Tri Handoyo and Amalia Zahra}, title = {XLS-R DEEP LEARNING MODEL FOR MULTILINGUAL ASR ON LOW-RESOURCE LANGUAGES: INDONESIAN, JAVANESE, AND SUNDANESE}, journal = {ICIC Express Letters, Part B: Applications}, volume = {15}, number = {6}, pages = {551--559}, year = {2024}, doi = {10.24507/icicelb.15.06.551}, issn = {2185-2766} } </code></pre>