panjiariputra/indonesian-xlsr_53-LARGE-4gram

<div align="center"> <h1>Indonesian Automatic Speech Recognition with XLSR-53</h1> <p>A fine-tuned model for Automatic Speech Recognition (ASR) in Indonesian, achieving competitive performance with a significantly reduced Word Error Rate (WER) using a KenLM language model.</p> </div>

<p align="center"> <a href="#how-to-use"><strong>How to Use</strong></a> · <a href="#evaluation-and-results"><strong>Evaluation Results</strong></a> · <a href="#citation"><strong>Citation</strong></a> · <a href="https://huggingface.co/spaces/panjiariputra/indonesian-speech-recognition" target="blank"><strong>Try on Spaces</strong></a> · <a href="https://doi.org/10.18280/isi.270614" target="blank"><strong>Read the Paper</strong></a> </p>

<br>

This repository contains the official fine-tuned model from the research paper "Indonesian Automatic Speech Recognition with XLSR-53". The study focuses on developing a robust Indonesian ASR system by fine-tuning the pre-trained cross-lingual XLSR-53 (<strong><a href="https://huggingface.co/facebook/wav2vec2-large-xlsr-53">facebook/wav2vec2-large-xlsr-53</a></strong>) model.

The key contribution of this work is demonstrating that a competitive Word Error Rate (WER) can be achieved with a relatively small dataset (24 hours). The model's accuracy is significantly boosted by integrating a 4-gram KenLM language model, which successfully reduces the WER from 20% to 12% on the Common Voice test set.

<p align="center"> <img alt="Proposed Methodology" src="https://cdn-uploads.huggingface.co/production/uploads/6185e35858cb1f8c362f943c/iiGX4hokk2SD1yySLiFWd.png" width="800"> <br> <em>Proposed Methodology </em> </p>

Model Details

<ul> <li><strong>Base Model:</strong> This model is built upon the wav2vec 2.0 architecture, specifically the <strong>XLSR-53</strong> pre-trained model (<code>facebook/wav2vec2-large-xlsr-53</code>) which was trained on 53 languages.</li> <li><strong>Task:</strong> Automatic Speech Recognition (ASR).</li> <li><strong>Language:</strong> Indonesian (id).</li> <li><strong>Library:</strong> Transformers.</li> <li><strong>Framework:</strong> The approach involves fine-tuning the pre-trained model using a Connectionist Temporal Classification (CTC) loss function.</li> </ul>

Authors

<ul> <li>Panji Arisaputra</li> <li>Amalia Zahra</li> </ul> <p><em>Computer Science Department, BINUS Graduate Program, Bina Nusantara University, Jakarta, Indonesia.</em></p>

Datasets Used for Training

<p> A total of three speech datasets were combined to fine-tune the model, and an additional large text corpus was used to build the language model. </p> <h4>Speech Data for Fine-Tuning:</h4> <p>The total combined duration of speech data is <strong>24 hours, 18 minutes, and 1 second</strong>.</p> <ul> <li><strong>TITML-IDN:</strong> A clean speech corpus containing <strong>14.5 hours</strong> of audio from 20 speakers reading phonetically balanced sentences.</li> <li><strong>Magic Data:</strong> A <strong>3.5-hour</strong> corpus of scripted daily-use sentences from 10 speakers, recorded in various environments.</li> <li><strong>Common Voice (Indonesian):</strong> A crowdsourced dataset containing <strong>~6.2 hours</strong> of speech from 170 speakers in diverse, non-clean environments.</li> </ul>

<h4>Text Data for Language Model:</h4> <ul> <li>In addition to the transcripts from the speech datasets, the <strong>OSCAR corpus</strong> (unshuffleddeduplicatedid subset) was used to build the KenLM language model. To ensure a balanced vocabulary, only 6% of its 2.3 billion Indonesian words were included.</li> </ul>

<h4>Data Preprocessing</h4> <p>The datasets underwent a standardized preprocessing pipeline:</p> <ol> <li><strong>Data Splitting:</strong> Datasets were split into training (90%) and validation (10%) sets.</li> <li><strong>Audio Standardization:</strong> All audio files were converted to WAV format with a single channel and resampled to a 16 kHz sampling rate to match the pre-trained model's requirements.</li> <li><strong>Text Normalization:</strong> Transcriptions were cleaned by removing special characters and converting all text to lowercase to create a unified vocabulary.</li> </ol>

<h3 id="evaluation-and-results">Evaluation and Results</h3> <p> The model was evaluated against a similar model from a previous study by Syahputra & Zahra (2021), using the <strong>Word Error Rate (WER)</strong> metric. The evaluation on the Common Voice test split serves as the primary benchmark. </p> <p> The results show that this XLSR-53 model outperforms the previous wav2vec 2.0-based model. The integration of a <strong>4-gram KenLM language model</strong> was crucial, providing an <strong>8% absolute reduction in WER</strong> (from 20% down to 12%). </p>

<table> <thead> <tr> <th>Model</th> <th>Data Training & Validation</th> <th>Language Model</th> <th>Test Set</th> <th>WER (%)</th> </tr> </thead> <tbody> <tr> <td rowspan="2"><b>This Study (XLSR-53)</b></td> <td rowspan="2">TITML-IDN + Magic Data + Common Voice (24h 18m)</td> <td>—</td> <td>Common Voice</td> <td>20.306%</td> </tr> <tr> <td><b>4-gram KenLM</b></td> <td>Common Voice</td> <td><b>12,213%</b></td> </tr> <tr> <td rowspan="2">Benchmark (Syahputra & Zahra, 2021)</td> <td rowspan="2">BahasaKita batch 10 – 12 (75h)</td> <td>—</td> <td>Common Voice</td> <td>21.000%</td> </tr> <tr> <td>3-gram KenLM</td> <td>Common Voice</td> <td>41.000%</td> </tr> </tbody> </table> <p style="font-size: smaller; text-align: center; margin-top: 5px;">*WER results extracted from Table 3 of the research paper. The benchmark model's high WER with LM is noted in the paper.</p>

<h3 id="how-to-use">How to Use</h3>

<p> You can use this model with the <code>transformers</code> library pipeline. For optimal performance, as demonstrated in the research paper, we strongly recommend integrating the provided <strong>4-gram KenLM language model</strong>. </p>

<pre><code>pip install transformers torch torchaudio librosa

For decoding with the language model:

pip install pyctcdecode==0.4.0 kenlm </code></pre>

<h4>Without Language Model</h4> <pre><code class="language-python">from transformers import AutoProcessor, AutoModelForCTC, pipeline import torch import librosa

\# Load processor and model processor = AutoProcessor.frompretrained("panjiarisaputra/indonesian-asr-xlsr-53") model = AutoModelForCTC.frompretrained("panjiarisaputra/indonesian-asr-xlsr-53")

\# Initialize ASR pipeline asrpipeline = pipeline( "automatic-speech-recognition", model=model, tokenizer=processor.tokenizer, featureextractor=processor.featureextractor, device=0 if torch.cuda.isavailable() else -1 )

\# Load an audio file (must be 16kHz, mono) audiopath = "path/to/your/audio.wav" speecharray, samplingrate = librosa.load(audiopath, sr=16000)

\# Run transcription transcription = asrpipeline(speecharray) print(transcription) \# Output: {'text': '...transcribed text...'} </code></pre>

<h4 id="with-language-model">With Language Model Integration (Recommended)</h4>

<p> For the best accuracy and lowest Word Error Rate (WER), use <code>pyctcdecode</code> with the 4-gram KenLM model (e.g., <code>4gram.arpa</code>) created during the research. </p>

<pre><code class="language-python">from transformers import AutoProcessor, AutoModelForCTC from pyctcdecode import build_ctcdecoder import torch import librosa

Load processor and model

processor = AutoProcessor.frompretrained("panjiariputra/indonesian-xlsr53-LARGE-4gram") model = AutoModelForCTC.frompretrained("panjiariputra/indonesian-xlsr53-LARGE-4gram")

Get vocabulary and build the decoder with the language model

vocabdict = processor.tokenizer.getvocab() sortedvocabdict = {k: v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

decoder = buildctcdecoder( labels=list(sortedvocabdict.keys()), kenlmmodel_path="path/to/your/4gram.arpa" # Path to your KenLM model )

Load audio (16kHz, mono)

audiopath = "path/to/your/audio.wav" speecharray, = librosa.load(audiopath, sr=16000)

Get model logits

with torch.nograd(): inputs = processor(speecharray, samplingrate=16000, returntensors="pt", padding=True) logits = model(**inputs).logits.cpu().numpy()[0]

Decode using KenLM

lmtranscription = decoder.decode(logits) print({"text": lmtranscription})

Output: {'text': '...more accurate transcribed text...'}

</code></pre>

<h3 id="citation">Publication and Citation</h3> <p>This work was published in <strong>Ingénierie des Systèmes d'Information, Vol. 27, No. 6, December, 2022</strong>. You can download the full paper <a href="https://doi.org/10.18280/isi.270614" target="_blank">here</a>. If you use this model or the findings from the paper in your research, please cite:</p> <pre><code>@article{Arisaputra2022XLSR53, author = {Panji Arisaputra and Amalia Zahra}, title = {Indonesian Automatic Speech Recognition with XLSR-53}, journal = {Ingénierie des Systèmes d'Information}, volume = {27}, number = {6}, pages = {973--982}, year = {2022}, doi = {10.18280/isi.270614} } </code></pre>