RafatK/Whisper_Largev2-Swahili-Decodis_Comb_FT

<p align="left"> <a href="https://decodis.com/"> <img src="https://static.wixstatic.com/media/41bde8fdfad2782d8641edb098e72f1ea10d65~mv2.png/v1/fill/w185,h50,alc,q85,usm0.661.000.01,encavif,qualityauto/41bde8fdfad2782d8641edb098e72f1ea10d65~mv2.png" style="display: inline-block; vertical-align: middle;" alt="DECODISWebsite" /> </a> </p>

🎤 General-Purpose Swahili ASR Model (Open Datasets + Domain Data)

<p>

<a href="https://github.com/Rafat-decodis/Robust-ASR-for-Low-Resource-Languages/tree/main" target="blank" style="margin: 2px;"> <img src="https://img.shields.io/badge/Decodis-Indepth Analysis-536af5?color=536af5&logo=github" style="display: inline-block; vertical-align: middle;" alt="Main code" /> </a> <a href="https://github.com/Rafat-decodis/Robust-ASR-for-Low-Resource-Languages/blob/main/LICENSE.txt" target="blank" style="margin: 2px;"> <img src="https://img.shields.io/badge/Decodis-LICENSE-D3E9D4?color=D3E9D4&logo=github" style="display: inline-block; vertical-align: middle;" alt="Main code" /> </a>

</p>

This automatic speech recognition (ASR) model is trained using open multilingual datasets and a multi-domain in-house dataset to provide high-accuracy transcription for clean, read-aloud Swahili speech.

It achieves strong generalization, maintaining benchmark accuracy while improving performance on real-world test data. The model can do well for both clean and noisy audios.

<!-- Model is Finetuned by DECODIS --> This model is part of a full ASR ablation study that analyzes and understands the robustness of data and in dealing with different modes and variations of data collections. 👉 View all models on GitHub

We are particularly interested in validating the conclusions we’ve observed through our ablation studies:

While benchmark datasets like FLEURS are useful for comparison, they do not fully capture the variability and challenges of real-world speech — especially for underrepresented languages like Swahili. We are inviting the community to try out these models and help assess:

1. 1.How well the models perform on natural, conversational, or noisy audio
2. 2.Open-source datasets (like Common Voice & FLEURS) perform well on clean, benchmark speech.
3. 3.Whether the improvements we've seen in combining diverse datasets generalize to your use case
4. 4.Gaps between benchmark results and real-world usability
5. 5.A combination of both yields balanced results but depends on data quality and label accuracy.

Model

Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.

🚀 How to Use

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from transformers import pipeline
from transformers.utils import is_flash_attn_2_available
import torch

processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("RafatK/Swahili-Whisper_Largev2-Decodis_Comb_FT", torch_dtype=torch.float16).to("cuda")
# model.generation_config.input_ids = model.generation_config.forced_decoder_ids
model.generation_config.forced_decoder_ids = None
forced_decoder_ids = processor.get_decoder_prompt_ids(language="swahili", task="transcribe")

pipe = pipeline(
  "automatic-speech-recognition",
  model=model,
  tokenizer=processor.tokenizer, # Explicitly pass the tokenizer
  feature_extractor=processor.feature_extractor, # Explicitly pass the feature extractor
  chunk_length_s=15,
  device=0,
  model_kwargs={"attn_implementation": "flash_attention_2"} if is_flash_attn_2_available() else {"attn_implementation": "sdpa"},
  generate_kwargs = {
                       'num_beams':5,
                       'max_new_tokens':440,
                       'early_stopping':True,
                        'repetition_penalty': 1.8,
                       'language': 'swahili',
                       'task': 'transcribe'
                       }
)
text_output = pipe("audio.wav")['text']

<!--

📦 Training Data

• —Common Voice 11.0 Crowdsourced dataset with validated Swahili recordings (~350 hours)
• —FLEURS Google’s multilingual dataset with 102 languages including Swahili (~50 hours)
• —OpenSLR African Languages in the Field: speech Fundamentals and Automation
• —DPP and DP data (Description) --> 📊 Total Duration: ~400 hours

📁 Languages: Swahili (sw)

🏋️‍♂️ Training Strategy

• —Architecture: whisper-large-v2
• —Framework: Whisper and Huggingface Transformers
• —Sampling rate: 16 kHz
• —Preprocessing: Volume normalization, High-Grade noise addition, Prosodic Augmentation, silence trimming
• —Learning Rate: 1e-5
• —Optimizer: Adamw_pytorch
• —Steps: 3000
• —Pretrained on open data
• —Fine-tuned on domain data

📈 Evaluation Metric (WER)

🎯 Intended Use

• —General-purpose transcription systems
• —Balanced performance on clean and noisy data
• —Speech interfaces in multilingual and informal settings

⚠️ Limitations

• —Slight trade-off in benchmark precision
• —May need more domain data for extreme acoustic variation

📝 Please try the models and share your feedback, issues, or results via:

GitHub Issues: Submit an issue

Hugging Face Discussions: Join the conversation

Your feedback will help us refine our dataset and improve ASR for underrepresented languages like Swahili and Yoruba.