keystats/w2v-bert-2.0-multilingual-waxal-shona-lingala-luganda-lusoga

w2v-bert-2.0-multilingual-waxal-shona-lingala-luganda-lusoga

A single multilingual automatic speech recognition (ASR) model covering Shona, Lingala, Luganda, and Lusoga, fine-tuned from facebook/w2v-bert-2.0 on the WAXAL corpus only, with all four languages pooled together into one shared model rather than trained as separate per-language checkpoints.

Model description

facebook/w2v-bert-2.0 — a large-scale, multilingual self-supervised speech encoder pretrained with a BERT-style masked prediction objective — is used as the backbone, with a from-scratch character-level CTC (Connectionist Temporal Classification) head fine-tuned across the four languages jointly. No language ID input is required at inference — the model transcribes directly from audio regardless of which of the four languages it is.

Text casing note: this model's vocabulary was built with NFKC-only normalization — case, punctuation, and diacritics from the source transcriptions are preserved as-is. This matches how the WAXAL competition (Zindi) actually scores submissions: raw, unnormalized WER/CER, where case and punctuation mismatches count as errors.

No language model: there is no KenLM language model paired with this checkpoint. Decoding is greedy (argmax) only — see How to use below.

Training data

Only WAXAL data was used — no additional community datasets were pooled in, unlike the Luganda-only -main-best models.

Training procedure

• —Base model: facebook/w2v-bert-2.0
• —Architecture: Wav2Vec2BertForCTC, add_adapter=True
• —Processor: Wav2Vec2BertProcessor — SeamlessM4TFeatureExtractor for audio features + a Wav2Vec2CTCTokenizer built from scratch on the combined training + validation transcriptions across all four languages (character-level vocabulary, case and punctuation preserved, NFKC Unicode normalization only, | as the word delimiter, [PAD] doubling as the CTC blank token)
• —Sample rate: 16 kHz mono
• —Hardware: single RTX PRO 6000
• —Epochs: 4 (with early stopping on steps, patience 5, on validation WER)

• —Data filtering: clips whose transcript is too long for CTC to align within the available encoder output length ("CTC-impossible" clips, roughly output_steps < 2 * label_length) are dropped from both train and validation before training
• —Seed: 42 (deterministic — same seed for Python/NumPy/PyTorch/CUDA)

Evaluation results

Evaluation numbers for this checkpoint aren't included here — this run wasn't the strongest of the multilingual attempts, and per-language WER/CER weren't retained. If you need numbers, re-run evaluation against the pooled WAXAL validation split (or, better, per-language splits to see how it performs on each language individually).

How to use

Greedy (argmax) decoding only — there is no matching language model for this checkpoint.

import torch
import librosa
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor

MODEL_ID = "keystats/w2v-bert-2.0-multilingual-waxal-shona-lingala-luganda-lusoga"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

processor = Wav2Vec2BertProcessor.from_pretrained(MODEL_ID)
model = Wav2Vec2BertForCTC.from_pretrained(MODEL_ID).to(DEVICE).eval()

audio_array, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(input_features=inputs.input_features.to(DEVICE)).logits

predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)[0]

print(transcription)  # cased, punctuated text in whichever of the 4 languages was spoken

No language ID needs to be passed in — the model was trained jointly across all four languages and infers directly from the audio.

Intended uses & limitations

• —Intended for transcribing spoken Shona, Lingala, Luganda, or Lusoga audio into cased, punctuated text, using a single shared model rather than four separate checkpoints.
• —As a CTC-based model, it assumes single-speaker, forward-only audio and has no mechanism for overlapping speech from multiple speakers.
• —Trained only on WAXAL data for these four languages — no additional community datasets were pooled in.
• —No language model is available for this checkpoint — output is from greedy decoding only, which typically scores somewhat worse than LM-assisted beam search.
• —This particular run was not the strongest result obtained across training attempts for this multilingual setup — treat it as a baseline multilingual checkpoint rather than a best-in-class one.
• —Sharing capacity across four languages in one model may trade off some per-language accuracy compared to a dedicated single-language checkpoint (e.g. the Luganda-only -main-best models).
• —Raw-text WER/CER (with case and punctuation counted as errors) will read higher than a normalized-text comparison of the same underlying transcription quality — this is expected and matches how the source competition (Zindi/WAXAL) actually scores submissions.

Citation

If you use this model, please cite the training/fine-tuning work and the underlying dataset:

@misc{keystats_wav2vec2bert_multilingual_waxal,
  title={w2v-bert-2.0-multilingual-waxal-shona-lingala-luganda-lusoga: A multilingual ASR model for Shona, Lingala, Luganda, and Lusoga fine-tuned from facebook/w2v-bert-2.0 on raw text},
  author={keystats},
  year={2026},
  howpublished={\url{https://huggingface.co/keystats/w2v-bert-2.0-multilingual-waxal-shona-lingala-luganda-lusoga}}
}

@misc{waxal,
  title={WAXAL: A Multilingual African Speech Dataset},
  author={Google},
  howpublished={\url{https://huggingface.co/datasets/google/WaxalNLP}}
}

@inproceedings{w2vbert2,
  title={Seamless: Multilingual Expressive and Streaming Speech Translation},
  author={Seamless Communication and others},
  year={2023},
  howpublished={\url{https://huggingface.co/facebook/w2v-bert-2.0}}
}