IbrahimDayax/xls-r-1b-somali-stt-asr

Somali STT/ASR — Wav2Vec2 XLS-R 1B

<p align="center"> <b>WER 6.83%&nbsp;&nbsp;·&nbsp;&nbsp;CER 1.42%</b><br> _{normalized (lowercased, punctuation-stripped) text · 781-row verified-clean, held-out test split}<br> _{the lowest error rate of every public Somali ASR model we could find and benchmark — see § Comparison below} </p>

A fine-tune of facebook/wav2vec2-xls-r-1b for Somali speech-to-text. This is an ASR model only — it does not do text-to-speech, and no TTS claim is made anywhere in this card.

2026-07-18: benchmarked against every other public Somali ASR model on the Hugging Face Hub we could locate and load, on the same held-out test set. See § Comparison to other public Somali ASR models. Training data/config is unchanged from the original release — this update is evaluation-only. 2026-07-19: added two more models to that comparison — openai/whisper-large-v3 (a general multilingual foundation model, not Somali-finetuned; Somali is one of its officially supported languages, but it still underperforms every Somali-specific model here) and Qwen/Qwen3-ASR-1.7B-hf (Somali is not one of its supported languages — included as an out-of-distribution check, not a fair comparison). Neither addition changes the ranking or the margin over the field; see below.

Model details

Training configuration

Full fine-tune (not LoRA/adapter), single stage, on an A100/H100-class GPU:

The tokenizer was deliberately borrowed from Mustafaa4a/ASR-Somali rather than built fresh from the training corpus — see § Known limitations for the consequence (an unused en-dash token, silently dropped ASCII hyphens).

Comparison to other public Somali ASR models

We searched the Hugging Face Hub for every model tagged or named for Somali ASR, excluded repos with no usable model weights (ooloteam10/wav2vec2-somali, ooloteam10/somali_wav2vec2, Mustafaa4a/ASR-Somali-v2 — all empty), and benchmarked the rest on the same 781-row verified-clean test split from § Evaluation, using the same text normalization (lowercased, punctuation stripped) for every model, so the numbers below are directly comparable.

~5.3x lower WER than the next-best model we could benchmark (Mustafaa4a/ASR-Somali, 35.9%) — not a marginal edge.

Notes on the table, in the interest of not overstating this:

• —This comparison isn't perfectly symmetric. This model's training data includes the train split of somali-combined-asr-stt-dataset (the test split scored here was held out and never seen — see § Training configuration and § Evaluation) — so it has seen the same source distribution the test split is drawn from, an advantage none of the other models in this table have, since none of them were trained on any part of this dataset. Worth weighing when reading the margin below.
• —`steja/whisper-large-somali` could not be benchmarked — it failed to load with the tokenizers library version in our environment (argument 'special_tokens': Expected Union[Tuple[str, int], Tuple[int, str], dict]), an incompatibility with an older repo format, not a measurement of model quality. It is the one reasonably prominent public Somali ASR checkpoint missing from this table.
• —laalays/mms1b-finetuned-somali's 100% WER was verified by hand — it produces incoherent, repetitive character strings (e.g. qayb → qisibssn) under standard loading, not a scoring artifact.
• —`openai/whisper-large-v3` is the fairest general-purpose reference point in this table: it wasn't fine-tuned for Somali at all, so its 92.7% WER is a read on how far a strong multilingual foundation model — at ~3.2x this model's parameter count — gets on Somali zero-shot. Not far, and clearly behind every Somali-specific model tested except the two broken/unsupported ones below it.
• —`Qwen/Qwen3-ASR-1.7B-hf`'s 115.5% WER (WER can exceed 100% when the hypothesis has far more insertions than the reference has words) is expected, not a quality signal: Somali is not among its 30 officially supported languages, and in our runs it auto-detected Arabic or Malay and transliterated the audio accordingly rather than transcribing Somali. Included as an out-of-distribution check only.
• —Given that, we call this the strongest publicly available open-source Somali ASR model we identified and were able to benchmark, as of 2026-07-19 — scoped explicitly to what we could find and load, across both Somali-specific models and general multilingual foundation models. It is not a claim that no better model exists anywhere, published or private.
• —This says nothing about Somali TTS: no text-to-speech models were evaluated (this is an ASR-only comparison, and TTS is a different task with different models — e.g. Somali-tts/somali_tts_model, saleolow/somali-mms-tts — that this card makes no claim about).

Evaluation

Full clean test split — IbrahimDayax/somali-combined-asr-stt-dataset (current, primary)

This dataset combines three Somali speech sources (jelle8000, soomali_asr, somali_tts), deduplicated by transcript text. Its train split was part of this model's training data (see § Training configuration) — but its test split, scored below, was held out and never seen during training, so this remains a genuine held-out evaluation and the basis for every number in § Comparison above. The 811-row test split was verified before scoring: each row must have non-empty text, audio that actually decodes, audio ≥0.2s, and a transcript free of two formatting artifacts found in a minority of rows —

• —somali_tts rows with words joined by _ instead of a space (e.g. Nadaafadda_gacmaha_waxay_...), and
• —soomali_asr rows with an embedded |-separated grammatical tag that was never spoken (e.g. Waa neef|m., hal|f — m/f/pl/prn markers).

30 of 811 rows failed verification (16 tag-annotated, 14 underscore-joined) and were excluded. All 781 surviving rows were scored — no subsampling.

WER 6.83% · CER 1.42% (normalized text: lowercased, punctuation stripped — the fair comparison, since this model's 33-token vocabulary has no capitalization or punctuation tokens to begin with).

Per-utterance: 635/781 (81.3%) transcribed perfectly (WER=0); 34/781 (4.4%) at WER>50%. The per-utterance mean (7.3%) sits above the median (0%) — most of the test set is exact, and the mean is pulled up by a long tail of very short utterances (median reference length: 3 words) where a single error is a large percentage.

Comparison to a prior Somali ASR system

This model's tokenizer/vocabulary was bootstrapped from Mustafaa4a/ASR-Somali — so beyond its place in the broader comparison above, it's worth calling out as the direct "did fine-tuning actually help?" baseline. Both were scored on the same 781 clean rows.

81% relative WER reduction (35.9% → 6.83%) over the prior Somali ASR this model's vocabulary came from. The random-head row is included only as a sanity floor — it measures an untrained decoder, not the pretrained encoder, and isn't a competing system.

Evaluation set provenance

The evaluation set (IbrahimDayax/somali-combined-asr-stt-dataset) is gold-standard: its somali_tts rows are gold-standard by construction (the text was the input to speech synthesis, not derived from the audio), and its jelle8000 and soomali_asr rows are human-annotated. The WER/CER above therefore measure correctness against real transcripts, not agreement with another ASR system's bootstrapped output.

This model's training data (IbrahimDayax/somali-asr-synthetic-youtube), by contrast, is silver-standard — produced by ASR bootstrapping from YouTube, not human annotators. That's a property of the training data, unrelated to how this evaluation was scored.

Known limitations

• —No digits in the vocabulary. The model cannot emit 2024; numerals must be spelled out. Training transcriptions contained no digits.
• —Hyphens are dropped. The vocabulary (borrowed from Mustafaa4a/ASR-Somali, not built from this model's own training corpus) contains an en-dash (–, U+2013) that never appears in the training data, while the training data's ASCII hyphens (-, U+002D) were silently stripped from the labels — so hyphenation is dropped and the en-dash token is never produced.
• —Greedy decoding has no language model — and adding one, measured, didn't help at this error rate. Characteristic greedy failures are spelling and word-boundary errors on acoustically ambiguous input (af-Carabi → afcarabi; cabbid → nabbid). Beam-search decoding with a 5-gram KenLM model (pyctcdecode, trained on this model's own ~250k-word training-split transcripts) was measured, not assumed: it moved WER from 6.83% → 7.28% (worse), and after a 16-point alpha/beta sweep tuned on a held-out validation split, the best configuration only reached 8.20% vs. a greedy baseline of 8.31% on that same split — a ~1% relative gain, not the 15–30% sometimes quoted for this technique. At this model's current error rate there is little left for a thin, same-domain n-gram model to fix; a substantially larger and more general external Somali text corpus (news, Wikipedia, books) is the more promising lever than tuning this one further.
• —30-second cap. Trained on clips ≤30s; longer audio should be chunked.
• —`steja/whisper-large-somali` is not in the § Comparison table — it failed to load in our environment (tokenizer format incompatibility), not because it was tested and excluded. Treat the ranking as "best of what loaded," not exhaustive.

Usage

import librosa
import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

model_id = "IbrahimDayax/xls-r-1b-somali-stt-asr"
processor = Wav2Vec2Processor.from_pretrained(model_id)
model = Wav2Vec2ForCTC.from_pretrained(model_id).eval()

speech, _ = librosa.load("somali_audio.wav", sr=16_000)  # 16 kHz mono required

inputs = processor(speech, sampling_rate=16_000, return_tensors="pt")
with torch.no_grad():
    logits = model(inputs.input_values).logits

transcription = processor.batch_decode(torch.argmax(logits, dim=-1))[0]
print(transcription)