itayinbar/whisper-base-he
whisper-base-he
A 50.2M-parameter Hebrew speech recognition model, built by replacing Whisper's multilingual vocabulary with a Hebrew one and training on 3,113 hours of human-transcribed Hebrew.
It is the most accurate open Hebrew recogniser of its size by a wide margin, and second only to models roughly sixteen times larger.
Results
Word error rate, scored with a harness that reproduces the ivrit.ai Hebrew leaderboard on 40 of 40 published model and dataset pairs.
Against every open Hebrew model we could find, on FLEURS-he:
It runs at 500 to 800 times realtime on an RTX 5070 Laptop: a 47-minute recording transcribes in 3.6 seconds.
Why it is smaller than the model it came from
Whisper's vocabulary holds 51,865 tokens for 99 languages. On 24,225 words of held-out Hebrew speech it spends 3.17 tokens per Hebrew word, against 1.28 for English, and its embedding table is 26.6M of whisper-base's 72.6M parameters, nearly all of it tokens Hebrew never emits.
An 8,192-token Hebrew byte-level BPE brings that to 1.76 tokens per word and the model to 50.23M parameters, 30.8% smaller. Every new embedding row is initialised from the pretrained table, averaging the old rows for the same text, so none start from noise.
Measured on identical data and schedule, that vocabulary is worth 7.5 WER points on its own, not merely the size saving.
Usage
from transformers import AutoModelForSpeechSeq2Seq, AutoTokenizer, AutoFeatureExtractor
from transformers.generation.utils import GenerationMixin
model = AutoModelForSpeechSeq2Seq.from_pretrained("itayinbar/whisper-base-he")
tokenizer = AutoTokenizer.from_pretrained("itayinbar/whisper-base-he")
features = AutoFeatureExtractor.from_pretrained("itayinbar/whisper-base-he")
inputs = features(audio_16khz, sampling_rate=16000,
return_tensors="pt", padding="max_length")
# Monolingual: no language token, no task token, no timestamps. Whisper's own
# generate() expects all three, so call the generic implementation.
ids = GenerationMixin.generate(model, **inputs, max_new_tokens=200)
print(tokenizer.decode(ids[0], skip_special_tokens=True))Three things differ from stock Whisper:
- Call the generic `generate`. This model has no language or task tokens.
- Pad mel features to the full 30-second window, as Whisper always requires.
- Cut audio longer than 30 seconds into overlapping windows and stitch the results. Without that,
eval-d1scores 91.94% instead of 14.87%, because everything past the first thirty seconds is scored as deleted.
ONNX weights are included for transformers.js. A browser downloads 160.2 MB (fp16 encoder plus fp32 decoder), against 563 MB for the large Hebrew model it is meant to replace.
Only those two weight files are shipped. An fp16 decoder and an int8 encoder were built and then removed: the fp16 decoder fails to load in ONNX Runtime, and the int8 encoder loads but changes the transcript. Every combination offered here was checked by transcribing with it and comparing against the fp32 output, not by loading it.
Training
3,112.7 hours over 552,327 utterances, all human-transcribed.
Schedule-free AdamW, learning rate 1e-4, batch 16, bf16, 103,551 steps on a single RTX 5070 Laptop GPU with 8 GB. Corpora are sampled by target share of audio time rather than of utterances, since Knesset ships 30-second windows and crowd-transcribe averages 5.2 seconds.
What was tried and did not work
Measured on identical data and budget, so the comparisons are like for like:
SpecAugment and hybrid CTC are standard practice in speech recognition, and neither did anything here.
Licence and provenance
Weights are Apache-2.0, following openai/whisper-base.
Training data from ivrit.ai under the ivrit.ai licence, which permits training models including commercially and requires attribution. Credit for that data belongs to ivrit.ai, whose own Hebrew models are the state of the art this one is measured against. FLEURS is CC-BY-4.0.
imvladikon/hebrew_speech_kan declares no licence on the Hub. It contributed 6.9 of 3,112.7 hours and roughly 4% of the audio the model heard after weighting. It is named here so anyone relying on this model's provenance can judge that themselves.
Code, evaluation harness and full method: github.com/itayinbarr/Hebrew-small-asr.