Sanjidh090/moonshine-base-bn
Moonshine-Base-BN: Bengali ASR via Tokenizer Transplantation
moonshine-base-bn is a 61.5M-parameter Bengali Automatic Speech Recognition (ASR) model built by adapting UsefulSensors/Moonshine-Base through a novel tokenizer transplantation pipeline. The model's original English-centric byte-level decoder vocabulary was surgically replaced with the native-script BanglaBERT WordPiece vocabulary, resolving the autoregressive collapse that high-fertility byte tokenization causes on morphologically rich languages like Bengali.
This is the official model release accompanying the paper:
Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR Sanjid Hasan, Md. Abdur Rahman — MuslimML Workshop @ ICML 2026
Why Tokenizer Transplantation?
Lightweight ASR models like Moonshine are optimized for fast, offline, edge deployment — but their English-centric tokenizers fragment Bengali words into long byte chains. This high tokenizer fertility (tokens-per-word) causes the autoregressive decoder to drift and collapse during inference, even when teacher-forced training loss looks fine.
By replacing the vocabulary with a native Bengali WordPiece tokenizer and re-aligning the model through a two-stage recovery schedule, decoding instability is fully resolved.
Performance
Evaluated on the held-out test split of the Lipi-Ghor-bn-882-SSTT dataset (882 hours, multi-speaker, multi-domain Bengali speech):
This model achieves the lowest CER among all tested architectures and matches the WER of a model 12x its size, while running natively ~3.5x faster than engineered Whisper CTranslate2 pipelines.
Model Architecture
This model is built by surgically replacing the decoder vocabulary of Moonshine-Base with the native-script BanglaBERT WordPiece vocabulary, then re-aligning the decoder through a recovery fine-tuning schedule. Full methodology is detailed in the paper (Section 4).
Usage
Record an audio in .wav format and replace the path with sample.wav!
# Ensure you have librosa installed
# !pip install -q librosa
import torch
import librosa
import numpy as np
from transformers import AutoTokenizer, AutoModelForSpeechSeq2Seq
# ═══════════════════════════════════════════════════════════════════════════════
# CONFIG & PATHS
# ═══════════════════════════════════════════════════════════════════════════════
REPO_ID = "Sanjidh090/moonshine-base-bn"
AUDIO_PATH = "sample.wav"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32
print(f"📡 Calling Model from Hugging Face: {REPO_ID}")
# 1. Load Tokenizer & Model
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
REPO_ID,
torch_dtype=dtype,
low_cpu_mem_usage=True
).to(device)
# 2. Extract Token IDs (Fallback to your verified defaults if missing)
START_ID = tokenizer.cls_token_id or 2
EOS_ID = tokenizer.sep_token_id or 3
PAD_ID = tokenizer.pad_token_id or 0
model.eval()
# ═══════════════════════════════════════════════════════════════════════════════
# INFERENCE FUNCTION
# ═══════════════════════════════════════════════════════════════════════════════
def transcribe(audio_file):
print("⏳ Processing audio...")
# Load and resample to 16kHz
audio, _ = librosa.load(audio_file, sr=16000)
# Pad to multiple of 160
remainder = len(audio) % 320
if remainder:
audio = np.concatenate([audio, np.zeros(320 - remainder, dtype=np.float32)])
# Cast the tensor to matching global precision
input_values = torch.tensor(audio).unsqueeze(0).to(device, dtype=dtype)
print("🔮 Running inference pipeline...")
with torch.no_grad():
generated_ids = model.generate(
input_values,
max_new_tokens=2000,
num_beams=5,
no_repeat_ngram_size=3,
repetition_penalty=1.2,
decoder_start_token_id=START_ID,
pad_token_id=PAD_ID,
eos_token_id=EOS_ID
)
# 3. Calculate Token Stats
output_tokens_len = generated_ids.shape[1]
# Decode
transcription = tokenizer.decode(generated_ids[0].tolist(), skip_special_tokens=True)
return transcription, output_tokens_len
# Run it!
if __name__ == "__main__":
try:
result, token_count = transcribe(AUDIO_PATH)
print(f"\n📝 Transcription:\n{result}")
print(f"\n--- Token Statistics ---")
print(f"Generated Output Text Tokens: {token_count} tokens")
except Exception as e:
print(f"❌ Error during inference execution: {e}")Citation
@inproceedings{hasan2026tokenizer,
title={Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR},
author={Hasan, Sanjid and Rahman, Md. Abdur},
booktitle={MuslimML Workshop at the 43rd International Conference on Machine Learning (ICML)},
year={2026}
}Paper on Arxiv...
@misc{hasan2026tokenizertransplantationmitigatingautoregressive,
title={Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR},
author={Sanjid Hasan and Md. Abdur Rahman},
year={2026},
eprint={2607.09598},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2607.09598},
}Acknowledgments
Built on UsefulSensors/Moonshine and BanglaBERT. Trained on the Lipi-Ghor-bn-882-SSTT dataset, with GPU support from the Department of CSE at Khulna University of Engineering & Technology (KUET).