Abhisingh-18/kannada-english-slam-asr

Kannada-English Bilingual SLAM-ASR: Speech Encoder Comparison

Five speech encoders compared as the frozen "ears" in a SLAM-LLM style ASR pipeline: frozen speech encoder → linear projector → Gemma-3-4B-IT (LoRA). Trained for Kannada-English code-switched speech recognition, evaluated on 4 Kannada benchmark testsets after ~1 epoch (33,265 steps).

Code: SLAM-LLM-for-Kanada (fork of ddlBoJack/SLAM-LLM)

What's in this repo

Only the trainable weights (LoRA adapters on the Gemma-3-4B-IT decoder + the linear projector) — each encoder itself is a large pretrained model kept frozen during training, so it is not re-uploaded here; see each subfolder's notes for where to get the base encoder checkpoint.

data2vec-ft/    data2vec-AQC encoder, Kannada CTC-finetuned (best WER)
data2vec-ssl/   data2vec-AQC encoder, multilingual SSL-pretrained-only
xeus/           XEUS (ESPnet E-Branchformer SSL) encoder
transformer/    ESPnet2 Transformer ASR encoder (45M params)
whisper/        OpenAI Whisper large-v3 encoder (635M params)

Each subfolder contains:

• —adapter_model.bin — LoRA (r=8, alpha=32) + linear projector weights, PyTorch state_dict
• —decode_results/ — ground-truth + predicted transcripts for all 4 testsets
• —wer/ — per-testset WER scoring output

Results (WER %, lower is better)

data2vec-AQC (Kannada CTC-finetuned) wins on every testset — it's the only encoder here already fine-tuned on Kannada speech before this SLAM-LLM stage. Whisper large-v3 is the best of the general-purpose encoders, closely matching XEUS. The 45M-param ESPnet Transformer (much smaller, less pretraining) trails noticeably.

Usage

These are adapter/projector weights, not a standalone model — you need the SLAM-LLM-for-Kanada codebase, the corresponding frozen speech encoder checkpoint, and google/gemma-3-4b-it to run inference. See that repo's examples/asr_librispeech/scripts/inference_data2vec_gemma3_kannada_*_4testsets.sh scripts for the exact hydra config each encoder needs (dims, inputtype, normalize flags, etc.) and set `++ckptpath=<adapter_model.bin>`.

Training details

• —LLM: google/gemma-3-4b-it, LoRA r=8/alpha=32/dropout=0.05 on q/k/v/o/gate/up/down_proj
• —Encoder: frozen, linear projector (downsample rate 5)
• —Data: bilingual Kannada-English code-switched speech, ~33k steps/epoch
• —Trained with DeepSpeed ZeRO-2, bf16

Several encoder-specific correctness bugs were found and fixed during this project (frame-rate mismatches, normalization, dtype issues under DeepSpeed bf16) — see the code repo commit history and RESULTS.md for details.