TieIncred/parakeet-tdt-blind-spots

Blind Spots of nvidia/parakeet-tdt-0.6b-v2

This dataset documents 14 systematically identified blind spots in NVIDIA's parakeet-tdt-0.6b-v2 automatic speech recognition model. The errors span 8 distinct categories and reveal a consistent pattern: the model struggles with inputs outside the distribution of its Western English-centric training data.

Model Under Test

How the Model Was Loaded

Hardware: NVIDIA L4 GPU (23GB VRAM), Python 3.11, PyTorch 2.10, CUDA 12.8

pip install nemo_toolkit[asr] soundfile librosa

import nemo.collections.asr as nemo_asr
from omegaconf import OmegaConf

# Load the model
asr_model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")
asr_model = asr_model.to("cuda")
asr_model.eval()

# Use greedy decoding (non-batch) to avoid CUDA graph compatibility issues
decoding_cfg = asr_model.cfg.decoding
decoding_cfg.strategy = "greedy"
OmegaConf.update(decoding_cfg, "greedy.max_symbols", 10, force_add=True)
asr_model.change_decoding_strategy(decoding_cfg)

# Transcribe
output = asr_model.transcribe(["path/to/audio.wav"])
print(output[0].text)

Note on CUDA Graph Compatibility

With PyTorch 2.10+ and CUDA 12.8, the default greedy_batch decoding strategy triggers a CUDA graph compilation error due to a mismatch in cudaStreamGetCaptureInfo return values. Switching to greedy (non-batch) decoding resolves this without affecting output quality.

Dataset Schema

Each row contains:

Blind Spot Categories and Findings

Summary

Detailed Error Analysis

1. Code-Switched Hindi-English (WER: 23–29%)

The most severe blind spot. When Hindi and English are mixed in the same utterance, the model:

• —Misspells Hindi words using English phonetic approximations: "Mujhe" → "Muje", "lagta" → "lakta", "bohot" → "bohut"
• —Drops Hindi words entirely: "humein extra resources chahiye" → "the extra resources chahi" (lost "humein")
• —Merges words: "Yeh wala" → "Yewala", "accha hai" → "Atshahe"

This is expected: the model was trained on ~120K hours of monolingual English. Hindi phonemes have no representation in its 1024-token BPE vocabulary.

2. Medical Terminology (WER: 27%)

• —Rare medical terms mangled: "hepaticojejunostomy" → "hepaticogynostomy" (surgical term absent from training data)
• —Number format normalization: "five hundred milligrams" → "500 mg" (the model learned to normalize spoken numbers to digits, which is technically correct but deviates from verbatim transcription)

3. Indian Proper Nouns (WER: 23–29%)

• —Spelling variations on names: "Shri Venkateshwara" → "Sri Venkateshwar", "Visakhapatnam" → "Vishakhapatnam"
• —Catastrophic errors on institutions: "AIIMS Bhubaneswar" → "Aims, Ghuvanish were" — the acronym AIIMS (All India Institute of Medical Sciences) is destroyed, and "Bhubaneswar" becomes unrecognizable

4. Noisy Background (WER: 20–33%)

At SNR -5dB (severe noise):

• —Word substitutions: "conference room B" → "conference roomy"
• —The model degrades gracefully under moderate noise (SNR 5dB) but breaks down rapidly below SNR 0dB

5–8. Other Categories

• —Indian numbers/dates: "hours IST" → "Rs Ist" (IST timezone confused with currency abbreviation)
• —Hospitality: Minor punctuation reformatting
• —Regional words: "the anganwadi" → "their Anganwadi" (determiner substitution)
• —Disfluent speech: "uh" → "ah" (filler word approximation)

Audio Generation Methodology

Test audio was generated using:

• —Edge-TTS with Indian English voices (en-IN-PrabhatNeural, en-IN-NeerjaNeural) for natural-sounding Indian English speech
• —Synthetic augmentation (additive white noise at various SNR levels, time-stretching) applied to clean TTS audio
• —All audio is 16kHz mono WAV format, matching the model's expected input

Note: Using TTS-generated audio rather than real human recordings means the accent patterns are somewhat idealized. Real Indian-accented English speakers would likely produce even higher error rates due to greater phonetic variation.

Recommended Fine-Tuning Dataset

To fix these blind spots, the model should be fine-tuned on a dataset combining:

Sources

1. 1.Indian-accented English (~200 hours):
2. 2.IndicAccentDb: 8,116 recordings across 6 Indian accent varieties
3. 3.Svarah: 9.6 hours from 117 speakers across 65 Indian locations
4. 4.CommonVoice Indian English: 163 hours

1. 1.Code-switched Hindi-English (~100 hours):
2. 2.MUCS Challenge Data: ~600 hours including Hindi-English code-switching
3. 3.IndicVoices: 23.7K hours across 22 Indian languages (subset with code-switching)

1. 1.Domain-specific English (~100 hours):
2. 2.Medical dictation corpora (e.g., from clinical NLP datasets)
3. 3.Indian institutional names and geographic terms (can be sourced from Indian news broadcast transcripts)

1. 1.Noisy speech augmentations:
2. 2.Apply MUSAN noise at SNR 0–10dB to clean training data
3. 3.Room impulse response simulation for reverberant conditions

Estimated Dataset Size

• —Minimum viable: ~200 hours of targeted data (accented English + code-switched) would significantly reduce WER on these categories
• —Recommended: ~500–1,000 hours combining all sources above, with noise augmentation applied to 30% of samples
• —Rationale: The base model was trained on 120K hours; even 0.5% of that volume (600 hours) of high-quality targeted data has been shown to substantially improve domain adaptation in ASR models (see Whisper fine-tuning literature)

How to Assemble

1. 1.Download and filter existing HF datasets for Indian English content
2. 2.Apply text normalization to unify transcription conventions
3. 3.Use speed perturbation (0.9x–1.1x) and noise augmentation for robustness
4. 4.Fine-tune using NeMo's ASR fine-tuning pipeline with a reduced learning rate (1e-5 to 5e-5)

Reproduction

All code used to generate this dataset is included in the companion notebook and scripts:

• —scripts/source_audio.py — audio generation and sourcing
• —scripts/run_inference.py — model inference and WER computation
• —scripts/build_hf_dataset.py — HuggingFace dataset construction

Citation

If you use this dataset, please cite:

@dataset{parakeet_blind_spots_2026,
  title={Blind Spots of nvidia/parakeet-tdt-0.6b-v2: Indian English and Code-Switching},
  year={2026},
  url={https://huggingface.co/datasets/TieIncred/parakeet-tdt-blind-spots}
}