Themozhi/tamil-dysarthric-voice-conversion

🎙️ Tamil Dysarthric Voice Conversion

Transform dysarthric (unclear) Tamil speech into clear, intelligible audio using state-of-the-art deep learning models.

🌟 Features

• —End-to-End Neural Conversion: Uses HuBERT, Acoustic Model, and HiFi-GAN Vocoder
• —Dataset Support: Automatically handles pre-computed healthy versions for known speakers (FSI, FVP, MPA, MPR, MRA, MPK, MAK)
• —External File Processing: Full neural pipeline for any audio file
• —High-Quality Output: Generates natural-sounding enhanced speech
• —Easy to Use: Simple upload and process interface
• —Tamil Language Optimized: Specifically trained for Tamil dysarthric speech patterns

🚀 How to Use

1. 1.Upload your dysarthric speech audio file (WAV, MP3, FLAC, etc.)
2. 2.Click the "Enhance Speech" button
3. 3.Wait for processing (10-30 seconds depending on audio length)
4. 4.Listen to the enhanced output and download if needed

File Types Supported:

Dataset Files (e.g., FSI123.wav, MPA45.wav):

• —Automatically uses pre-computed healthy version if available in dataset
• —Instant processing (no neural network inference needed)
• —Speakers: FSI, FVP, MPA, MPR, MRA, MPK, MAK

External Files (any other .wav, .mp3, etc.):

• —Processes through full neural pipeline
• —Takes 10-30 seconds (first run may take longer for model loading)
• —Works with any Tamil dysarthric speech

🔧 Technical Details

Model Pipeline

The system uses a three-stage neural pipeline:

1. 1.HuBERT (Hidden Unit BERT): Extracts discrete speech units from the input audio
2. 2.Acoustic Model: Generates target mel-spectrogram from speech units
3. 3.HiFi-GAN Vocoder: Synthesizes high-quality audio waveform from spectrogram

Processing Flow

Dysarthric Audio → HuBERT → Speech Units → Acoustic Model → Mel-Spectrogram → HiFi-GAN → Enhanced Audio

📊 Model Information

• —HuBERT Model: Themozhi/hubert
• —Acoustic Model: Themozhi/acoustic-model
• —Vocoder: Themozhi/hifigan
• —Sample Rate: 16kHz
• —Architecture: Transformer-based sequence-to-sequence with neural vocoder

🎯 Use Cases

• —Assistive Technology: Help individuals with dysarthria communicate more clearly
• —Research: Study speech disorders and rehabilitation techniques
• —Medical Applications: Support speech therapy and monitoring
• —Accessibility: Improve communication accessibility for people with speech impairments

⚙️ System Requirements

For GPU Processing (Recommended)

• —Hardware: NVIDIA GPU with CUDA support
• —Processing Time: ~10-20 seconds per audio file
• —Best For: Real-time or frequent use

For CPU Processing

• —Hardware: Any modern CPU
• —Processing Time: ~30-60 seconds per audio file
• —Best For: Occasional use

📝 Notes

• —First Run: Initial model loading may take 1-2 minutes
• —Audio Format: Supports WAV, MP3, FLAC, M4A and other common formats
• —Audio Length: Works best with 5-30 second audio clips
• —Language: Optimized for Tamil language, but may work with other languages

🔬 Research Background

This project is based on research in dysarthric speech conversion using self-supervised learning models. Dysarthria is a motor speech disorder that affects the ability to produce clear speech due to muscle weakness or difficulty controlling speech muscles.

🤝 Contributing

Contributions, issues, and feature requests are welcome!

⚖️ License

MIT License - See LICENSE file for details

🙏 Acknowledgments

• —HuBERT model architecture from Facebook AI Research
• —HiFi-GAN vocoder from NVIDIA
• —Gradio for the web interface framework
• —Hugging Face for hosting infrastructure

Note: This is a research prototype. For clinical or production use, please consult with speech-language pathologists and conduct thorough validation.