iteratehack/voice-model-rl-training

1---2title: Voice Model RL Training3emoji: 🎙️4colorFrom: blue5colorTo: purple6sdk: gradio7sdk_version: 4.44.08app_file: app.py9pinned: false10license: mit11python_version: 3.1112hardware: t4-small13---14 15# Voice Model RL Training16 17Train open-source voice models using Reinforcement Learning with PPO and REINFORCE algorithms.18 19## Features20 21- 🎯 **Multiple RL Algorithms**: Choose between PPO and REINFORCE22- 🚀 **GPU Acceleration**: Automatic GPU detection and usage23- 📊 **Real-time Monitoring**: Track training progress in real-time24- 🎵 **Model Comparison**: Compare base vs trained models25- 💾 **Checkpoint Management**: Automatic model saving and loading26- 🎤 **Multiple Base Models**: Support for Wav2Vec2, WavLM, and more27 28## Supported Models29 30- Facebook Wav2Vec2 (Base & Large)31- Microsoft WavLM Base Plus32- Any compatible HuggingFace speech model33 34## How to Use35 36### 1. Training Tab37 381. **Select Base Model**: Choose from available pretrained models392. **Configure Algorithm**: Select PPO (recommended) or REINFORCE403. **Set Parameters**:41   - Episodes: 10-100 (start with 20 for testing)42   - Learning Rate: 1e-5 to 1e-3 (default: 3e-4)43   - Batch Size: 4-64 (depends on GPU memory)444. **Start Training**: Click "Start Training" and monitor progress45 46### 2. Compare Results Tab47 481. **Upload Audio**: Provide a test audio sample492. **Generate Comparison**: Process through both models503. **Listen**: Compare base vs trained model outputs51 52## Reward Functions53 54The training optimizes for three key metrics:55 56- **Clarity** (33%): Audio signal quality and noise reduction57- **Naturalness** (33%): Natural speech patterns and prosody58- **Accuracy** (34%): Fidelity to original content59 60## Hardware Requirements61 62- **CPU**: Works but slow (5-10 min per episode)63- **GPU**: Recommended (T4 or better) (1-2 min per episode)64- **Memory**: 8GB+ RAM, 4GB+ VRAM65 66## Technical Details67 68### RL Algorithms69 70**PPO (Proximal Policy Optimization)**71- More stable training72- Uses value function73- Better for most cases74- Slightly slower per episode75 76**REINFORCE**77- Simpler algorithm78- Higher variance79- Faster per episode80- May need more episodes81 82### Training Process83 841. Load pretrained base model852. Add RL policy/value heads863. Train using custom reward function874. Save checkpoints periodically885. Generate comparisons89 90## Local Development91 92Clone and run locally:93 94```bash95git clone https://huggingface.co/spaces/USERNAME/voice-model-rl-training96cd voice-model-rl-training97pip install -r requirements.txt98python app.py99```100 101## Repository Structure102 103```104voice-rl-training/105├── app.py                 # Main Gradio application106├── requirements.txt       # Python dependencies107├── README.md             # This file108├── voice_rl/             # Core training modules109│   ├── models/           # Model wrappers110│   ├── rl/               # RL algorithms111│   ├── training/         # Training orchestration112│   ├── data/             # Data handling113│   ├── monitoring/       # Metrics and visualization114│   └── evaluation/       # Model evaluation115└── workspace/            # Training outputs (git-ignored)116```117