Usagert12/voice-contrastive-demo

🎤 Voice Conversion with Contrastive Learning

Enhancement of FreeVC Using Contrastive Learning Techniques

MS Thesis Project Hassan | ITU Pakistan | Computer Science

🎯 About This Demo

This demo showcases the enhancement of voice conversion using contrastive learning in the speaker latent space.

Two Models Compared:

1. Baseline FreeVC

• —Standard voice conversion architecture
• —Speaker encoder: WavLM embeddings
• —Content encoder: Information Perturbation
• —Flow-based generator

2. Enhanced Contrastive Model ✨

• —Incorporates contrastive learning in 256-dim speaker space
• —Positive pairs (same speaker) pulled together
• —Negative pairs (different speakers) pushed apart
• —Result: Better speaker-content disentanglement

🚀 How to Use

1. 1.Upload Source Audio - The content you want to convert (what to say)
2. 2.Upload Target Speaker - The voice identity (whose voice to use)
3. 3.Select Model(s) - Baseline, Enhanced, or Both for comparison
4. 4.Click Convert - Listen to the results!

Tips for Best Results:

• —Use clear audio with minimal background noise
• —Provide 5-10 seconds of target speaker audio
• —Try "Both" mode to hear the improvement from contrastive learning
• —Source and target can be different genders

📊 Performance Improvements

Contrastive learning demonstrates significant improvements:

🔬 Technical Details

Dataset

• —Primary: VCTK Corpus (109 speakers, English)
• —Duration filtering: Files ≥5 seconds prioritized
• —Split: 70% train, 10% validation, 20% test

Architecture

• —Generator: Modified HiFi-GAN with speaker conditioning
• —Discriminator: Multi-scale, multi-period
• —Loss Functions:
• —Standard: Generator, Discriminator, Mel-spectrogram, KL divergence
• —Novel: Contrastive loss with on-the-fly pair calculation

Training

• —Platform: Kaggle (Tesla P100, 16GB VRAM)
• —Epochs: 960 epochs
• —Steps: 326,400 steps
• —Optimizer: AdamW

Key Innovation

Contrastive learning enforces better speaker-content separation by:

1. 1.Creating positive pairs: same speaker, different utterances
2. 2.Creating negative pairs: different speakers, same content
3. 3.Optimizing embeddings to maximize inter-speaker distance
4. 4.Minimizing intra-speaker variance

📚 Research Context

Problem Statement

Traditional voice conversion models struggle with speaker-content disentanglement:

• —Speaker identity leaks into content representation
• —Content information bleeds into speaker embeddings
• —Poor generalization to unseen speakers

Solution

Explicit separation enforcement through contrastive learning in the speaker latent space, resulting in:

• —Better speaker identity preservation
• —Improved content quality
• —Enhanced generalization
• —Clearer speaker-content boundaries (validated via t-SNE)

Related Work

This research builds upon:

• —FreeVC: One-Shot Voice Conversion
• —Contrastive Learning in Speech Processing
• —Speaker Embedding Disentanglement

🎓 Academic Information

Thesis Title: Enhancement of Voice Conversion using Contrastive Learning Techniques

Student: Hassan Institution: Information Technology University (ITU), Pakistan Program: MS Computer Science Year: 2025

Advisor: [Advisor Name] Committee: [Committee Members]

💻 Implementation

Technologies Used

• —Framework: PyTorch
• —UI: Gradio
• —Audio Processing: librosa, soundfile
• —Model Architecture: FreeVC (enhanced)

Code Structure

app.py                  # Entry point
demo_integrated.py      # Gradio interface
model_wrapper.py        # Model loading
models.py              # FreeVC architecture
utils.py               # Utilities

🔗 Links

• —📄 Thesis Paper: [Coming Soon]
• —💻 Code Repository: [GitHub Link]
• —🎓 Institution: ITU Pakistan

📝 Citation

If you use this work in your research, please cite:

@mastersthesis{hassan2025voice,
  title={Enhancement of Voice Conversion using Contrastive Learning Techniques},
  author={Hassan},
  year={2025},
  school={Information Technology University, Pakistan},
  type={MS Thesis}
}

⚙️ Technical Notes

First Load

The first time you use this demo, model loading may take 1-2 minutes. Subsequent uses will be faster.

Performance

• —With GPU: ~5-10 seconds per conversion
• —With CPU: ~30-60 seconds per conversion

Limitations

• —Optimized for English (VCTK dataset)
• —Best with clear, studio-quality audio
• —May struggle with extreme cross-gender conversion
• —Background noise affects quality

🔮 Future Work

• —Extension to Urdu language voice conversion
• —Integration with TTS for complete synthesis pipeline
• —Real-time conversion optimization
• —Multi-speaker simultaneous conversion
• —Mobile deployment

🤝 Acknowledgments

• —VCTK Corpus creators
• —FreeVC original authors
• —ITU Pakistan CS Department
• —Thesis advisor and committee
• —Kaggle for GPU resources

⚠️ Disclaimer

This is a research prototype for academic demonstration. Audio quality may vary with different inputs. Not intended for production use without further validation.

📞 Contact

For questions or collaboration:

• —Email: [Your Email]
• —LinkedIn: [Your LinkedIn]
• —GitHub: [Your GitHub]

Built with ❤️ using:

• —🤗 Gradio
• —🔥 PyTorch
• —🎵 librosa
• —📊 NumPy

Last Updated: December 2025