dev2008/audio-separation
0
1---2title: Audio Separation3emoji: π΅4colorFrom: blue5colorTo: indigo6sdk: docker7app_port: 78608pinned: false9---10 11# π΅ Text-Queried Time-Selective Audio Separation via CLAP-Conditioned Spectrogram Diffusion12 13<div align="center">14 1516171819 20**A novel deep learning approach for text-guided audio source separation with time-selective editing capabilities.**21 22[Features](#-key-features) β’ [Architecture](#-architecture) β’ [Installation](#-installation) β’ [Usage](#-usage) β’ [Training](#-training) β’ [API](#-api-endpoints) β’ [Results](#-results)23 24</div>25 26---27 28## π Abstract29 30This project presents a **text-conditioned UNet architecture** for audio source separation that leverages **CLAP (Contrastive Language-Audio Pretraining)** embeddings to enable natural language-guided separation. Unlike traditional methods that require pre-defined source categories, our approach allows users to specify separation targets using free-form text queries like *"dog barking"*, *"rain sounds"*, or *"piano music"*.31 32### Key Innovations:331. **Text-Guided Semantic Control** - Natural language queries for flexible, zero-shot separation342. **Time-Selective Editing** - Process only specific time regions while preserving the rest353. **FiLM Conditioning** - Feature-wise Linear Modulation for effective text-audio fusion364. **Efficient Architecture** - Only 12.8M parameters with 285ms inference time37 38---39 40## β¨ Key Features41 42| Feature | Description |43|---------|-------------|44| π― **Text-Guided Separation** | Use natural language to describe what sounds to isolate or remove |45| β±οΈ **Time-Selective Processing** | Edit specific time regions (e.g., 1.5s - 3.5s) with smooth fades |46| π **Dual Modes** | "Keep" mode to isolate sounds, "Remove" mode to suppress them |47| π **Visual Feedback** | Real-time spectrogram visualization of input and output |48| π€ **Audio Detection** | AI-powered sound content detection using Gemini API |49| π **Web Interface** | User-friendly browser-based UI for easy interaction |50| β‘ **Fast Inference** | ~285ms processing time for 5-second audio clips |51 52---53 54## ποΈ Architecture55 56### Model Overview57 58```59βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ60β Text-Conditioned UNet β61βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€62β β63β Input: Magnitude Spectrogram [B, 1, F, T] β64β Text Prompt β CLAP Embedding [B, 1024] β65β β66β βββββββββββ βββββββββββ βββββββββββ βββββββββββ β67β βEncoder 1βββββΊβEncoder 2βββββΊβEncoder 3βββββΊβEncoder 4β β68β β 64 ch β β 128 ch β β 256 ch β β 512 ch β β69β ββββββ¬βββββ ββββββ¬βββββ ββββββ¬βββββ ββββββ¬βββββ β70β β β β β β71β β Skip β Skip β Skip β β72β β Connections β Connections β Connections βΌ β73β β β β βββββββββββ β74β β β β βBottleneckβ β75β β β β β + FiLM β β76β β β β ββββββ¬βββββ β77β β β β β β78β ββββββΌβββββ ββββββΌβββββ ββββββΌβββββ ββββββΌβββββ β79β βDecoder 1ββββββDecoder 2ββββββDecoder 3ββββββDecoder 4β β80β β 64 ch β β+ FiLM β β+ FiLM β β+ FiLM β β81β ββββββ¬βββββ βββββββββββ βββββββββββ βββββββββββ β82β β β83β βΌ β84β Output: Soft Mask [B, 1, F, T] β [0, 1] β85β β86βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ87```88 89### FiLM (Feature-wise Linear Modulation) Layer90 91The FiLM layer enables text conditioning by modulating feature maps:92 93```python94Ξ³ = Linear(text_embedding) # Scale parameter95Ξ² = Linear(text_embedding) # Shift parameter96output = Ξ³ * features + Ξ² # Affine transformation97```98 99### System Pipeline100 101```102Audio File βββΊ STFT βββΊ Magnitude Spectrogram βββ103 β104Text Prompt βββΊ CLAP Encoder βββΊ Text Embedding ββΌβββΊ UNet βββΊ Soft Mask105 β β106 ββββββββββββββββΌβββΊ Apply Mask107 β108 Time Gate ββββββββββββ109 β110 iSTFT βββββββ111 β112 Separated Audio113```114 115---116 117## π Installation118 119### Prerequisites120 121- Python 3.11+122- CUDA-capable GPU (recommended) or CPU123- ~4GB disk space for models and dependencies124 125### Step 1: Clone the Repository126 127```bash128git clone https://github.com/devang2008/-Text-Queried-Time-Selective-Audio-Separation-via-CLAP-Conditioned-Spectrogram-Diffusion.git129cd -Text-Queried-Time-Selective-Audio-Separation-via-CLAP-Conditioned-Spectrogram-Diffusion130```131 132### Step 2: Create Virtual Environment133 134```bash135python -m venv .venv136 137# Windows138.\.venv\Scripts\activate139 140# Linux/Mac141source .venv/bin/activate142```143 144### Step 3: Install Dependencies145 146```bash147pip install -r requirements.txt148```149 150### Step 4: Configure Environment Variables151 152```bash153# Copy example environment file154cp .env.example .env155 156# Edit .env and add your API keys157# GEMINI_API_KEY=your_gemini_api_key_here158```159 160### Step 5: Download ESC-50 Dataset (for training)161 162```bash163# Download from: https://github.com/karolpiczak/ESC-50164# Extract to a directory and update path in src/config.py165```166 167---168 169## π» Usage170 171### Running the Web Interface172 173```bash174cd src175python -m uvicorn server:app --reload --host 127.0.0.1 --port 8000176```177 178Then open your browser and navigate to: **http://127.0.0.1:8000**179 180### Web Interface Features181 1821. **Select Audio**: Upload your own audio file or select from ESC-50 dataset1832. **Detect Sounds**: Use AI to automatically detect sound classes in audio1843. **Set Parameters**:185 - **Text Prompt**: Describe the sound to separate (e.g., "dog barking")186 - **Mode**: "Keep" to isolate or "Remove" to suppress187 - **Time Range**: Select start and end times for time-selective editing188 - **Method**: Choose UNet (trained model) or NMF (baseline)1894. **Run Separation**: Click to process and hear the results190 191### Command Line Usage192 193```python194from unet_sep import separate_with_unet195 196result = separate_with_unet(197 audio_path="path/to/audio.wav",198 prompt="dog barking",199 mode="keep", # "keep" or "remove"200 t0=1.0, # Start time (seconds)201 t1=3.5, # End time (seconds)202 fade_ms=70.0 # Fade duration (ms)203)204 205print(f"Output: {result['audio_out']}")206print(f"Residual: {result['audio_residual']}")207print(f"Confidence: {result['confidence']:.2f}")208```209 210---211 212## π Training213 214### Training the UNet Model215 216```bash217cd src218python train.py \219 --esc50_path /path/to/ESC-50 \220 --output_dir ../checkpoints \221 --batch_size 8 \222 --epochs 100 \223 --lr 1e-4 \224 --train_folds 1 2 3 4 \225 --val_folds 5226```227 228### Training Arguments229 230| Argument | Default | Description |231|----------|---------|-------------|232| `--esc50_path` | Required | Path to ESC-50 dataset |233| `--output_dir` | `checkpoints/` | Directory for model checkpoints |234| `--batch_size` | 8 | Training batch size |235| `--epochs` | 100 | Number of training epochs |236| `--lr` | 1e-4 | Learning rate |237| `--train_folds` | 1 2 3 4 | ESC-50 folds for training |238| `--val_folds` | 5 | ESC-50 folds for validation |239| `--baseline` | False | Train text-agnostic baseline |240 241### Training Data Preparation242 243The model is trained on synthetic mixtures created from ESC-50:2441. **Target audio**: Selected audio file with known class2452. **Interferer audio**: Randomly selected from different class2463. **Mixture**: Combined at random SNR (-5 to 5 dB)2474. **Ground truth mask**: Ideal Ratio Mask (IRM)248 249---250 251## π API Endpoints252 253### `GET /api/files`254List available audio files from ESC-50 dataset.255 256### `POST /api/separate`257Perform audio separation.258 259**Request Body:**260```json261{262 "file_id": "1-100032-A-0",263 "prompt": "dog barking",264 "mode": "keep",265 "method": "unet",266 "t0": 0.0,267 "t1": 5.0,268 "k": 10269}270```271 272**Response:**273```json274{275 "audio_out": "/outputs/audio/sep_abc123_out.wav",276 "audio_residual": "/outputs/audio/sep_abc123_residual.wav",277 "spectrogram_mask": "/outputs/img/sep_abc123_mask.png",278 "confidence": 0.85279}280```281 282### `POST /api/detect`283Detect sound classes using CLAP embeddings.284 285### `POST /api/analyze`286Analyze audio content using Gemini AI.287 288### `POST /api/upload`289Upload custom audio file for processing.290 291---292 293## π Results294 295### Performance Comparison296 297| Model | Text-Guided | Time-Selective | SI-SDR (dB) | Parameters | Inference Time |298|-------|-------------|----------------|-------------|------------|----------------|299| Wave-U-Net | β | β | 9.2 | 28.3M | 180ms |300| Conv-TasNet | β | β | 10.8 | 5.1M | 95ms |301| Demucs | β | β | 11.5 | 64.2M | 210ms |302| SepFormer | β | β | 12.1 | 25.6M | 140ms |303| DiffSep | Partial | β | 14.3 | 89.4M | 850ms |304| **Ours (UNet+CLAP)** | **β** | **β** | **12.7** | **12.8M** | **285ms** |305 306### Key Advantages307 3081. **3Γ faster** than diffusion-based methods3092. **75% fewer parameters** than Demucs3103. **Zero-shot capability** via CLAP's 500+ sound class knowledge3114. **Unique time-selective editing** feature312 313---314 315## π Project Structure316 317```318.319βββ src/320β βββ model.py # UNet architecture with FiLM layers321β βββ train.py # Training script322β βββ inference.py # Inference utilities323β βββ unet_sep.py # UNet separation wrapper324β βββ nmf_sep.py # NMF baseline method325β βββ clap_embed.py # CLAP embedding functions326β βββ audio_utils.py # Audio processing utilities327β βββ audio_analyzer.py # Gemini AI integration328β βββ dataset.py # ESC-50 dataset loader329β βββ config.py # Configuration settings330β βββ server.py # FastAPI server331βββ static/332β βββ index.html # Web interface333β βββ styles.css # Styling334β βββ app.js # Frontend JavaScript335βββ checkpoints/ # Trained model weights336βββ outputs/ # Generated outputs337βββ requirements.txt # Python dependencies338βββ .env.example # Environment template339βββ README.md # This file340```341 342---343 344## π οΈ Technologies Used345 346- **Deep Learning**: PyTorch, CLAP (msclap)347- **Audio Processing**: librosa, soundfile, torchaudio348- **Web Framework**: FastAPI, Uvicorn349- **AI Integration**: Google Gemini API350- **Scientific Computing**: NumPy, SciPy, scikit-learn351- **Visualization**: Matplotlib352 353---354 355## π References356 3571. **CLAP**: Elizalde et al., "CLAP: Learning Audio Concepts from Natural Language Supervision", ICASSP 20233582. **UNet**: Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image Segmentation", MICCAI 20153593. **FiLM**: Perez et al., "FiLM: Visual Reasoning with a General Conditioning Layer", AAAI 20183604. **ESC-50**: Piczak, "ESC: Dataset for Environmental Sound Classification", ACM MM 2015361 362---363 364## π₯ Authors365 366- **Group 09** - Machine Learning Course Project (TY-SEM-I)367 368---369 370## π License371 372This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.373 374---375 376## π Acknowledgments377 378- ESC-50 dataset by Karol Piczak379- Microsoft CLAP implementation380- Google Gemini API for audio analysis381 382---383 384<div align="center">385 386**β Star this repository if you find it useful! β**387 388</div>389