Apppublic

dev2008/audio-separation

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes

App README

🎵 Text-Queried Time-Selective Audio Separation via CLAP-Conditioned Spectrogram Diffusion

A novel deep learning approach for text-guided audio source separation with time-selective editing capabilities.

Features • Architecture • Installation • Usage • Training • API • Results

</div>

📋 Abstract

This project presents a text-conditioned UNet architecture for audio source separation that leverages CLAP (Contrastive Language-Audio Pretraining) embeddings to enable natural language-guided separation. Unlike traditional methods that require pre-defined source categories, our approach allows users to specify separation targets using free-form text queries like "dog barking", "rain sounds", or "piano music".

Key Innovations:

1.Text-Guided Semantic Control - Natural language queries for flexible, zero-shot separation
2.Time-Selective Editing - Process only specific time regions while preserving the rest
3.FiLM Conditioning - Feature-wise Linear Modulation for effective text-audio fusion
4.Efficient Architecture - Only 12.8M parameters with 285ms inference time

✨ Key Features

Feature	Description
🎯 Text-Guided Separation	Use natural language to describe what sounds to isolate or remove
⏱️ Time-Selective Processing	Edit specific time regions (e.g., 1.5s - 3.5s) with smooth fades
🔄 Dual Modes	"Keep" mode to isolate sounds, "Remove" mode to suppress them
📊 Visual Feedback	Real-time spectrogram visualization of input and output
🎤 Audio Detection	AI-powered sound content detection using Gemini API
🌐 Web Interface	User-friendly browser-based UI for easy interaction
⚡ Fast Inference	~285ms processing time for 5-second audio clips

🏗️ Architecture

Model Overview

┌─────────────────────────────────────────────────────────────────┐
│                    Text-Conditioned UNet                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Input: Magnitude Spectrogram [B, 1, F, T]                    │
│          Text Prompt → CLAP Embedding [B, 1024]                │
│                                                                 │
│   ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐    │
│   │Encoder 1│───►│Encoder 2│───►│Encoder 3│───►│Encoder 4│    │
│   │  64 ch  │    │  128 ch │    │  256 ch │    │  512 ch │    │
│   └────┬────┘    └────┬────┘    └────┬────┘    └────┬────┘    │
│        │              │              │              │          │
│        │    Skip      │    Skip      │    Skip      │          │
│        │  Connections │  Connections │  Connections ▼          │
│        │              │              │         ┌─────────┐     │
│        │              │              │         │Bottleneck│    │
│        │              │              │         │ + FiLM  │     │
│        │              │              │         └────┬────┘     │
│        │              │              │              │          │
│   ┌────▼────┐    ┌────▼────┐    ┌────▼────┐    ┌────▼────┐    │
│   │Decoder 1│◄───│Decoder 2│◄───│Decoder 3│◄───│Decoder 4│    │
│   │  64 ch  │    │+ FiLM   │    │+ FiLM   │    │+ FiLM   │    │
│   └────┬────┘    └─────────┘    └─────────┘    └─────────┘    │
│        │                                                       │
│        ▼                                                       │
│   Output: Soft Mask [B, 1, F, T] ∈ [0, 1]                     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

FiLM (Feature-wise Linear Modulation) Layer

The FiLM layer enables text conditioning by modulating feature maps:

python

γ = Linear(text_embedding)  # Scale parameter
β = Linear(text_embedding)  # Shift parameter
output = γ * features + β   # Affine transformation

System Pipeline

Audio File ──► STFT ──► Magnitude Spectrogram ──┐
                                                 │
Text Prompt ──► CLAP Encoder ──► Text Embedding ─┼──► UNet ──► Soft Mask
                                                 │              │
                                                 └──────────────┼──► Apply Mask
                                                                │
                                           Time Gate ───────────┘
                                                                │
                                                    iSTFT ◄─────┘
                                                      │
                                               Separated Audio

🚀 Installation

Prerequisites

—Python 3.11+
—CUDA-capable GPU (recommended) or CPU
—~4GB disk space for models and dependencies

Step 1: Clone the Repository

bash

git clone https://github.com/devang2008/-Text-Queried-Time-Selective-Audio-Separation-via-CLAP-Conditioned-Spectrogram-Diffusion.git
cd -Text-Queried-Time-Selective-Audio-Separation-via-CLAP-Conditioned-Spectrogram-Diffusion

Step 2: Create Virtual Environment

bash

python -m venv .venv

# Windows
.\.venv\Scripts\activate

# Linux/Mac
source .venv/bin/activate

Step 3: Install Dependencies

bash

pip install -r requirements.txt

Step 4: Configure Environment Variables

bash

# Copy example environment file
cp .env.example .env

# Edit .env and add your API keys
# GEMINI_API_KEY=your_gemini_api_key_here

Step 5: Download ESC-50 Dataset (for training)

bash

# Download from: https://github.com/karolpiczak/ESC-50
# Extract to a directory and update path in src/config.py

💻 Usage

Running the Web Interface

bash

cd src
python -m uvicorn server:app --reload --host 127.0.0.1 --port 8000

Then open your browser and navigate to: http://127.0.0.1:8000

Web Interface Features

1.Select Audio: Upload your own audio file or select from ESC-50 dataset
2.Detect Sounds: Use AI to automatically detect sound classes in audio
3.Set Parameters:
4.Text Prompt: Describe the sound to separate (e.g., "dog barking")
5.Mode: "Keep" to isolate or "Remove" to suppress
6.Time Range: Select start and end times for time-selective editing
7.Method: Choose UNet (trained model) or NMF (baseline)
8.Run Separation: Click to process and hear the results

Command Line Usage

python

from unet_sep import separate_with_unet

result = separate_with_unet(
    audio_path="path/to/audio.wav",
    prompt="dog barking",
    mode="keep",        # "keep" or "remove"
    t0=1.0,             # Start time (seconds)
    t1=3.5,             # End time (seconds)
    fade_ms=70.0        # Fade duration (ms)
)

print(f"Output: {result['audio_out']}")
print(f"Residual: {result['audio_residual']}")
print(f"Confidence: {result['confidence']:.2f}")

🎓 Training

Training the UNet Model

bash

cd src
python train.py \
    --esc50_path /path/to/ESC-50 \
    --output_dir ../checkpoints \
    --batch_size 8 \
    --epochs 100 \
    --lr 1e-4 \
    --train_folds 1 2 3 4 \
    --val_folds 5

Training Arguments

Argument	Default	Description
`--esc50_path`	Required	Path to ESC-50 dataset
`--output_dir`	`checkpoints/`	Directory for model checkpoints
`--batch_size`	8	Training batch size
`--epochs`	100	Number of training epochs
`--lr`	1e-4	Learning rate
`--train_folds`	1 2 3 4	ESC-50 folds for training
`--val_folds`	5	ESC-50 folds for validation
`--baseline`	False	Train text-agnostic baseline

Training Data Preparation

The model is trained on synthetic mixtures created from ESC-50:

1.Target audio: Selected audio file with known class
2.Interferer audio: Randomly selected from different class
3.Mixture: Combined at random SNR (-5 to 5 dB)
4.Ground truth mask: Ideal Ratio Mask (IRM)

🔌 API Endpoints

`GET /api/files`

List available audio files from ESC-50 dataset.

`POST /api/separate`

Perform audio separation.

Request Body:

json

{
    "file_id": "1-100032-A-0",
    "prompt": "dog barking",
    "mode": "keep",
    "method": "unet",
    "t0": 0.0,
    "t1": 5.0,
    "k": 10
}

Response:

json

{
    "audio_out": "/outputs/audio/sep_abc123_out.wav",
    "audio_residual": "/outputs/audio/sep_abc123_residual.wav",
    "spectrogram_mask": "/outputs/img/sep_abc123_mask.png",
    "confidence": 0.85
}

`POST /api/detect`

Detect sound classes using CLAP embeddings.

`POST /api/analyze`

Analyze audio content using Gemini AI.

`POST /api/upload`

Upload custom audio file for processing.

📊 Results

Performance Comparison

Model	Text-Guided	Time-Selective	SI-SDR (dB)	Parameters	Inference Time
Wave-U-Net	✗	✗	9.2	28.3M	180ms
Conv-TasNet	✗	✗	10.8	5.1M	95ms
Demucs	✗	✗	11.5	64.2M	210ms
SepFormer	✗	✗	12.1	25.6M	140ms
DiffSep	Partial	✗	14.3	89.4M	850ms
Ours (UNet+CLAP)	✓	✓	12.7	12.8M	285ms

Key Advantages

1.3× faster than diffusion-based methods
2.75% fewer parameters than Demucs
3.Zero-shot capability via CLAP's 500+ sound class knowledge
4.Unique time-selective editing feature

📁 Project Structure

.
├── src/
│   ├── model.py           # UNet architecture with FiLM layers
│   ├── train.py           # Training script
│   ├── inference.py       # Inference utilities
│   ├── unet_sep.py        # UNet separation wrapper
│   ├── nmf_sep.py         # NMF baseline method
│   ├── clap_embed.py      # CLAP embedding functions
│   ├── audio_utils.py     # Audio processing utilities
│   ├── audio_analyzer.py  # Gemini AI integration
│   ├── dataset.py         # ESC-50 dataset loader
│   ├── config.py          # Configuration settings
│   └── server.py          # FastAPI server
├── static/
│   ├── index.html         # Web interface
│   ├── styles.css         # Styling
│   └── app.js             # Frontend JavaScript
├── checkpoints/           # Trained model weights
├── outputs/               # Generated outputs
├── requirements.txt       # Python dependencies
├── .env.example           # Environment template
└── README.md              # This file

🛠️ Technologies Used

—Deep Learning: PyTorch, CLAP (msclap)
—Audio Processing: librosa, soundfile, torchaudio
—Web Framework: FastAPI, Uvicorn
—AI Integration: Google Gemini API
—Scientific Computing: NumPy, SciPy, scikit-learn
—Visualization: Matplotlib

📚 References

1.CLAP: Elizalde et al., "CLAP: Learning Audio Concepts from Natural Language Supervision", ICASSP 2023
2.UNet: Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image Segmentation", MICCAI 2015
3.FiLM: Perez et al., "FiLM: Visual Reasoning with a General Conditioning Layer", AAAI 2018
4.ESC-50: Piczak, "ESC: Dataset for Environmental Sound Classification", ACM MM 2015

👥 Authors

—Group 09 - Machine Learning Course Project (TY-SEM-I)

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

—ESC-50 dataset by Karol Piczak
—Microsoft CLAP implementation
—Google Gemini API for audio analysis

⭐ Star this repository if you find it useful! ⭐

</div>