jnwnlee/vgg-monoaudio

VGG-MonoAudio

 

A curated evaluation benchmark designed for text-conditioned selective video-to-audio (V2A) generation.

<div style="margin:16px auto; justify-content:center; text-align:center;"> <img src="vgg_monoaudio.png" alt="VGG-MonoAudio" style="width:70%; height:auto;" /> <figcaption>VGG-MonoAudio overview.</figcaption> </div>

TL;DR

VGG-MonoAudio comprises triplets of video (featuring multiple visible sound sources), text (identifying the target source), and audio (containing only the clean, isolated target sound). It is constructed by visually mixing single-source clips from VGGSound and UnAV-100 while retaining only the specific mono-audio track for the target. This design addresses the limitations of existing datasets, such as inseparable audio mixtures or off-screen noise, making it a precise standard for evaluating object-selective sound generation. The mono-source video clips are carefully filtered and verified by human experts.

Stats and Formats

1,071 8-second mixed videos in total, 560 inter-class and 511 intra-class pairs. Sourced from 67 curated videos with 39 unique text labels and 8 sound categories.

• —Video: 1280 × 720 resolution, H.264, 25 fps.
• —Audio: mono-channel, AAC, 16kHz.

Structure

inter_class/
├─ mixed/              # mixed videos
│  └─ {file_name}.mp4
├─ target_audio/       # target audios
│  ├─ embeddings/      # audio features for KAD
│  ├─ kernel_stats/    # kernel stats for KAD
│  └─ {file_name}.wav
├─ gt_cache/           # video/text features for FAD, KL, IS, CLAP, IB, DeSync
│  └─ {features}.pth
├─ cache/              # audio features for FAD, KL, IS, CLAP, IB, DeSync
│  └─ {features}.pth
└─ metadata.csv
intra_class/
└─ ...

Metadata

For metadata.csv in each subset folder:

• —file_name: Unique id for the mixed sample (target + paired clip + spatial position).
• —target_file: Source clip YouTube ID used as the target audio/video.
• —paired_file: Source clip YouTube ID used as the paired (distractor) video.
• —target_position: Spatial placement of the target in the mix (left or right).
• —target_start_sec: Start time (seconds) for the target clip segment (originally 10 seconds).
• —paired_start_sec: Start time (seconds) for the paired clip segment.
• —label: Text label of the target sound.
• —target_category: Coarse category of the target (e.g., Human, Music, Vehicle).
• —paired_category: Coarse category of the paired (distractor) sound.
• —paired_label: Fine-grained class label of the paired sound.

Citation

@article{selva,
  title={Hear What Matters! Text-conditioned Selective Video-to-Audio Generation},
  author={Lee, Junwon and Nam, Juhan and Lee, Jiyoung},
  journal={arXiv preprint arXiv:2512.02650},
  year={2025}
}

Note: Since this is a derivative of VGGSound, usage is subject to the original Creative Commons Attribution 4.0 International License (CC BY 4.0) and the terms of service of the source videos (YouTube).