Datasetpublic

Rakancorle1/vggsync-3k

VGGSync-3K · out-of-domain audio-visual sync benchmark Out-of-domain evaluation set used in the paper When Vision Speaks for Sound. Derived from VGGSoundSync, this 3,000-clip slice tests whether a video-capable MLLM can detect audio temporal offsets on everyday sound events outside the THUD in-domain training distribution. Each item is one VGGSound clip in one of three conditions: Condition Count gt_synced gt_direction gt_offset_sec Audio aligned (no shift) 1,000 true… See the full description on the dataset page: https://huggingface.co/datasets/Rakancorle1/vggsync-3k.

sourceHugging Facecc-by-nc-4.0updated 4mo agoView on Hugging Face

0likes28downloads

Dataset Card

VGGSync-3K · out-of-domain audio-visual sync benchmark

Out-of-domain evaluation set used in the paper [When Vision Speaks for Sound](https://arxiv.org/abs/2605.16403).

Derived from VGGSoundSync, this 3,000-clip slice tests whether a video-capable MLLM can detect audio temporal offsets on everyday sound events outside the THUD in-domain training distribution.

Each item is one VGGSound clip in one of three conditions:

Condition	Count	`gt_synced`	`gt_direction`	`gt_offset_sec`
Audio aligned (no shift)	1,000	`true`	`none`	`0.0`
Audio shifted early	1,000	`false`	`early`	shift amount
Audio shifted delay	1,000	`false`	`delay`	shift amount

Shift magnitudes span 4 difficulty levels (very_easy, easy, medium, hard); synced clips are tagged difficulty=synced.

What's in this repo

File	Description
`eval/test_3k.jsonl`	Test records — uid / label / paths / ground truth
`media.zip`	All referenced videos + audios

How to use

bash

# Download
hf download Rakancorle1/vggsync-3k --repo-type=dataset --local-dir vggsync-3k
cd vggsync-3k

# Unzip media — paths in JSONL resolve automatically
unzip -q media.zip

python

from datasets import load_dataset
ds = load_dataset("Rakancorle1/vggsync-3k")

Record schema

jsonc

{
  "uid": "-r3nM90RCNs_medium_early_0.5s",
  "ytid": "-r3nM90RCNs",
  "label": "sharpen knife",
  "difficulty": "very_easy" | "easy" | "medium" | "hard" | "synced",
  "video_path": "videos/<uid>.mp4",
  "audio_path": "audios/<uid>.wav",
  "gt_synced": true | false,
  "gt_direction": "none" | "early" | "delay",
  "gt_offset_sec": 0.0          // > 0 for shifted; sign implied by direction
}

uid follows the pattern <ytid>_<difficulty>_<direction>_<offset>s for shifted clips, and <ytid> alone for synced clips.

Reference eval scripts

Paper evaluation code lives in the wvs-code repository on GitHub.

The JSONL is self-contained — video_path, audio_path, and the gt_* fields are all you need to compare any model's output against ground truth.

Citation

bibtex

@article{wen2026whenvisionspeaksforsound,
  title     = {When Vision Speaks for Sound},
  author    = {Xiaofei Wen and Wenjie Jacky Mo and Xingyu Fu and Rui Cai and
               Tinghui Zhu and Wendi Li and Yanan Xie and Muhao Chen and Peng Qi},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.16403}
}

Please also cite the original VGGSoundSync work:

bibtex

@InProceedings{Chen21b,
  title        = "Audio-Visual Synchronization in the Wild",
  author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
  booktitle    = "BMVC",
  year         = "2021"}

Related releases

—[Rakancorle1/thud-eval](https://huggingface.co/datasets/Rakancorle1/thud-eval) — in-domain audio-visual Clever Hans benchmark (sync / mute / swap)
—[Rakancorle1/hans-10k](https://huggingface.co/datasets/Rakancorle1/hans-10k) — DPO training data
—[Rakancorle1/hans-sft-4k](https://huggingface.co/datasets/Rakancorle1/hans-sft-4k) — SFT training data
—[Collection](https://huggingface.co/collections/Rakancorle1/when-vision-speaks-for-sound) — everything in one place

License

This subset is released under CC-BY-NC-4.0 for the annotations and shifted-clip pairings. The underlying audio/video content is derived from VGGSound — please follow VGGSound's licensing terms when using or redistributing.