CoolFace
Datasetpublic

urgent-challenge/vmc2026-track1-dev

vmc2026-track1-dev Development subset of the VMC 2026 Track 1 data. The data is organized into two configs corresponding to two subjective evaluation paradigms: absolute rating (acr) and pairwise comparison (ccr). The sample_id values are namespaced strings such as vmc2026-track1-dev-acr_489 and vmc2026-track1-dev-ccr_7233. acr -- Absolute Category Rating 1,008 samples. Each row pairs a sample_id with one speech audio file, its released Mean Opinion Score (MOS)… See the full description on the dataset page: https://huggingface.co/datasets/urgent-challenge/vmc2026-track1-dev.

sourceHugging Facecc-by-4.0updated 11d agoView on Hugging Face
0likes85downloads
Dataset Card

vmc2026-track1-dev

Development subset of the VMC 2026 Track 1 data.

The data is organized into two configs corresponding to two subjective evaluation paradigms: absolute rating (acr) and pairwise comparison (ccr). The sample_id values are namespaced strings such as vmc2026-track1-dev-acr_489 and vmc2026-track1-dev-ccr_7233.

acr -- Absolute Category Rating

1,008 samples. Each row pairs a sample_id with one speech audio file, its released Mean Opinion Score (MOS), and the original utterance and system identifiers.

ColumnTypeDescription
sample_idstringNamespaced identifier such as vmc2026-track1-dev-acr_489
audioAudioSpeech audio (FLAC, mono);
sample_rateintSample rate in Hz
durationfloatDuration in seconds
labelfloatMean Opinion Score (MOS), in [1, 5]
utterance_idstringOriginal utterance ID, such as fileid_54
system_idstringOriginal system submission ID, such as 1412
languagestringSource utterance language code
speaker_idstringAnonymized speaker grouping key
textstringSource utterance transcript
is_simulatedbooltrue for simulated noisy speech with a clean reference; false for real recorded noisy speech without a clean reference

ccr -- Comparative Category Rating

2,520 samples. Each row pairs a sample_id with two speech audio files from different systems processing the same source utterance, its released CMOS (Comparative Mean Opinion Score), and the original utterance and system identifiers. Positive CMOS means audio_a is better than audio_b.

ColumnTypeDescription
sample_idstringNamespaced identifier such as vmc2026-track1-dev-ccr_7233
audio_aAudioSpeech audio from system A (FLAC, mono);
audio_bAudioSpeech audio from system B (FLAC, mono);
sample_rateintSample rate in Hz
durationfloatDuration in seconds
labelfloatComparative Mean Opinion Score (CMOS), in [-3, 3]; positive means audio_a is better
utterance_idstringOriginal utterance ID shared by the pair, such as fileid_561
system_id_astringOriginal system submission ID corresponding to audio_a
system_id_bstringOriginal system submission ID corresponding to audio_b
languagestringSource utterance language code
speaker_idstringAnonymized speaker grouping key
textstringSource utterance transcript
is_simulatedbooltrue for simulated noisy speech with a clean reference; false for real recorded noisy speech without a clean reference

Original Challenge Submission Format

Submit one space-delimited, predictions.csv file with one prediction per line:

csv
sample_id,pred_score
vmc2026-track1-test-acr_4588,3.42
vmc2026-track1-test-ccr_3061,-0.15

The file should contain exactly 1,008 ACR rows and 2,520 CCR rows -- one prediction per sample_id in this dataset. ACR scores must lie in [1, 5]; CCR scores in [-3, +3].

Loading

Check https://github.com/pytorch/torchcodec to install the right version. Then load and iterate normally:

python
from datasets import load_dataset

acr = load_dataset("urgent-challenge/vmc2026-track1-dev", "acr", split="dev")  # 1,008 rows
ccr = load_dataset("urgent-challenge/vmc2026-track1-dev", "ccr", split="dev")  # 2,520 rows

Each row's audio (or audio_a / audio_b) is a torchcodec AudioDecoder -- not a dict. Call get_all_samples() to materialise the waveform as a torch.Tensor of shape [num_channels, num_samples]:

python
>>> acr[0]
{'sample_id': 'vmc2026-track1-dev-acr_489',
 'audio': <datasets.features._torchcodec.AudioDecoder object at 0x...>,
 'sample_rate': 32000,
 'duration': 8.424,
 'label': 3.0,
 'utterance_id': 'fileid_54',
 'system_id': '1412',
 'language': 'ita',
 'speaker_id': 'spk_438',
 'text': 'Iniziano un feud con Bill Goldberg, il wrestler più amato della federazione di Atlanta.',
 'is_simulated': True}

>>> samples = acr[0]["audio"].get_all_samples()
>>> samples
AudioSamples:
  data (shape): torch.Size([1, 269568])
  pts_seconds: 0.0
  duration_seconds: 8.424
  sample_rate: 32000

>>> waveform = samples.data        # torch.float32, shape [1, 269568]
>>> sr = samples.sample_rate       # 32000

>>> ccr[0]
{'sample_id': 'vmc2026-track1-dev-ccr_7233',
 'audio_a': <datasets.features._torchcodec.AudioDecoder object at 0x...>,
 'audio_b': <datasets.features._torchcodec.AudioDecoder object at 0x...>,
 'sample_rate': 16000,
 'duration': 7.41,
 'label': 0.625,
 'utterance_id': 'fileid_561',
 'system_id_a': '1348',
 'system_id_b': '1412',
 'language': 'hin',
 'speaker_id': 'spk_212',
 'text': 'है वह पूजा है तो मैं समझता हूं कि बड़े-बड़े लोग हैं या कुछ जानना चाहते हैं कि एक प्रधानमंत्री',
 'is_simulated': False}

>>> ccr[0]["audio_a"].get_all_samples()
AudioSamples:
  data (shape): torch.Size([1, 118560])
  pts_seconds: 0.0
  duration_seconds: 7.41
  sample_rate: 16000

Subgroup Analysis

Join predictions to the inline metadata by sample_id, then summarize performance for language, anonymized-speaker, or simulation-status subgroups:

python
import pandas as pd

dataset = acr  # or ccr
predictions = pd.read_csv("predictions.csv")
metadata = dataset.select_columns(
    ["sample_id", "language", "speaker_id", "is_simulated"]
).to_pandas()
analysis = predictions.merge(metadata, on="sample_id", validate="one_to_one")
subgroup_summary = analysis.groupby(
    ["language", "speaker_id", "is_simulated"]
)["pred_score"].agg(["count", "mean"])

Note

All audio is mono FLAC. Sample rates vary across samples (16 / 22.05 / 24 / 32 / 44.1 kHz); see the sample_rate column on the row, or samples.sample_rate after decoding.

The audios are from the subjective listening test data from the ICASSP 2026 URGENT Challenge.

bibtex
@inproceedings{urgent2026,
  title={{ICASSP 2026 URGENT Speech Enhancement Challenge}},
  author={Li, Chenda and Wang, Wei and Sach, Marvin and Zhang, Wangyou and Saijo, Kohei and Cornell, Samuele and Fu, Yihui and Ni, Zhaoheng and Fingscheidt, Tim and Watanabe, Shinji and Qian, Yanmin},
  booktitle={Proc. ICASSP 2026},
  year={2026}
}

License

CC-BY-4.0