CoolFace
Datasetpublic

KZL96/ModalityFaultLines-SCEval

SCEval — Modality Fault Lines Data for Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning (Findings of EMNLP 2026). SCEval is a human-verified benchmark for omni-modal robustness. Text, vision, and audio all remain present, but controlled corruptions make the evidence inside a channel unreliable. Each corrupted item is paired with its clean counterpart at the example level, so clean-to-corrupted comparisons are made on the same underlying question… See the full description on the dataset page: https://huggingface.co/datasets/KZL96/ModalityFaultLines-SCEval.

sourceHugging Faceapache-2.0updated 26d agoView on Hugging Face
0likes475downloads
Dataset Card

SCEval — Modality Fault Lines

Data for Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning (Findings of EMNLP 2026).

SCEval is a human-verified benchmark for omni-modal robustness. Text, vision, and audio all remain present, but controlled corruptions make the evidence inside a channel unreliable. Each corrupted item is paired with its clean counterpart at the example level, so clean-to-corrupted comparisons are made on the same underlying question, answer options, and reference label.

Code, full documentation, and the evaluation protocol: <https://github.com/Zhaolu-K/ModalityFaultLines-SCEval>

What is in this repository

data/
├─ dataset_baseline.jsonl          273 clean tri-modal questions (the baseline reference)
├─ dataset_single.jsonl         29,144 single-modality corruptions (text / vision / audio)
├─ dataset_combined.jsonl       17,875 bimodal + trimodal joint corruptions
└─ dataset_*_preview.json          first 30 rows of each file, indented and field-trimmed
media/
├─ Social_IQ_single_question/    video + audio, 100 base examples
├─ omnibench/                    image + audio, 77 base examples
└─ valor/                        video + audio, 96 base examples

data/*.jsonl is one question per line and carries everything needed to evaluate: the question text, the answer options, the reference answer, the corruption metadata, and direct URLs to the media in media/. The media tree mirrors <source>/<sample_id>/<modality>/<operator>/<severity>/<variant>/<file>.

SplitRowsContents
baseline273Clean examples. Every corruption is compared against these.
single29,144One modality corrupted at a time, 14 operators × severity 10/30/50/70.
combined17,87518 bimodal + 11 trimodal preset conditions.

Loading

Metadata only (fast — media stays remote):

python
from datasets import load_dataset

baseline = load_dataset("KZL96/ModalityFaultLines-SCEval", "baseline", split="train")
single   = load_dataset("KZL96/ModalityFaultLines-SCEval", "single",   split="train")

row = single[0]
print(row["question"], row["options"], row["gold_key"])
print(row["visual_url"], row["audio_url"])   # resolve directly over HTTPS

Media, all of it (about 112 GB — pick a subfolder if you do not need everything):

python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="KZL96/ModalityFaultLines-SCEval",
    repo_type="dataset",
    allow_patterns=["media/omnibench/**"],   # drop this argument for the full tree
    local_dir="./SCEval",
)

Field reference

FieldMeaning
row_id, question_id, sample_idIdentifiers. sample_id ties a corrupted row to its clean counterpart.
datasetSource benchmark: Social_IQ_single_question, omnibench, or valor.
question, original_questionQuestion text after / before text corruption (identical unless the text channel was corrupted).
options, gold_key, gold_answer_rawAnswer options A0–A3, the correct key, and its full text.
visual_url, audio_urlMedia actually shown to the model for this row.
original_visual_url, original_audio_urlThe clean media for the same example.
corrupted_urlWhichever channel this row corrupts; equals visual_url or audio_url.
visual_kindvideo or image.
corruption_modalityWhich channel is corrupted (文本 text / 视觉 vision / 音频 audio).
corruption_group, corruption_typeOperator family and the specific operator.
severity_base10, 30, 50, or 70.
severity_variantDistinguishes multiple random realizations at one severity.
is_baselinetrue only in the baseline split.
condition_id, condition_labelcombined split only: which joint-corruption preset.
human_understandable, human_valid, human_must_three_modalHuman-verification outcomes (是 yes / 否 no).

Two conventions matter when reporting numbers. Corrupted rows are only interpretable against their same-sample_id baseline row, so the baseline split should be evaluated alongside any corruption split. And for stochastic operators, the multiple severity_variant realizations within one (model, modality, operator, severity) bucket are aggregated worst-case: if any variant is answered incorrectly, the bucket counts as incorrect.

Licensing and source media

This repository is released under Apache-2.0. The corrupted media is derived from three existing benchmarks, and the underlying source material remains subject to their original licenses and terms: Social-IQ, OmniBench, and VALOR. Please cite those datasets in addition to this work.

Citation

bibtex
@inproceedings{sceval2026,
  title     = {Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}
KZL96/ModalityFaultLines-SCEval · CoolFace