Datasetpublic

Rakancorle1/hans-sft-4k

Hans-SFT-4K · SFT recipe for the audio-visual Clever Hans Supervised fine-tuning (SFT) data accompanying the paper When Vision Speaks for Sound. Like the original Clever Hans — the horse that looked like he could do arithmetic but was actually reading his trainer's body language — video-capable MLLMs often look like they can hear: they answer audio questions by reading visual cues and never verifying the audio stream. Hans-SFT-4K is the 3,834-sample SFT mix that teaches models… See the full description on the dataset page: https://huggingface.co/datasets/Rakancorle1/hans-sft-4k.

sourceHugging Facecc-by-nc-4.0updated 4mo agoView on Hugging Face

1likes24downloads

Dataset Card

Hans-SFT-4K · SFT recipe for the audio-visual Clever Hans

Supervised fine-tuning (SFT) data accompanying the paper [When Vision Speaks for Sound](https://arxiv.org/abs/2605.16403).

Like the original Clever Hans — the horse that looked like he could do arithmetic but was actually reading his trainer's body language — video-capable MLLMs often look like they can hear: they answer audio questions by reading visual cues and never verifying the audio stream.

Hans-SFT-4K is the 3,834-sample SFT mix that teaches models to honestly listen for audio temporal cues (Shift-style interventions), producing the SFT checkpoint that the larger preference dataset Hans-10K further aligns via DPO.

What's in this repo

File	Description	Size
`train.json`	3,834 SFT records	~3 MB
`sft_train_media.zip`	All referenced videos + audios	~7 GB

How to use

bash

# 1. Download
hf download Rakancorle1/hans-sft-4k --repo-type=dataset --local-dir hans-sft

cd hans-sft

# 2. Unzip media — paths inside train.json resolve automatically
unzip -q sft_train_media.zip

# After unzipping, you'll have:
#   videos/original/<file>.mp4    (1,278 files)
#   videos/shifted/<file>.mp4     (2,556 files)
#   audios/original/<file>.wav
#   audios/shifted/<file>.wav

python

from datasets import load_dataset

ds = load_dataset("Rakancorle1/hans-sft-4k")

Composition

3,834 SFT records over 1,278 unique source clips, each clip seen in three conditions (1 original + 2 audio-shifted versions: early-shifted + delay-shifted). The chosen response describes the actual audio-visual alignment honestly:

—original clips → "the audio is aligned with the visual event"
—shifted clips → "the audio is offset by ~X seconds, the visual moment does not line up"

Schema

Each record in train.json:

jsonc

{
  "id": "sft_000001",
  "messages": [
    { "role": "user",      "content": "<video><audio>...question..." },
    { "role": "assistant", "content": "...honest answer..." }
  ],
  "videos": ["videos/{original|shifted}/<file>.mp4"],
  "audios": ["audios/{original|shifted}/<file>.wav"]
}

Relation to other releases

—[Rakancorle1/hans-10k](https://huggingface.co/datasets/Rakancorle1/hans-10k) — the 10,383-sample DPO preference data used after SFT to further cure the Clever Hans effect.
—The two datasets share the same source video corpus for the THUD intervention subset; you don't need to download both to access videos, but each dataset is self-contained for convenience.

Citation

bibtex

@article{wen2026whenvisionspeaksforsound,
  title     = {When Vision Speaks for Sound},
  author    = {Xiaofei Wen and Wenjie Jacky Mo and Xingyu Fu and Rui Cai and
               Tinghui Zhu and Wendi Li and Yanan Xie and Muhao Chen and Peng Qi},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.16403}
}

License

Released under CC-BY-NC-4.0 for the annotations. Underlying media (UAG-Oops style clips) retains its respective upstream license — please consult the original source datasets when using or redistributing.