multilingual-vlm-conflict/pendulum-conflict

Pendulum Conflict Dataset

This dataset is a manually curated subset of 100 isolated multimodal conflict examples derived from the akomand/counterfactual_pendulum dataset.

It is designed for evaluating Multimodal Large Language Models (MLLMs) under controlled visual-textual conflicts.

Dataset Statistics

• —Total Samples: 100
• —Categories: angle (25), light (25), shadow_len (25), shadow_pos (25)
• —Language: English
• —Corrected / Audited Samples: 20

Data Fields

• —image: The visual pendulum scene.
• —file_name: Original image filename.
• —original_caption: The full 4-sentence truthful description of the image.
• —conflicting_caption: The full 4-sentence conflicting description containing one false attribute.
• —question: The evaluation question targeting the conflicting attribute.
• —image_bias: The correct answer option that aligns with the visual truth.
• —text_bias: The incorrect answer option that aligns with the flawed text.
• —distractor: A third answer option that matches neither the image nor the text.
• —serial_no: The sample index (1-100) in this curated subset.
• —conflict_type: The specific attribute in conflict (angle, light, shadow_len, shadow_pos).
• —language: The language of the text (English).
• —original_row_id: The row index from the original akomand/counterfactual_pendulum dataset.
• —original_text_bias: The original flawed text option from the paper (if patched).
• —is_patched: Boolean flag indicating if this sample was manually corrected in our audit.

Dataset Audit & Corrections

During extraction, we identified 20 instances in the original dataset where the author-provided counterfactuals were 'weak' (e.g. paraphrasing the truthful visual state rather than contradicting it).

We have rigorously patched these 20 instances to ensure strong structural opposition.

For a full breakdown of the errors in the original paper and our corrections, please refer to the [Audit Log](audit_log.md).

How to Reproduce

This dataset was programmatically extracted and audited.

1. 1.Source: akomand/counterfactual_pendulum (train split, 7391 rows).
2. 2.Filtering: We isolate the multi-sentence descriptions and split them into 4 distinct physical attributes. We extract rows where the base attribute conflicts with the counterfactual attribute.
3. 3.Audit Patching: Rows demonstrating weak semantic conflicts were overridden with explicitly contradictory statements sampled from the dataset pool.
4. 4.Script: The extraction and patching script (build_pendulum_dataset.py) is available in this repository.

To run the reproduction script locally:

pip install datasets
python build_pendulum_dataset.py