aplesner-eth/FLIP-Challenge

1---2language:3- en4tags:5- VLMs6- Reasoning7- Language8- Vision9- Image10- Understanding11pretty_name: FLIP Reasoning Challenge12---13 14 15# FLIP Reasoning Challenge Dataset16 17This repository contains the FLIP dataset, a benchmark for evaluating AI reasoning capabilities based on human verification tasks from the Idena blockchain. The dataset focuses on testing sequential reasoning, visual storytelling, and common sense understanding in multimodal AI systems.18 19Paper: https://arxiv.org/abs/2504.12256.20 21## Dataset Description22 23FLIP challenges present users with two orderings (stacks) of 4 images, requiring them to identify which ordering forms a coherent story. These tasks are designed to test complex reasoning abilities rather than simple recognition.24 25Key features of the FLIP dataset:26- Created from human-generated and human-verified tasks from the Idena blockchain27- Tests sequential reasoning and visual storytelling abilities28- Provides clear ground truth, making it easy to diagnose model failures29- High human performance baseline (95.3% accuracy)30 31## Dataset Structure and Overview32 33```34flip_dataset/35├── train/36│   ├── images/37│   │   ├── image1.png38│   │   ├── image2.png39│   │   └── ...40│   └── tasks/41│       ├── task1.json42│       ├── task2.json43│       └── ...44├── validation/45│   ├── images/46│   └── tasks/47└── test/48    ├── images/49    └── tasks/50```51---52dataset_info:53  features:54  - name: task_id55    dtype: string56  - name: task_data57    dtype: string58  - name: image_id59    dtype: string60  - name: image61    dtype: image62  splits:63  - name: train64    num_bytes: 1381093867.3765    num_examples: 3421066  - name: test67    num_bytes: 289834313.95868    num_examples: 735469  - name: validation70    num_bytes: 297405372.21671    num_examples: 731772  download_size: 183437664573  dataset_size: 1968333553.54474---75 76### Task Format77 78Each task is stored as a JSON file with the following structure:79 80```json81{82  "task_id": "_flip_bafkreianuvtem5nababzw5z4iscr5ocvgaviilmemwn3o73jkak7bqrjde",83  "images": {84    "0": "46efd91c-be17-42b8-8f5e-2a84b96d21af",85    "1": "9d1fac84-0c9f-4ab7-9d3b-a3b4c61dc390",86    "2": "ceecdc8b-840c-46d7-b694-74f05839447f",87    "3": "cbdf27d1-aa84-405b-86db-cb336d0bc4a7"88  },89  "left_stack": ["2", "3", "1", "0"],90  "right_stack": ["3", "0", "2", "1"],91  "agreed_answer": ["Right", "Strong"],92  "votes": {"Left": "1", "Right": "4", "Reported": "0"},93  "details": {94    "Author:": "0x63f7aa6C19A0f7D4BBB4177000Af671ED212e490",95    "Epoch:": "#0027",96    "Size:": "86140 bytes",97    "Created:": "12/24/2019 13:23:51",98    "Block:": "669858",99    "Tx:": "0xdbca60c3d10770f4bc2f73fd9119d9509117a8db08196f128382bffbf3d8c79f"100  }101}102```103 104When processing tasks:105- The task ID is derived from the `name` field by replacing "/" with "_"106- Image IDs are extracted by removing the prefix "blob:https://scan.idena.io/"107- The dataset stores the image orderings as "left stack" and "right stack"108- Images are shuffled to prevent any accidental ordering cues109 110## Dataset Statistics111 112- Total flips: 11,674113- Train set: 3,502 flips (30%)114- Validation set: 3,502 flips (30%)115- Test set: 4,670 flips (40%)116- Small subsets are also available for computationally intensive experimentation117 118Solutions are nearly evenly distributed between Left (49.4%) and Right (50.6%), with most challenges having strong consensus (95.7%).119 120## Research Findings121 122The FLIP dataset has been used to evaluate various state-of-the-art AI models:123 124- Best open-source models achieve 75.5% accuracy in zero-shot settings125- Best closed-source models reach 77.9% accuracy126- Human performance is 95.3% accurate127- Captioning models aid reasoning models by providing text descriptions128- Ensemble methods can boost performance to 85.2%129 130These findings highlight the gap between current AI capabilities and human-level reasoning on complex multimodal tasks.131 132## Citation133 134If you use this dataset in your research, please cite:135 136```137@inproceedings{plesner2025flip,138  title={FLIP Reasoning Challenge},139  author={Plesner, Andreas and Kuzhagaliyev, Turlan and Wattenhofer, Roger},140  booktitle={First Workshop on Open Science for Foundation Models at ICLR 2025},141  year={2025}142}143```144 145## Acknowledgements146 147This dataset is derived from the Idena blockchain. We thank the Idena community for creating and validating these challenges.148 149## Contact150 151For questions or feedback, please contact:152- Andreas Plesner (aplesner@ethz.ch)