FreedomIntelligence/MileBench

1---2license: cc-by-2.03dataset_info:4  features:5  - name: sample_id6    dtype: int327  - name: task_instruction8    dtype: string9  - name: task_instance10    struct:11    - name: context12      dtype: string13    - name: images_path14      sequence: string15    - name: choice_list16      sequence: string17    - name: combined_1_images18      sequence: string19  - name: response20    dtype: string21  splits:22  - name: ActionLocalization_test23    num_bytes: 29119924    num_examples: 20025  - name: ActionLocalization_adv26    num_bytes: 29119927    num_examples: 20028  - name: ActionPrediction_test29    num_bytes: 25568730    num_examples: 20031  - name: ActionPrediction_adv32    num_bytes: 25568733    num_examples: 20034  - name: ActionSequence_test35    num_bytes: 26223436    num_examples: 20037  - name: ActionSequence_adv38    num_bytes: 26223439    num_examples: 20040  - name: ALFRED_test41    num_bytes: 11271542    num_examples: 20043  - name: ALFRED_adv44    num_bytes: 11271545    num_examples: 20046  - name: CharacterOrder_test47    num_bytes: 27482148    num_examples: 20049  - name: CharacterOrder_adv50    num_bytes: 27482151    num_examples: 20052  - name: CLEVR_Change_test53    num_bytes: 11479254    num_examples: 20055  - name: CLEVR_Change_adv56    num_bytes: 11479257    num_examples: 20058  - name: CounterfactualInference_test59    num_bytes: 12907460    num_examples: 20061  - name: CounterfactualInference_adv62    num_bytes: 12907463    num_examples: 20064  - name: DocVQA_test65    num_bytes: 7666066    num_examples: 20067  - name: DocVQA_adv68    num_bytes: 7666069    num_examples: 20070  - name: EgocentricNavigation_test71    num_bytes: 55919372    num_examples: 20073  - name: EgocentricNavigation_adv74    num_bytes: 55919375    num_examples: 20076  - name: GPR1200_test77    num_bytes: 57962478    num_examples: 60079  - name: IEdit_test80    num_bytes: 5090781    num_examples: 20082  - name: IEdit_adv83    num_bytes: 5090784    num_examples: 20085  - name: ImageNeedleInAHaystack_test86    num_bytes: 30342387    num_examples: 32088  - name: MMCoQA_test89    num_bytes: 34462390    num_examples: 20091  - name: MMCoQA_adv92    num_bytes: 34462393    num_examples: 20094  - name: MovingAttribute_test95    num_bytes: 9729996    num_examples: 20097  - name: MovingAttribute_adv98    num_bytes: 9729999    num_examples: 200100  - name: MovingDirection_test101    num_bytes: 115832102    num_examples: 200103  - name: MovingDirection_adv104    num_bytes: 115832105    num_examples: 200106  - name: MultiModalQA_test107    num_bytes: 87978108    num_examples: 200109  - name: MultiModalQA_adv110    num_bytes: 87978111    num_examples: 200112  - name: nuscenes_test113    num_bytes: 87282114    num_examples: 200115  - name: nuscenes_adv116    num_bytes: 87282117    num_examples: 200118  - name: ObjectExistence_test119    num_bytes: 94139120    num_examples: 200121  - name: ObjectExistence_adv122    num_bytes: 94139123    num_examples: 200124  - name: ObjectInteraction_test125    num_bytes: 264032126    num_examples: 200127  - name: ObjectInteraction_adv128    num_bytes: 264032129    num_examples: 200130  - name: ObjectShuffle_test131    num_bytes: 289186132    num_examples: 200133  - name: ObjectShuffle_adv134    num_bytes: 289186135    num_examples: 200136  - name: OCR_VQA_test137    num_bytes: 80940138    num_examples: 200139  - name: OCR_VQA_adv140    num_bytes: 80940141    num_examples: 200142  - name: SceneTransition_test143    num_bytes: 266203144    num_examples: 200145  - name: SceneTransition_adv146    num_bytes: 266203147    num_examples: 200148  - name: SlideVQA_test149    num_bytes: 89462150    num_examples: 200151  - name: SlideVQA_adv152    num_bytes: 89462153    num_examples: 200154  - name: Spot_the_Diff_test155    num_bytes: 47823156    num_examples: 200157  - name: Spot_the_Diff_adv158    num_bytes: 47823159    num_examples: 200160  - name: StateChange_test161    num_bytes: 286783162    num_examples: 200163  - name: StateChange_adv164    num_bytes: 286783165    num_examples: 200166  - name: TextNeedleInAHaystack_test167    num_bytes: 11140730168    num_examples: 320169  - name: TQA_test170    num_bytes: 92861171    num_examples: 200172  - name: TQA_adv173    num_bytes: 92861174    num_examples: 200175  - name: WebQA_test176    num_bytes: 202682177    num_examples: 200178  - name: WebQA_adv179    num_bytes: 202682180    num_examples: 200181  - name: WikiVQA_test182    num_bytes: 2557847183    num_examples: 200184  - name: WikiVQA_adv185    num_bytes: 2557847186    num_examples: 200187  download_size: 12035444188  dataset_size: 26288285189configs:190- config_name: default191  data_files:192  - split: ActionLocalization_test193    path: preview/ActionLocalization_test-*194  - split: ActionLocalization_adv195    path: preview/ActionLocalization_adv-*196  - split: ActionPrediction_test197    path: preview/ActionPrediction_test-*198  - split: ActionPrediction_adv199    path: preview/ActionPrediction_adv-*200  - split: ActionSequence_test201    path: preview/ActionSequence_test-*202  - split: ActionSequence_adv203    path: preview/ActionSequence_adv-*204  - split: ALFRED_test205    path: preview/ALFRED_test-*206  - split: ALFRED_adv207    path: preview/ALFRED_adv-*208  - split: CharacterOrder_test209    path: preview/CharacterOrder_test-*210  - split: CharacterOrder_adv211    path: preview/CharacterOrder_adv-*212  - split: CLEVR_Change_test213    path: preview/CLEVR_Change_test-*214  - split: CLEVR_Change_adv215    path: preview/CLEVR_Change_adv-*216  - split: CounterfactualInference_test217    path: preview/CounterfactualInference_test-*218  - split: CounterfactualInference_adv219    path: preview/CounterfactualInference_adv-*220  - split: DocVQA_test221    path: preview/DocVQA_test-*222  - split: DocVQA_adv223    path: preview/DocVQA_adv-*224  - split: EgocentricNavigation_test225    path: preview/EgocentricNavigation_test-*226  - split: EgocentricNavigation_adv227    path: preview/EgocentricNavigation_adv-*228  - split: GPR1200_test229    path: preview/GPR1200_test-*230  - split: IEdit_test231    path: preview/IEdit_test-*232  - split: IEdit_adv233    path: preview/IEdit_adv-*234  - split: ImageNeedleInAHaystack_test235    path: preview/ImageNeedleInAHaystack_test-*236  - split: MMCoQA_test237    path: preview/MMCoQA_test-*238  - split: MMCoQA_adv239    path: preview/MMCoQA_adv-*240  - split: MovingAttribute_test241    path: preview/MovingAttribute_test-*242  - split: MovingAttribute_adv243    path: preview/MovingAttribute_adv-*244  - split: MovingDirection_test245    path: preview/MovingDirection_test-*246  - split: MovingDirection_adv247    path: preview/MovingDirection_adv-*248  - split: MultiModalQA_test249    path: preview/MultiModalQA_test-*250  - split: MultiModalQA_adv251    path: preview/MultiModalQA_adv-*252  - split: nuscenes_test253    path: preview/nuscenes_test-*254  - split: nuscenes_adv255    path: preview/nuscenes_adv-*256  - split: ObjectExistence_test257    path: preview/ObjectExistence_test-*258  - split: ObjectExistence_adv259    path: preview/ObjectExistence_adv-*260  - split: ObjectInteraction_test261    path: preview/ObjectInteraction_test-*262  - split: ObjectInteraction_adv263    path: preview/ObjectInteraction_adv-*264  - split: ObjectShuffle_test265    path: preview/ObjectShuffle_test-*266  - split: ObjectShuffle_adv267    path: preview/ObjectShuffle_adv-*268  - split: OCR_VQA_test269    path: preview/OCR_VQA_test-*270  - split: OCR_VQA_adv271    path: preview/OCR_VQA_adv-*272  - split: SceneTransition_test273    path: preview/SceneTransition_test-*274  - split: SceneTransition_adv275    path: preview/SceneTransition_adv-*276  - split: SlideVQA_test277    path: preview/SlideVQA_test-*278  - split: SlideVQA_adv279    path: preview/SlideVQA_adv-*280  - split: Spot_the_Diff_test281    path: preview/Spot_the_Diff_test-*282  - split: Spot_the_Diff_adv283    path: preview/Spot_the_Diff_adv-*284  - split: StateChange_test285    path: preview/StateChange_test-*286  - split: StateChange_adv287    path: preview/StateChange_adv-*288  - split: TextNeedleInAHaystack_test289    path: preview/TextNeedleInAHaystack_test-*290  - split: TQA_test291    path: preview/TQA_test-*292  - split: TQA_adv293    path: preview/TQA_adv-*294  - split: WebQA_test295    path: preview/WebQA_test-*296  - split: WebQA_adv297    path: preview/WebQA_adv-*298  - split: WikiVQA_test299    path: preview/WikiVQA_test-*300  - split: WikiVQA_adv301    path: preview/WikiVQA_adv-*302task_categories:303- visual-question-answering304- question-answering305- text-generation306- image-to-text307- video-classification308language:309- en310tags:311- Long-context312- MLLM313- VLM314- LLM315- Benchmark316pretty_name: MileBench317size_categories:318- 1K<n<10K319---320 321 322# MileBench323 324## Introduction325 326We introduce MileBench, a pioneering benchmark designed to test the **M**ult**I**modal **L**ong-cont**E**xt capabilities of MLLMs. 327This benchmark comprises not only multimodal long contexts, but also multiple tasks requiring both comprehension and generation. 328We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs’ long-context adaptation capacity and their ability to completetasks in long-context scenarios329 330 <img src="./images/MileBench.png" width="600" alt="MileBench" align="center" />331 332To construct our evaluation sets, we gather 6,440 multimodal long-context samples from 21 pre-existing or self-constructed datasets, 333with an average of 15.2 images and 422.3 words each, as depicted in the figure, and we categorize them into their respective subsets.334<center class="half">335  <img src="./images/stat2.png" width="300" alt="stat2"/><img src="./images/stat1.png" width="300" alt="stat1"/>336</center>337 338## How to use?339 340Please download MileBench_part*.tar.gz and unzip them using the following command. 341 342```bash343for file in MileBench_part*.tar.gz344do345  tar -xzvf "$file"346done347```348 349Then please refer to [Code for MileBench](https://github.com/MileBench/MileBench?tab=readme-ov-file#-dataset-preparation) to evaluate.350 351## Links352 353- **Homepage:** [MileBench Homepage](https://milebench.github.io/)354- **Repository:** [MileBench GitHub](https://github.com/MileBench/MileBench)355- **Paper:** [Arxiv](https://arxiv.org/abs/2404.18532)356- **Point of Contact:** [Dingjie Song](mailto:bbsngg@outlook.com)357 358## Citation359 360If you find this project useful in your research, please consider citing:361```BibTeX362@article{song2024milebench,363  title={MileBench: Benchmarking MLLMs in Long Context},364  author={Song, Dingjie and Chen, Shunian and Chen, Guiming Hardy and Yu, Fei and Wan, Xiang and Wang, Benyou},365  journal={arXiv preprint arXiv:2404.18532},366  year={2024}367}368```