Cie1/MMSearch-Plus

1---2task_categories:3- question-answering4- visual-question-answering5language:6- en7tags:8- Multimodal Search9- Multimodal Long Context10size_categories:11- n<1K12configs:13- config_name: default14  data_files:15  - split: train16    path: '*.arrow'17dataset_info:18  features:19  - name: question20    dtype: string21  - name: answer22    sequence: string23  - name: num_images24    dtype: int6425  - name: arxiv_id26    dtype: string27  - name: video_url28    dtype: string29  - name: category30    dtype: string31  - name: difficulty32    dtype: string33  - name: subtask34    dtype: string35  - name: img_136    dtype: image37  - name: img_238    dtype: image39  - name: img_340    dtype: image41  - name: img_442    dtype: image43  - name: img_544    dtype: image45  splits:46  - name: train47    num_examples: 31148license: cc-by-4.049---50# MMSearch-Plus✨: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents51 52Official repository for the paper "[MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents](https://arxiv.org/abs/2508.21475)".53 54🌟 For more details, please refer to the project page with examples: [https://mmsearch-plus.github.io/](https://mmsearch-plus.github.io).55 56 57[[🌐 Webpage](https://mmsearch-plus.github.io/)] [[📖 Paper](https://arxiv.org/pdf/2508.21475)] [[🤗 Huggingface Dataset](https://huggingface.co/datasets/Cie1/MMSearch-Plus)] [[🏆 Leaderboard](https://mmsearch-plus.github.io/#leaderboard)]58 59 60## 💥 News61 62- **[2025.09.26]** 🔥 We update the [arXiv paper](https://arxiv.org/abs/2508.21475) and release all MMSearch-Plus data samples in [huggingface dataset](https://huggingface.co/datasets/Cie1/MMSearch-Plus).63- **[2025.08.29]** 🚀 We release the [arXiv paper](https://arxiv.org/abs/2508.21475).64 65## 📌 ToDo66 67- Agentic rollout framework code68- Evaluation script69- Set-of-Mark annotations70 71## Usage72 73**⚠️ Important: This dataset is encrypted to prevent data contamination. However, decryption is handled transparently by the dataset loader.**74 75### Dataset Usage76 77For better compatibility with newer versions of the datasets library, we provide explicit decryption functions, downloadable from our GitHub/HF repo.78 79```bash80wget https://raw.githubusercontent.com/mmsearch-plus/MMSearch-Plus/main/decrypt_after_load.py81```82 83```python84import os85from datasets import load_dataset86from decrypt_after_load import decrypt_dataset87 88encrypted_dataset = load_dataset("Cie1/MMSearch-Plus", split='train')89decrypted_dataset = decrypt_dataset(90    encrypted_dataset=encrypted_dataset,91    canary='your_canary_string' # Set the canary string (hint: it's the name of this repo without username)92)93 94# Access a sample95sample = decrypted_dataset[0]96print(f"Question: {sample['question']}")97print(f"Answer: {sample['answer']}")98print(f"Category: {sample['category']}")99print(f"Number of images: {sample['num_images']}")100 101# Access images (PIL Image objects)102sample['img_1'].show()  # Display the first image103```104 105## 👀 About MMSearch-Plus106 107MMSearch-Plus is a challenging benchmark designed to test multimodal browsing agents' ability to perform genuine visual reasoning. Unlike existing benchmarks where many tasks can be solved with text-only approaches, MMSearch-Plus requires models to extract and use fine-grained visual cues through iterative image-text retrieval.108 109### Key Features110 111🔍 **Genuine Multimodal Reasoning**: 311 carefully curated tasks that cannot be solved without visual understanding112 113🎯 **Fine-grained Visual Analysis**: Questions require extracting spatial cues and temporal traces from images to find out-of-image facts like events, dates, and venues114 115🛠️ **Agent Framework**: Model-agnostic web agent with standard browsing tools (text search, image search, zoom-in)116 117📍 **Set-of-Mark (SoM) Module**: Enables provenance-aware cropping and targeted searches with human-verified bounding box annotations118 119### Dataset Structure120 121Each sample contains:122- Quuestion text and images123- Ground truth answers and alternative valid responses124- Metadata including arXiv id (if an event is a paper), video URL (if an event is a video), area and subfield125 126### Performance Results127 128Evaluation of closed- and open-source MLLMs shows:129- Best accuracy is achieved by o3 with full rollout: **36.0%** (indicating significant room for improvement)130- SoM integration provides consistent gains up to **+3.9 points**131- Models struggle with multi-step visual reasoning and cross-modal information integration 132 133<p align="center">134    <img src="https://raw.githubusercontent.com/mmsearch-plus/mmsearch-plus.github.io/main/static/images/teaser.png" width="80%"> <br>135    The overview of three paradigms for multimodal browsing tasks that demand fine-grained visual reasoning.136</p>137 138 139 140<p align="center">141    <img src="https://raw.githubusercontent.com/mmsearch-plus/mmsearch-plus.github.io/main/static/images/real-teaser.jpg" width="80%"> <br>142    The overview of an example trajectory for a task in <b>MMSearch-Plus</b>.143</p>144 145## 🏆 Leaderboard146 147### Contributing to the Leaderboard148 149🚨 The [Leaderboard](https://mmsearch-plus.github.io/#leaderboard) is continuously being updated, welcoming the contribution of your excellent LMMs!150 151 152## 🔖 Citation153 154If you find **MMSearch-Plus** useful for your research and applications, please kindly cite using this BibTeX:155 156```latex157@article{tao2025mmsearch,158  title={MMSearch-Plus: A Simple Yet Challenging Benchmark for Multimodal Browsing Agents},159  author={Tao, Xijia and Teng, Yihua and Su, Xinxing and Fu, Xinyu and Wu, Jihao and Tao, Chaofan and Liu, Ziru and Bai, Haoli and Liu, Rui and Kong, Lingpeng},160  journal={arXiv preprint arXiv:2508.21475},161  year={2025}162}163```