General-Level/General-Bench-Closeset

1 2<div align="center">3<img src='https://cdn-uploads.huggingface.co/production/uploads/647773a1168cb428e00e9a8f/N8lP93rB6lL3iqzML4SKZ.png'  width=100px>4 5<h1 align="center"><b>On Path to Multimodal Generalist: General-Level and General-Bench</b></h1>6<p align="center">7<a href="https://generalist.top/">[📖 Project]</a>8<a href="https://generalist.top/leaderboard">[🏆 Leaderboard]</a>9<a href="https://arxiv.org/abs/2505.04620">[📄 Paper]</a>10<a href="https://huggingface.co/papers/2505.04620">[🤗 Paper-HF]</a>11<a href="https://huggingface.co/General-Level">[🤗 Dataset-HF]</a>12<a href="https://github.com/path2generalist/GeneralBench">[📝 Dataset-Github]</a>13</p>14 15<h1 align="center" style="color: red">Close Set of General-Bench</h1>16 17</div>18 19 20 21---22We divide our `General-Bench` into two settings: **`open`** and **`close`**.23 24 25This is the **`Close Set`**, where we release only the sample inputs—**without ground-truth answers**—for [🏆 `Leaderboard`](https://generalist.top/leaderboard) purpose.26To participate the leaderboard, please follow the detailed [instructions](https://generalist.top/document) to submit the evaluation results ([submission](https://generalist.top/submit)).27 28Final results will be updated on the [🏆 Leaderboard](https://generalist.top/leaderboard).29 30 31 32If you wish to use the General-Bench for open research purposes (e.g., experiments), please go for the 33[👉 **`Open Set`**](https://huggingface.co/datasets/General-Level/General-Bench-Openset), where the full ground-truth annotations of samples are provided.34 35 36 37 38 39 40 41 42---43 44##  📕 Table of Contents45 46- [✨ File Origanization Structure](#filestructure)47- [🍟 Usage](#usage)48- [🌐 General-Bench](#bench)49  - [🍕 Capabilities and Domians Distribution](#distribution)50- [🖼️ Image Task Taxonomy](#imagetaxonomy)51- [📽️ Video Task Taxonomy](#videotaxonomy)52- [📞 Audio Task Taxonomy](#audiotaxonomy)53- [💎 3D Task Taxonomy](#3dtaxonomy)54- [📚 Language Task Taxonomy](#languagetaxonomy)55 56 57 58 59 60---61 62<span id='filestructure'/>63 64# ✨✨✨ **File Origanization Structure**65 66Here is the organization structure of the file system:67 68```69General-Bench70├── Image71│   ├── comprehension72│   │   ├── Bird-Detection73│   │   │   ├── annotation.json74│   │   │   └── images75│   │   │       └── Acadian_Flycatcher_0070_29150.jpg76│   │   ├── Bottle-Anomaly-Detection77│   │   │   ├── annotation.json78│   │   │   └── images79│   │   └── ...80│   └── generation81│       └── Layout-to-Face-Image-Generation82│           ├── annotation.json83│           └── images84│           └── ...85├── Video86│   ├── comprehension87│   │   └── Human-Object-Interaction-Video-Captioning88│   │       ├── annotation.json89│   │       └── videos90│   │       └── ...91│   └── generation92│       └── Scene-Image-to-Video-Generation93│           ├── annotation.json94│           └── videos95│           └── ...96├── 3d97│   ├── comprehension98│   │   └── 3D-Furniture-Classification99│   │       ├── annotation.json100│   │       └── pointclouds101│   │       └── ...102│   └── generation103│       └── Text-to-3D-Living-and-Arts-Point-Cloud-Generation104│           ├── annotation.json105│           └── pointclouds106│           └── ...107├── Audio108│   ├── comprehension109│   │   └── Accent-Classification110│   │       ├── annotation.json111│   │       └── audios112│   │       └── ...113│   └── generation114│       └── Video-To-Audio115│           ├── annotation.json116│           └── audios117│           └── ...118├── NLP119│   ├── History-Question-Answering120│   │   └── annotation.json121│   ├── Abstractive-Summarization122│   │   └── annotation.json123│   └── ...124 125```126 127 128An illustrative example of file formats:129 130 131![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/RD3b7Jwu0dftVq-4KbpFr.png)132 133 134<span id='usage'/>135 136## 🍟🍟🍟 Usage 137 138Please download all the files in this repository. We also provide overview.json, which is an example of the format of our dataset.139 140For more instructions, please go to the [document page](https://generalist.top/document).141 142 143---144 145 146 147 148 149<span id='bench'/>150 151 152 153# 🌐🌐🌐 **General-Bench** 154 155 156 157 158A companion  massive multimodal benchmark dataset, encompasses a broader spectrum of skills, modalities, formats, and capabilities, including over **`700`** tasks and **`325K`** instances. 159 160<div align="center">161<img src='https://cdn-uploads.huggingface.co/production/uploads/647773a1168cb428e00e9a8f/d4TIWw3rlWuxpBCEpHYJB.jpeg'>162<p> Overview of General-Bench, which covers 145 skills for more than 700 tasks with over 325,800 samples under163comprehension and generation categories in various modalities</p>164</div>165 166 167<span id='distribution'/>168 169## 🍕🍕🍕 Capabilities and Domians Distribution170 171<div align="center">172<img src='https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/fF3iH95B3QEBvJYwqzZVG.png'>173<p> Distribution of various capabilities evaluated in General-Bench.</p>174</div>175 176 177<div align="center">178<img src='https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/wQvllVeK-KC3Edp8Zjh-V.png'>179<p>Distribution of various domains and disciplines covered by General-Bench.</p>180</div>181 182 183 184 185 186<span id='imagetaxonomy'/>187 188# 🖼️ Image Task Taxonomy189<div align="center">190<img src='https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/2QYihQRhZ5C9K5IbukY7R.png'>191<p>Taxonomy and hierarchy of data in terms of Image modality.</p>192</div>193 194 195 196 197<span id='videotaxonomy'/>198 199# 📽️ Video Task Taxonomy200 201<div align="center">202<img src='https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/A7PwfW5gXzstkDH49yIG5.png'>203<p>Taxonomy and hierarchy of data in terms of Video modality.</p>204</div>205 206 207 208 209 210 211 212 213 214<span id='audiotaxonomy'/>215 216# 📞 Audio Task Taxonomy217 218 219 220<div align="center">221<img src='https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/e-QBvBjeZy8vmcBjAB0PE.png'>222<p>Taxonomy and hierarchy of data in terms of Audio modality.</p>223</div>224 225 226 227<span id='3dtaxonomy'/>228 229# 💎 3D Task Taxonomy230 231 232<div align="center">233<img src='https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/EBXb-wyve14ExoLCgrpDK.png'>234<p>Taxonomy and hierarchy of data in terms of 3D modality.</p>235</div>236 237 238 239 240<span id='languagetaxonomy'/>241 242# 📚 Language Task Taxonomy243 244<div align="center">245<img src='https://cdn-uploads.huggingface.co/production/uploads/64c139d867eff857ea51caa8/FLfk3QGdYb2sgorKTj_LT.png'>246<p>Taxonomy and hierarchy of data in terms of Language modality.</p>247</div>248 249 250 251 252---253 254 255 256 257# 🚩🚩🚩 **Citation**258 259If you find this project useful to your research, please kindly cite our paper:260 261```262@articles{fei2025pathmultimodalgeneralistgenerallevel,263  title={On Path to Multimodal Generalist: General-Level and General-Bench},264  author={Hao Fei and Yuan Zhou and Juncheng Li and Xiangtai Li and Qingshan Xu and Bobo Li and Shengqiong Wu and Yaoting Wang and Junbao Zhou and Jiahao Meng and Qingyu Shi and Zhiyuan Zhou and Liangtao Shi and Minghe Gao and Daoan Zhang and Zhiqi Ge and Weiming Wu and Siliang Tang and Kaihang Pan and Yaobo Ye and Haobo Yuan and Tao Zhang and Tianjie Ju and Zixiang Meng and Shilin Xu and Liyu Jia and Wentao Hu and Meng Luo and Jiebo Luo and Tat-Seng Chua and Shuicheng Yan and Hanwang Zhang},265  eprint={2505.04620},266  archivePrefix={arXiv},267  primaryClass={cs.CV}268  url={https://arxiv.org/abs/2505.04620},269}270```