Datasetpublic

OmniGUI/OmniGUI

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments [🍎 Project Page] [📖 arXiv Paper] [📊 Dataset] [💻 GitHub] [🏆 Leaderboard] OmniGUI is a step-level GUI agent benchmark designed for omni-modal smartphone interaction. At each action step, the agent receives interleaved multimodal observations, including static screenshots, synchronous audio cues, short video clips, and action history, and must predict the next GUI action such as TAP or TYPE. The… See the full description on the dataset page: https://huggingface.co/datasets/OmniGUI/OmniGUI.

sourceHugging Facecc-by-nc-sa-4.0updated 4mo agoView on Hugging Face

3likes1.8kdownloads

Dataset Card

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

<font size=4><div align='center' > [🍎 Project Page] [📖 arXiv Paper] [📊 Dataset] [💻 GitHub] [🏆 Leaderboard] </div></font>

OmniGUI is a step-level GUI agent benchmark designed for omni-modal smartphone interaction. At each action step, the agent receives interleaved multimodal observations, including static screenshots, synchronous audio cues, short video clips, and action history, and must predict the next GUI action such as TAP or TYPE.

The benchmark contains 708 expert-demonstrated episodes and 2,572 action steps across 29 real smartphone applications in both Chinese and English. OmniGUI evaluates five core capabilities: Localization, Semantic Understanding, Cross-modal Discrimination, Temporal Reasoning, and Instant Response.

🚀 Quick Start

Download the dataset

You can download the dataset using the huggingface-cli:

bash

huggingface-cli download XIAOCHENLIN00zz/OmniGUI \
  --repo-type dataset \
  --local-dir data

After downloading, your local data directory should contain:

—data/merged_sorted.json
—data/benchmark/

🛠️ APP STATISTICS

📂 Chinese Applications (ZH)

App Name	Ep	Stp	Loc	Sem	Cro	Tem	Ins
Bilibili	29	89	7	5	6	6	5
Douyin	29	97	7	5	6	6	5
Meituan	26	103	6	5	6	5	4
WeChat	25	84	7	5	4	5	4
DiDi	25	92	5	5	5	5	5
Amap	25	64	5	5	5	5	5
JD	25	80	5	5	5	5	5
PDD	25	72	5	5	5	5	5
QQ Music	25	95	5	5	5	5	5
Taobao	21	78	5	5	2	4	5
Weibo	25	106	5	5	5	5	5
RedNote	25	99	5	5	5	5	5
Kuaishou	24	120	4	5	5	6	4
iQIYI	20	56	0	5	0	12	3
Alipay	14	64	5	0	5	4	0
SUBTOTAL	363	1299	76	70	69	83	65

📂 English Applications (EN)

App Name	Ep	Stp	Loc	Sem	Cro	Tem	Ins
Duolingo	35	133	5	5	5	15	5
GTrans	25	88	5	5	5	5	5
Snapchat	25	104	5	5	5	5	5
Spotify	25	91	5	5	5	5	5
Tasty	25	69	5	5	10	5	0
TED	25	115	5	5	5	5	5
TikTok	25	77	5	5	5	5	5
Vimeo	25	86	5	5	5	5	5
X	25	61	5	5	5	5	5
YouTube	25	123	5	5	5	5	5
RedBull	24	110	5	4	5	5	5
Amazon	22	69	5	5	4	4	4
IMDb	20	95	5	4	4	3	4
Insta	19	52	4	4	4	1	6
SUBTOTAL	345	1273	69	67	72	73	64

📈 FINAL SUMMARY (TOTAL)

Total	Ep	Stp	Loc	Sem	Cro	Tem	Ins
ALL APPS	708	2572	145	137	141	156	129

Citation

Feel free to cite the following article if you find OmniGUI helpful:

latex

@article{henry2026omnigui,
  title={OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments},
  author={Henry, Felix and Lin, Xiaochen and Zhu, Jiangyou and Zhang, Bingqian and Chen, Min and Huang, Shiyu and others},
  journal={arXiv preprint arXiv:2605.18758},
  year={2026}
}