Datasetpublic

openbmb/CAGUI

CAGUI: Chinese Android GUI Benchmark A real-world Chinese Android GUI benchmark designed to evaluate GUI agent models on two complementary capabilities: Grounding – understanding individual GUI components and linking them to semantics. Agent – planning and executing multi-step actions to complete user goals on Chinese Android apps. See AgentCPM-GUI for more details. 🌟 Key Features Aspect Grounding Agent Objective GUI widgets grounding / OCR text… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/CAGUI.

sourceHugging Facecc-by-nc-4.0updated 1y agoView on Hugging Face

25likes6kdownloads

Dataset Card

CAGUI: Chinese Android GUI Benchmark

A real-world Chinese Android GUI benchmark designed to evaluate GUI agent models on two complementary capabilities:

—Grounding – understanding individual GUI components and linking them to semantics.
—Agent – planning and executing multi-step actions to complete user goals on Chinese Android apps.

See AgentCPM-GUI for more details.

🌟 Key Features

Aspect	Grounding	Agent
Objective	GUI widgets grounding / OCR text	Follow natural-language instructions to operate an app
Data	2 × JSONL files (`cap.jsonl`, `ocr.jsonl`) + screenshots	Per-episode JSON + step-level screenshots
Actions	n/a	Tap, long-tap, text-input, scroll, etc. (`result_action_type`, see here)

🗂️ Repository Structure

CAGUI
├─ CAGUI_agent
│  └─ domestic/
│     └─ <episode_id>/
│        ├─ <episode_id>.json       # episode file
│        ├─ <episode_id>_0.jpeg     # step-0 screenshot
│        ├─ <episode_id>_1.jpeg
│        └─ ...
└─ CAGUI_grounding
   ├─ code/
   │  ├─ cap.jsonl                  # function to point & text to point
   │  └─ ocr.jsonl                  # bbox to text
   └─ images/
      ├─ cap/
      │   ├─ 0.jpeg
      │   └─ ...
      └─ ocr/
          ├─ 0.jpeg
          └─ ...

📑 Data Format

1. Agent episodes (`CAGUI_agent/domestic/<episode_id>/<episode_id>.json`)

Each file is a list of steps:

Field	Type	Description
`episode_id`	str	Unique id
`episode_length`	int	Total steps
`step_id`	int	Step index (0-based)
`instruction`	str	user goal
`image_path`	str	Relative path to screenshot
`image_width` / `image_height`	int	Raw resolution
`ui_positions`	str (JSON list)	Normalised \[[y, x, h, w], …] UI element boxes
`result_action_type`	int	Action code, see here
`result_action_text`	str	Text typed (if any)
`result_touch_yx` / `result_lift_yx`	str	Normalised touch coords, `[-1,-1]` if no touch
`duration`	float \	null	Action time (s)

2. Grounding annotations (`cap.jsonl`, `ocr.jsonl`)

One JSON object per line:

Field	Example	Description
`task`	`"bbox2function"` / `"bbox2text"`	Sub-task type
`image`	`"grounding_eval/dataset/images/0.jpeg"`	Screenshot path
`id`	`0`	Unique int id
`abs_position`	`"<x1, y1, x2, y2>"`	Pixel-level bbox
`rel_position`	`"<x1, y1, x2, y2>"`	Normalised bbox
`text`	`"UI元素是一个菜单按钮…"`	Target description / OCR string

📜 License

CAGUI is released under the CC-BY-NC 4.0 license for non-commercial research. Screenshots originate from publicly available Chinese apps and are used under fair-use for research purposes only. Remove them if local regulations require.

✏️ Citation

This benchmark is used for evaluating AgentCPM-GUI. If our model and benchmark are useful for your research, please cite:

bibtex

@article{zhang2025agentcpmgui,
      title={Agent{CPM}-{GUI}: Building Mobile-Use Agents with Reinforcement Fine-Tuning}, 
      author={Zhong Zhang and Yaxi Lu and Yikun Fu and Yupeng Huo and Shenzhi Yang and Yesai Wu and Han Si and Xin Cong and Haotian Chen and Yankai Lin and Jie Xie and Wei Zhou and Wang Xu and Yuanheng Zhang and Zhou Su and Zhongwu Zhai and Xiaoming Liu and Yudong Mei and Jianming Xu and Hongyan Tian and Chongyi Wang and Chi Chen and Yuan Yao and Zhiyuan Liu and Maosong Sun},
      year={2025},
      journal={arXiv preprint arXiv:2506.01391},
}