yuecao0119/AndroidLens

AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents

AndroidLens is a challenging benchmark for mobile GUI agents, featuring 571 real-world, long-horizon tasks in both Chinese and English, with an average of 26.1 steps per task. It supports evaluation of critical capabilities:

• —Long-horizon planning under multi-constraint & multi-goal scenarios
• —298 cross-app tasks and 273 single-app tasks, covering 74 real-world applications (e.g., WeChat, Google Drive, Taobao, Maps).
• —Robustness to real anomalies: ads, permission pop-ups, login redirects
• —Multi-trajectory ground truth to reduce path bias
• —Milestone-based progress tracking via nested sub-targets

This dataset is designed for both static (step-wise prediction) and dynamic (real-device execution) evaluation.

📄 Paper: AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents 💾 GitHub: https://github.com/alibaba/AndroidLens 🤗 Hugging Face: yuecao0119/AndroidLens

🗂️ Dataset Structure

Your data is organized as:

test/
├─ en/                        # English tasks
│  └─ <episode_id>/
│     ├─ <episode_id>.json   # Full episode trajectory (list of steps)
│     ├─ <episode_id>_0.png  # Screenshot at step 0
│     ├─ <episode_id>_1.png
│     └─ ...
└─ zh/                        # Chinese tasks
   └─ <episode_id>/
      ├─ <episode_id>.json
      ├─ <episode_id>_0.png
      └─ ...

Each <episode_id>.json file contains a list of step objects, with one object per interaction step.

🏷️ Task Category Codes (types)

The types field uses a hierarchical two-digit code system to classify task complexity and structure. These categories align with AndroidLens’s taxonomy of Multi-goal (1-X), Multi-constraint (2-X), and Domain-specific (3-X) tasks, enabling fine-grained analysis of agent performance across different challenge dimensions.

📑 Step-level Data Format

Each step in the JSON list includes:

🔍 Coordinate Note: - result_touch_yx uses relative coordinates in [0, 1], with format "[y, x]" (note: y first). - To convert to absolute pixel: ``python y_abs = float(y_rel) * image_height x_abs = float(x_rel) * image_width ``

🎯 Milestone Format

The milestone field enables fine-grained progress evaluation:

{
  "sub-target": "Open Google Chrome and search for 'panda'",
  "idx": 1,
  "bbox": [0.023, 0.121, 0.976, 0.189],  // [x1, y1, x2, y2] in normalized coords
  "text": "panda",
  "state": ["selected"]
}

• —idx: milestone index (ordered)
• —bbox: bounding box of key UI element (normalized, xy format)
• —text / state: expected content or widget state

Milestones support ordered and unordered sub-goals for complex tasks.

📊 Action Type Mapping

Although the original AgentCPM-GUI defines actions via names, your data uses numeric codes in result_action_type. Based on AndroidLens annotation practice, the common mapping is:

Confirm exact mapping from your annotation code if needed. AndroidLens uses ADB-based actions with explicit start/end for swipe.

📜 License

AndroidLens is released under the Apache-2.0 License. Screenshots are derived from real app usage for research purposes only. Comply with app store policies and local regulations.

✏️ Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@article{cao2025androidlens,
    title={AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents}, 
    author={Yue Cao and Yingyao Wang and Pi Bu and Jingxuan Xing and Wei Jiang and Zekun Zhu and Junpeng Ma and Sashuai Zhou and Tong Lu and Jun Song and Yu Cheng and Yuning Jiang and Bo Zheng},
    year={2025},
    journal={arXiv preprint arXiv:2512.21302},
}