endlesswinds/AndroidControl-JSON-Splits

1# AndroidControl (JSON)2 3## Overview4 5This dataset provides **all splits** of [AndroidControl](https://arxiv.org/abs/2406.03679) in a ready-to-use JSON format, extracted from the original Google Cloud data release.6 7**Motivation:** Most GUI agent works use [GUI-Actor-Data](https://huggingface.co/datasets/cckevinn/GUI-Actor-Data) for training, which includes AndroidControl images but **does not include the test set JSON**. Obtaining any split from the original source requires downloading the full dataset (~47 GB) from Google Cloud, which is cumbersome. This dataset provides all splits as lightweight JSON files so researchers can use them directly for training or evaluation.8 9**Images:** Screenshots are not included in this repository. Use the images from [cckevinn/GUI-Actor-Data](https://huggingface.co/datasets/cckevinn/GUI-Actor-Data), which covers all splits.10 11---12 13## Files14 15| File | Episodes | Steps | Description |16|------|----------|-------|-------------|17| `train-AndroidControl.json` | 13,594 | 74,714 | Train — all action types |18| `train-AndroidControl-click.json` | 13,106 | 46,424 | Train — click actions only |19| `validation-AndroidControl.json` | 137 | 690 | Validation — all action types |20| `validation-AndroidControl-click.json` | 133 | 420 | Validation — click actions only |21| `test-AndroidControl.json` | 1,543 | 8,444 | Test — all action types |22| `test-AndroidControl-click.json` | 1,484 | 5,074 | Test — click actions only |23 24---25 26## Data Format27 28Both files share the same structure: a **list of episodes**, where each episode is a **list of steps**.29 30```json31[32  [                          // episode33    {                        // step34      "img_url": "android_control_episode_[27]_0.png",35      "img_size": [1080, 2400],36      "episode_id": 27,37      "goal": "Open the Zoho Meet app, view the scheduled meetings.",38      "instruction": "Open the Zoho Meet app",39      "action": {40        "action_type": "open_app",41        "app_name": "Zoho Meeting"42      }43    },44    ...45  ],46  ...47]48```49 50### Fields51 52| Field | Type | Description |53|-------|------|-------------|54| `img_url` | string | Filename of the corresponding screenshot, format: `android_control_episode_[{episode_id}]_{step_idx}.png` |55| `img_size` | [int, int] | Screenshot resolution `[width, height]`, typically `[1080, 2400]` |56| `episode_id` | int | Episode ID from the original AndroidControl dataset |57| `goal` | string | High-level task goal in natural language |58| `instruction` | string | Step-level instruction |59| `action` | object | Ground-truth action (see below) |60 61### Action Types62 63| `action_type` | Additional Fields | Description |64|---------------|------------------|-------------|65| `click` | `x`, `y` (normalized 0–1) | Tap on screen coordinates |66| `scroll` | `direction` (`up`/`down`/`left`/`right`) | Scroll gesture |67| `type` | `text` | Type text input |68| `navigate_back` | — | Press back button |69| `navigate_home` | — | Press home button |70| `open_app` | `app_name` | Launch an application |71| `wait` | — | Wait for screen to load |72 73> `-click` variants contain only `click` steps and are intended for point-grounding evaluation.74 75---76 77## Usage78 79```python80import json81 82with open("test-AndroidControl.json") as f:83    episodes = json.load(f)84 85# episodes[i]               → i-th episode (list of steps)86# episodes[i][j]            → j-th step of i-th episode87# episodes[i][j]["img_url"] → match against GUI-Actor-Data images88# episodes[i][j]["action"]  → ground-truth action for evaluation89```90 91---92 93## Source94 95Derived from the [AndroidControl](https://arxiv.org/abs/2406.03679) dataset released by Google Research. All splits are reformatted into JSON; no images are redistributed.96 97Please cite the original work if you use this dataset:98 99```bibtex100@article{li2024effects,101  title={On the Effects of Data Scale on Computer Control Agents},102  author={Li, Wei and Bishop, William and Li, Alice and Rawles, Chris and Campbell-Ajala, Folawiyo and Tyamagundlu, Divya and Riva, Oriana},103  journal={arXiv preprint arXiv:2406.03679},104  year={2024}105}106```107 108## License109 110The original AndroidControl dataset is licensed under the **Apache License 2.0**:111 112> Copyright 2024 The Google Research Authors.113> Licensed under the Apache License, Version 2.0. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0114 115This repository redistributes a subset of that data in reformatted form and is therefore also released under the **Apache License 2.0**.116