Datasetpublic

endlesswinds/AndroidControl-JSON-Splits

AndroidControl (JSON) Overview This dataset provides all splits of AndroidControl in a ready-to-use JSON format, extracted from the original Google Cloud data release. Motivation: Most GUI agent works use GUI-Actor-Data for training, which includes AndroidControl images but does not include the test set JSON. Obtaining any split from the original source requires downloading the full dataset (~47 GB) from Google Cloud, which is cumbersome. This dataset provides all… See the full description on the dataset page: https://huggingface.co/datasets/endlesswinds/AndroidControl-JSON-Splits.

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes48downloads

Dataset Card

AndroidControl (JSON)

Overview

This dataset provides all splits of AndroidControl in a ready-to-use JSON format, extracted from the original Google Cloud data release.

Motivation: Most GUI agent works use GUI-Actor-Data for training, which includes AndroidControl images but does not include the test set JSON. Obtaining any split from the original source requires downloading the full dataset (~47 GB) from Google Cloud, which is cumbersome. This dataset provides all splits as lightweight JSON files so researchers can use them directly for training or evaluation.

Images: Screenshots are not included in this repository. Use the images from cckevinn/GUI-Actor-Data, which covers all splits.

Files

File	Episodes	Steps	Description
`train-AndroidControl.json`	13,594	74,714	Train — all action types
`train-AndroidControl-click.json`	13,106	46,424	Train — click actions only
`validation-AndroidControl.json`	137	690	Validation — all action types
`validation-AndroidControl-click.json`	133	420	Validation — click actions only
`test-AndroidControl.json`	1,543	8,444	Test — all action types
`test-AndroidControl-click.json`	1,484	5,074	Test — click actions only

Data Format

Both files share the same structure: a list of episodes, where each episode is a list of steps.

json

[
  [                          // episode
    {                        // step
      "img_url": "android_control_episode_[27]_0.png",
      "img_size": [1080, 2400],
      "episode_id": 27,
      "goal": "Open the Zoho Meet app, view the scheduled meetings.",
      "instruction": "Open the Zoho Meet app",
      "action": {
        "action_type": "open_app",
        "app_name": "Zoho Meeting"
      }
    },
    ...
  ],
  ...
]

Fields

Field	Type	Description
`img_url`	string	Filename of the corresponding screenshot, format: `android_control_episode_[{episode_id}]_{step_idx}.png`
`img_size`	[int, int]	Screenshot resolution `[width, height]`, typically `[1080, 2400]`
`episode_id`	int	Episode ID from the original AndroidControl dataset
`goal`	string	High-level task goal in natural language
`instruction`	string	Step-level instruction
`action`	object	Ground-truth action (see below)

Action Types

`action_type`	Additional Fields	Description
`click`	`x`, `y` (normalized 0–1)	Tap on screen coordinates
`scroll`	`direction` (`up`/`down`/`left`/`right`)	Scroll gesture
`type`	`text`	Type text input
`navigate_back`	—	Press back button
`navigate_home`	—	Press home button
`open_app`	`app_name`	Launch an application
`wait`	—	Wait for screen to load

-click variants contain only click steps and are intended for point-grounding evaluation.

Usage

python

import json

with open("test-AndroidControl.json") as f:
    episodes = json.load(f)

# episodes[i]               → i-th episode (list of steps)
# episodes[i][j]            → j-th step of i-th episode
# episodes[i][j]["img_url"] → match against GUI-Actor-Data images
# episodes[i][j]["action"]  → ground-truth action for evaluation

Source

Derived from the AndroidControl dataset released by Google Research. All splits are reformatted into JSON; no images are redistributed.

Please cite the original work if you use this dataset:

bibtex

@article{li2024effects,
  title={On the Effects of Data Scale on Computer Control Agents},
  author={Li, Wei and Bishop, William and Li, Alice and Rawles, Chris and Campbell-Ajala, Folawiyo and Tyamagundlu, Divya and Riva, Oriana},
  journal={arXiv preprint arXiv:2406.03679},
  year={2024}
}

License

The original AndroidControl dataset is licensed under the Apache License 2.0:

Copyright 2024 The Google Research Authors. Licensed under the Apache License, Version 2.0. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

This repository redistributes a subset of that data in reformatted form and is therefore also released under the Apache License 2.0.