leosltl/Android-in-the-Wild

Android in the Wild (AITW)

This is a mirror of Google's Android in the Wild (AITW) dataset, re-hosted on Hugging Face for easier community access.

Original Source

• —Paper: Android in the Wild: A Large-Scale Dataset for Android Device Control
• —Original Repository: google-research/google-research/tree/master/android_in_the_wild

Dataset Description

Android in the Wild (AITW) is a large-scale dataset for Android device control. It contains human demonstrations of natural language instructions being carried out on Android devices. Each demonstration consists of a sequence of screenshots paired with corresponding actions (taps, swipes, types, etc.) and UI annotations.

Dataset Structure

The dataset is organized into 5 subsets, stored as gzip-compressed TFRecord files:

Additionally, the splits/ directory contains JSON files defining train/test splits:

• —standard.json
• —unseen_android_version.json
• —unseen_domain.json
• —unseen_subject.json
• —unseen_verb.json

Data Format

Each TFRecord contains examples with the following fields:

• —image/encoded — screenshot image (encoded)
• —image/ui_annotations_ui_types — UI element type annotations (e.g., ICON_STOP, ICON_V_BACKWARD)
• —Additional action and metadata fields

Usage

import tensorflow as tf
import gzip

def read_tfrecord(file_path):
    with gzip.open(file_path, 'rb') as f:
        raw = f.read()
    dataset = tf.data.TFRecordDataset([file_path], compression_type='GZIP')
    return dataset

For detailed usage instructions, refer to the original repository.

License

This dataset is licensed under CC BY 4.0, following the original release.

Citation

@article{rawles2023android,
  title={Android in the Wild: A Large-Scale Dataset for Android Device Control},
  author={Rawles, Christopher and Li, Alice and Rodriguez, Daniel and Ber, Oriana and Zitkovich, Brianna},
  journal={arXiv preprint arXiv:2307.10088},
  year={2023}
}

Disclaimer

This is an unofficial mirror. All credit goes to the original authors at Google Research. This copy is provided solely to facilitate easier access for the research community.