leosltl/Android-in-the-Wild

---
license: cc-by-4.0
task_categories:
- image-classification
- visual-question-answering
tags:
- android
- mobile
- ui-automation
- screen-understanding
pretty_name: Android in the Wild (AITW)
size_categories:
- 100M<n<1B
---

# Android in the Wild (AITW)

This is a mirror of Google's **Android in the Wild (AITW)** dataset, re-hosted on Hugging Face for easier community access.

## Original Source

- **Paper:** [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://arxiv.org/abs/2307.10088)
- **Original Repository:** [google-research/google-research/tree/master/android_in_the_wild](https://github.com/google-research/google-research/tree/master/android_in_the_wild)

## Dataset Description

Android in the Wild (AITW) is a large-scale dataset for Android device control. It contains human demonstrations of natural language instructions being carried out on Android devices. Each demonstration consists of a sequence of screenshots paired with corresponding actions (taps, swipes, types, etc.) and UI annotations.

## Dataset Structure

The dataset is organized into 5 subsets, stored as **gzip-compressed TFRecord** files:

| Subset | Shards | Description |
|--------|--------|-------------|
| `general` | 321 | General Android tasks |
| `google_apps` | 8,688 | Tasks on Google applications |
| `install` | 1,052 | App installation tasks |
| `single` | 252 | Single-step tasks |
| `web_shopping` | 1,025 | Web shopping tasks |

Additionally, the `splits/` directory contains JSON files defining train/test splits:
- `standard.json`
- `unseen_android_version.json`
- `unseen_domain.json`
- `unseen_subject.json`
- `unseen_verb.json`

## Data Format

Each TFRecord contains examples with the following fields:
- `image/encoded` — screenshot image (encoded)
- `image/ui_annotations_ui_types` — UI element type annotations (e.g., `ICON_STOP`, `ICON_V_BACKWARD`)
- Additional action and metadata fields

## Usage

```python
import tensorflow as tf
import gzip

def read_tfrecord(file_path):
with gzip.open(file_path, 'rb') as f:
raw = f.read()
dataset = tf.data.TFRecordDataset([file_path], compression_type='GZIP')
return dataset
```

For detailed usage instructions, refer to the [original repository](https://github.com/google-research/google-research/tree/master/android_in_the_wild).

## License

This dataset is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), following the original release.

## Citation

```bibtex
@article{rawles2023android,
title={Android in the Wild: A Large-Scale Dataset for Android Device Control},
author={Rawles, Christopher and Li, Alice and Rodriguez, Daniel and Ber, Oriana and Zitkovich, Brianna},
journal={arXiv preprint arXiv:2307.10088},
year={2023}
}
```

## Disclaimer

This is an unofficial mirror. All credit goes to the original authors at Google Research. This copy is provided solely to facilitate easier access for the research community.