macpaw-research/Screen2AX-Tree

---
license: apache-2.0
dataset_info:
features:
- name: image
dtype: image
- name: accessibility
dtype: string
splits:
- name: train
num_bytes: 653490753.125
num_examples: 1127
download_size: 618157948
dataset_size: 653490753.125
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
task_categories:
- object-detection
language:
- en
tags:
- accessibility
- macOS
- hierarchy
pretty_name: Screen2AX-Tree
size_categories:
- 1K<n<10K
---
# 📦 Screen2AX-Tree

Screen2AX-Tree is part of the **Screen2AX** dataset suite, a research-driven collection for advancing accessibility in macOS applications using computer vision and deep learning.

This dataset provides **hierarchical accessibility annotations** of macOS application screenshots, structured as serialized trees. It is designed for training models that reconstruct accessibility hierarchies from visual input.

---

## 🧠 Dataset Summary

Each sample in the dataset consists of:
- An application **screenshot** (`image`)
- A serialized **accessibility tree** (`accessibility`): A JSON-formatted string representing the UI structure, including roles, bounds, and child relationships.

**Task Category:**
- `object-detection` (structured / hierarchical)

**Language:**
- English (`en`)

---

## 📚 Usage

### Load with `datasets` library

```python
from datasets import load_dataset

dataset = load_dataset("macpaw-research/Screen2AX-Tree")
```

### Example structure

```python
sample = dataset["train"][0]
print(sample.keys())
# dict_keys(['image', 'accessibility'])

print(sample["accessibility"])
# '{ "role": "AXWindow", "children": [ ... ] }'
```

You can parse the accessibility field as JSON to work with the structured hierarchy:

```python
import json

tree = json.loads(sample["accessibility"])
```

---

## 📜 License

This dataset is licensed under the **Apache 2.0 License**.

---

## 🔗 Related Projects

- [Screen2AX Main Project Page](https://github.com/MacPaw/Screen2AX)
- [Screen2AX HuggingFace Collection](https://huggingface.co/collections/macpaw-research/screen2ax)

---

## ✍️ Citation

If you use this dataset, please cite the Screen2AX paper:

```bibtex
@misc{muryn2025screen2axvisionbasedapproachautomatic,
title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation},
author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai},
year={2025},
eprint={2507.16704},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2507.16704},
}
```

---

## 🌐 MacPaw Research

Learn more at [https://research.macpaw.com](https://research.macpaw.com)