macpaw-research/Screen2AX-Tree

---
license: apache-2.0
dataset_info:
features:
- name: image
dtype: image
- name: accessibility
dtype: string
splits:
- name: train
num_bytes: 653490753.125
num_examples: 1127
download_size: 618157948
dataset_size: 653490753.125
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
task_categories:
- object-detection
language:
- en
tags:
- accessibility
- macOS
- hierarchy
pretty_name: Screen2AX-Tree
size_categories:
- 1K<n<10K
---
# 📦 Screen2AX-Tree

Screen2AX-Tree is part of the **Screen2AX** dataset suite, a research-driven collection for advancing accessibility in macOS applications using computer vision and deep learning.

This dataset provides **hierarchical accessibility annotations** of macOS application screenshots, structured as serialized trees. It is designed for training models that reconstruct accessibility hierarchies from visual input.

---

## 🧠 Dataset Summary

Each sample in the dataset consists of:
- An application **screenshot** (`image`)
- A serialized **accessibility tree** (`accessibility`): A JSON-formatted string representing the UI structure, including roles, bounds, and child relationships.

**Task Category:**
- `object-detection` (structured / hierarchical)

**Language:**
- English (`en`)

---

## 📚 Usage

### Load with `datasets` library

```python
from datasets import load_dataset

dataset = load_dataset("macpaw-research/Screen2AX-Tree")
```

### Example structure

```python
sample = dataset["train"][0]
print(sample.keys())
# dict_keys(['image', 'accessibility'])

print(sample["accessibility"])
# '{ "role": "AXWindow", "children": [ ... ] }'
```

You can parse the accessibility field as JSON to work with the structured hierarchy:

```python
import json

tree = json.loads(sample["accessibility"])
```

---

## 📜 License

This dataset is licensed under the **Apache 2.0 License**.

---

## 🔗 Related Projects

- [Screen2AX Main Project Page](https://github.com/MacPaw/Screen2AX)
- [Screen2AX HuggingFace Collection](https://huggingface.co/collections/macpaw-research/screen2ax)

---

## ✍️ Citation

If you use this dataset, please cite the Screen2AX paper:

```bibtex
@misc{muryn2025screen2axvisionbasedapproachautomatic,
title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation},
author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai},
year={2025},
eprint={2507.16704},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2507.16704},
}
```

---

## 🌐 MacPaw Research

Learn more at [https://research.macpaw.com](https://research.macpaw.com)

---
license: apache-2.0
dataset_info:
features:
- name: 图像（image）
dtype: 图像
- name: 可访问性（accessibility）
dtype: 字符串
splits:
- name: 训练集（train）
num_bytes: 653490753.125
num_examples: 1127
download_size: 618157948 字节
dataset_size: 653490753.125 字节
configs:
- config_name: 默认（default）
data_files:
- split: 训练集
path: data/train-*
task_categories:
- 目标检测（object-detection）
language:
- 英语（en）
tags:
- 可访问性（accessibility）
- macOS
- 层级结构（hierarchy）
pretty_name: Screen2AX-Tree
size_categories:
- 1K<n<10K
---
# 📦 Screen2AX-Tree 数据集

Screen2AX-Tree 是 **Screen2AX** 数据集套件的组成部分，该套件是一项以研究为导向的集合，旨在借助计算机视觉与深度学习技术，推进macOS应用程序的可访问性优化。

本数据集提供macOS应用程序截图的**层级化可访问性标注**，以序列化树状结构的形式呈现，其设计目标是训练可从视觉输入重构可访问性层级结构的模型。

---

## 🧠 数据集摘要

本数据集的每条样本包含以下内容：
- 应用程序**截图（image）**：即`image`字段，对应应用界面的截图
- 序列化**可访问性树（accessibility）**：格式为JSON的字符串，用于表示用户界面（UI）结构，包含控件角色、边界范围与子级关联关系。

**任务类别：**
- 目标检测（结构化/层级化）

**语言：**
- 英语（`en`）

---

## 📚 使用方法

### 使用`datasets`库加载

python
from datasets import load_dataset

dataset = load_dataset("macpaw-research/Screen2AX-Tree")

### 样本结构示例

python
sample = dataset["train"][0]
print(sample.keys())
# dict_keys(['image', 'accessibility'])

print(sample["accessibility"])
# '{ "role": "AXWindow", "children": [ ... ] }'

你可以将`accessibility`字段解析为JSON格式，以处理该结构化层级结构：

python
import json

tree = json.loads(sample["accessibility"])

---

## 📜 许可证

本数据集采用**Apache 2.0许可证**进行授权。

---

## 🔗 相关项目

- [Screen2AX 主项目页面](https://github.com/MacPaw/Screen2AX)
- [Screen2AX Hugging Face 合集](https://huggingface.co/collections/macpaw-research/screen2ax)

---

## ✍️ 引用方式

若你使用本数据集，请引用如下Screen2AX论文：

bibtex
@misc{muryn2025screen2axvisionbasedapproachautomatic,
title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation},
author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai},
year={2025},
eprint={2507.16704},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2507.16704},
}

---

## 🌐 MacPaw 研究团队

了解更多信息请访问[https://research.macpaw.com](https://research.macpaw.com)