macpaw-research/UiPad

---
license: mit
task_categories:
- question-answering
language:
- en
- uk
tags:
- Screen Representation
- MacOS
- UI
pretty_name: UI Parsing and Accessibility Dataset (UiPad)
size_categories:
- 1K<n<10K
---

# UiPad - UI Parsing and Accessibility Dataset

- Curated by: [MacPaw Way Ltd.](https://huggingface.co/MacPaw)
- Language(s): Mostly EN, UA
- License: MIT

**Overview**
UiPad is a dataset created for the [IASA Champ 2024 Challenge](http://champ.iasa.kpi.ua/), focusing on the accessibility and interface understanding of MacOS applications. With growing interest in AI-driven user interface analysis, the dataset aims to bridge the gap in available resources for desktop app accessibility. While mobile apps and web platforms benefit from datasets like RICO and Mind2Web, MacOS apps remain mostly underexplored, particularly regarding accessibility parsing and textual representation.


## Dataset Structure
UiPad contains 352 unique screens from 63 different MacOS applications. Of these, 68% include accessibility data in the form of JSON trees. A screenshot accompanies each app screen and, if available, a JSON file detailing the accessibility elements.

```plaintext
dataset
│
├── application/
│ ├── screen_state_id (no accessibility) /
│ │ ├── screenshot.png
│ ├── screen_state_id (with accessibility) /
│ │ ├── screenshot.png
│ │ ├── accessibility_tree.json
│ ...
```

**Screenshot**
PNG image of the app screen

**Accessibility Tree Data**
The accessibility tree captures essential UI elements such as:
- `name`: Element name
- `role`: The role of the UI element (e.g., button, image)
- `description` and `role_description`
- `value`: Element state or value
- `children`: Nested UI components
- `bbox` and `visible_bbox`: Bounding box coordinates of elements


**Questions (for evaluation)**

The dataset includes several types of questions to evaluate UI understanding:
- Numeric: "How many checkboxes are checked on the screen?" (485 instances)
- Yes/No: "Is there a '+' button on the screen?" (306 instances)
- String: "What is the name of the app on the screen?" (143 instances)
- Coordinate: "Where do I click to connect Gmail?" (122 instances)

The dataset provides real-world challenges in accessibility recognition. Some screens may lack full accessibility support, with common issues like misidentifying roles (e.g., a button as an image), inaccurate bboxes or missing selected states.


**Task and Objectives**

UiPad's primary goal is to create an AI agent that understands and enhances UI accessibility in MacOS scenarios. The quality of the generated UI representation and the effectiveness of the AI agent are measured using Question Answering tasks related to UI understanding.

**Limitations and Challenges**

- Accessibility data may be incomplete, redundant or missing.
- The dataset size is limited, which may not be sufficient for training models from scratch.
- Human labelling of the Q/A introduces the potential for errors.

**Dataset Card Contact**

Feel free to reach out [e-mail] if you have any questions or need further information about the dataset!

license: MIT协议
task_categories:
- 问答
language:
- 英语
- 乌克兰语
tags:
- 屏幕表征（Screen Representation）
- MacOS
- 用户界面（UI）
pretty_name: UI解析与可访问性数据集（UiPad）
size_categories:
- 1000<n<10000
---

# UiPad——UI解析与可访问性数据集

- 整理方：[MacPaw Way有限公司](https://huggingface.co/MacPaw)
- 语言：以英语、乌克兰语为主
- 许可证：MIT协议

**概述**
UiPad是为[IASA Champ 2024挑战赛](http://champ.iasa.kpi.ua/)构建的数据集，聚焦MacOS应用的可访问性与界面理解任务。随着AI驱动的用户界面分析领域关注度持续提升，本数据集旨在填补桌面应用可访问性相关可用资源的空白。当前移动端应用与Web平台已拥有RICO、Mind2Web等成熟数据集，但MacOS应用相关资源仍较为匮乏，尤其是在可访问性解析与文本表征领域。


## 数据集结构
UiPad包含来自63款不同MacOS应用的352个独特屏幕界面。其中68%的样本包含JSON格式的可访问性数据。每个应用屏幕均配有截图，若样本包含可访问性数据，则同时附带一份描述可访问性元素的JSON文件。

plaintext
数据集
│
├── 应用程序/
│ ├── 屏幕状态ID（无可访问性数据）/
│ │ ├── 截图.png
│ ├── 屏幕状态ID（含可访问性数据）/
│ │ ├── 截图.png
│ │ ├── 可访问性树.json
│ ...

**截图**：应用屏幕的PNG格式图像。

**可访问性树数据**
可访问性树涵盖了核心UI元素，包括：
- `name`：元素名称
- `role`：UI元素的角色（例如按钮、图像）
- `description`与`role_description`：元素描述与角色描述
- `value`：元素状态或取值
- `children`：嵌套UI组件
- `bbox`与`visible_bbox`：元素的边界框坐标


**评估用问题**
本数据集包含多种类型的问题，用于评估UI理解能力：
- 数值型：「屏幕上有多少个已勾选的复选框？」（共485个实例）
- 是/否型：「屏幕上是否存在「+」按钮？」（共306个实例）
- 字符串型：「屏幕上的应用名称是什么？」（共143个实例）
- 坐标型：「点击何处可连接Gmail？」（共122个实例）

本数据集涵盖了可访问性识别中的真实场景挑战。部分屏幕可能未提供完整的可访问性支持，常见问题包括角色识别错误（例如将按钮识别为图像）、边界框不准确或缺失选中状态等。


**任务与目标**
UiPad的核心目标是开发能够理解并优化MacOS场景下UI可访问性的AI智能体（AI Agent）。UI表征生成的质量与AI智能体的效能，将通过与UI理解相关的问答任务进行评估。

**局限性与挑战**
- 可访问性数据可能存在不完整、冗余或缺失的情况
- 数据集规模有限，可能不足以支撑从零开始训练模型
- 问答任务的人工标注可能引入误差

**数据集卡片联系方式**
若您对本数据集有任何疑问或需要进一步信息，请发送邮件至[e-mail]！