OS-Copilot/OS-Atlas-data

---
license: apache-2.0
viewer: false
---

# GUI Grounding Pre-training Data for OS-ATLAS
This document describes the acquisition of the pre-training data used by OS-ATLAS [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://huggingface.co/papers/2410.23218).

<div align="center">

[\[🏠Homepage\]](https://osatlas.github.io) [\[💻Code\]](https://github.com/OS-Copilot/OS-Atlas) [\[🚀Quick Start\]](#quick-start) [\[📝Paper\]](https://arxiv.org/abs/2410.23218) [\[🤗Models\]](https://huggingface.co/collections/OS-Copilot/os-atlas-67246e44003a1dfcc5d0d045) [\[🤗ScreenSpot-v2\]](https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2)

</div>


**Notes:** In GUI grounding data, the position of the target element is recorded in the `bbox` key, represented by `[left, top, right, bottom]`.
Each value is a [0, 1] decimal number indicating the ratio of the corresponding position to the width or height of the image.

The data stored in this dataset consists of raw data containing **only** element grounding information. When training a model, you need to use the corresponding prompts to wrap these data.

The data we released is divided into three domains: mobile, desktop and web.

All annotation data is stored in JSON format and each sample contains:
* `img_filename`: the interface screenshot file
* `instruction`: human instruction or referring expression extracted from ally tree or html
* `bbox`: the bounding box of the target element corresponding to instruction

Some data also contains a `data_type`, which records the type of an element in its structured information, if it can be obtained.

***

### Mobile data

This part of data is stored under the *mobile_domain* directory. Our mobile grounding data consists of four parts.

#### AMEX

Android Multi-annotation EXpo (AMEX) is a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents [1].

The annotation data is stored in

-`amex_raw.json`

Due to the single file size limitation of Hugging Face datasets, we stored the Amex images in *zip* format and split them into several sub-files.

- `amex_images_part_aa`
- `amex_images_part_ab`
- `amex_images_part_ac`

You need to first merge these split files back into the original file and then extract the contents.

```
cat amex_images_part_* > amex_images.zip
7z x amex_images.zip -aoa -o/path/to/extract/folder
```

#### UIBert

UIBert [2] is a dataset extended from Rico dataset [3] for two tasks: similar UI component retrieval and referring expression component retrieval.

The annotation data is stored in

- `uibert_raw.json`

The UIBert images are stored in

- `UIBert.zip`

#### Widget Captioning and RICOSCA

Widget Captioning data are collected by [4].

RICOSCA is a dataset automatically labeled using Android VH in [5]

The annotation data is stored in

- `widget_captioning.json`
- `ricosca.json`

The rico images are stored in

- `rico_imgs.zip`

#### Android_world_data

This part of data are sampled from a android environment for building and benchmarking autonomous computer control agents [6].

The annotation data is stored in

- `aw_mobile.json`

The rico images are stored in

- `mobile_images.zip`

***

### Desktop data

This part of data is stored under the *desktop_domain* directory.

All of the desktop grounding data is collected from the real environments of personal computers running different operating systems. Each image is split into multiple sub-images to enhance data diversity.

Our desktop grounding data consists of three parts: Windows, Linux and MacOS.

**The image and annotation data for each operating system are stored in corresponding zip and json files.**

It is worth noting that, due to the large size of the Windows image data, the split files need to be merged before extraction.

```
cat windows_image_part_* > windows_images.zip
7z x windows_images.zip -aoa -o/path/to/extract/folder
```

***

### Web data

This part of data is stored under the *web_domain* directory.

Our desktop grounding data consists of two parts.

#### Seeclick web data

The web data from SeeClick [7] was crawled from websites provided by Common Crawl, containing more than 270k webpage screenshots and over 3 million webpage elements.

The annotation data is stored in

- `seeclick_web.json`

The images are stored into split files and need to be merged before extraction.

```
cat seeclick_web_image_part_* > seeclick_web_images.zip
7z x seeclick_web_images.zip -aoa -o/path/to/extract/folder
```

#### Fineweb_crawled_data

This part of data is crawled from web pages from the latest URLs obtained from FineWeb [8], a cleaned and deduplicated English dataset derived from Common Crawl.

Since this portion of the data contains at least 1.6 million images, we have compressed them into 10 zip files, from `fineweb_3m_s11.zip` to `fineweb_3m_s52.zip`.

Please extract them into the same directory.

As an example,

```
7z x fineweb_3m_s11.zip -aoa -o/same/path/to/extract/fineweb
```

The annotation data is stored in

- `fineweb_3m.json`

***

### Best practice

During the training of **OS-Atlas-4B**, we randomly sampled predefined prompts to wrap the grounding data. Additionally, we scaled the relative coordinates of each element (in the range [0, 1]) by multiplying them by 1000 before inputting them into the model for training.

Below is an example of a data entry:

```
{
"conversations": [
{
"from": "human",
"value": "<image>\nUsing the provided screenshot, I'll describe webpage elements for you to locate (with bbox).\n<ref>media-maniacs.org</ref>\n<ref>Home</ref>\n<ref>Sitemap</ref>\n<ref>shop you can have what you choose 2012</ref>"
},
{
"from": "gpt",
"value": "<ref>media-maniacs.org</ref><box>[[70,856,141,871]]</box>\n<ref>Home</ref><box>[[21,935,43,951]]</box>\n<ref>Sitemap</ref><box>[[21,919,52,934]]</box>\n<ref>shop you can have what you choose 2012</ref><box>[[368,839,523,855]]</box>"
}
]
}
```

**OS-Atlas-7B**

```
{
"conversations": [
{
"from": "human",
"value": "<image>\nUsing the provided screenshot, I'll describe webpage elements for you to locate (with bbox).\n<|object_ref_start|>Facebook<|object_ref_end|>\n<|object_ref_start|>Subscribe<|object_ref_end|>\n<|object_ref_start|>Twitter<|object_ref_end|>\n<|object_ref_start|>Read More<|object_ref_end|>\n<|object_ref_start|>Read More<|object_ref_end|>"
},
{
"from": "gpt",
"value": "<|object_ref_start|>Facebook<|object_ref_end|><|box_start|>(4,955),(36,970)<|box_end|>\n<|object_ref_start|>Subscribe<|object_ref_end|><|box_start|>(4,913),(43,932)<|box_end|>\n<|object_ref_start|>Twitter<|object_ref_end|><|box_start|>(39,955),(62,970)<|box_end|>\n<|object_ref_start|>Read More<|object_ref_end|><|box_start|>(30,138),(73,157)<|box_end|>\n<|object_ref_start|>Read More<|object_ref_end|><|box_start|>(30,139),(73,155)<|box_end|>"
}
]
}
```

The prompts we used are stored in `prompts.json`.

***

**The following are the open-source datasets we used as data sources. We welcome everyone to check the details and cite these sources accordingly！**

[1] [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://arxiv.org/abs/2407.17490)

[2] [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://arxiv.org/abs/2107.13731)

[3] [Rico: A mobile app dataset for building data-driven design applications](https://dl.acm.org/doi/pdf/10.1145/3126594.3126651)

[4] [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://arxiv.org/pdf/2010.04295.pdf)

[5] [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://arxiv.org/pdf/2005.03776)

[6] [ANDROIDWORLD: A Dynamic Benchmarking Environment for Autonomous Agents](https://arxiv.org/abs/2405.14573)

[7] [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://arxiv.org/abs/2401.10935)

[8] [The fineweb datasets: Decanting the web for the finest text data at scale](https://arxiv.org/abs/2406.17557)

---
许可证：Apache-2.0
查看权限：关闭
---

# 用于OS-ATLAS的图形用户界面定位（GUI Grounding）预训练数据集
本文档介绍了OS-ATLAS[《OS-ATLAS：面向通用图形用户界面智能体的基础动作模型》](https://huggingface.co/papers/2410.23218)所使用的预训练数据的采集流程。

<div align="center">

[[🏠主页]](https://osatlas.github.io) [[💻代码库]](https://github.com/OS-Copilot/OS-Atlas) [[🚀快速入门]](#quick-start) [[📝论文]](https://arxiv.org/abs/2410.23218) [[🤗模型库]](https://huggingface.co/collections/OS-Copilot/os-atlas-67246e44003a1dfcc5d0d045) [[🤗ScreenSpot-v2]](https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2)

</div>


**注意：** 在GUI Grounding数据中，目标元素的位置以`bbox`键记录，采用`[左, 上, 右, 下]`的格式表示。每个取值均为范围在[0, 1]内的小数，代表对应位置占图像宽或高的比例。

本数据集存储的数据仅包含元素定位信息。在训练模型时，需使用对应的提示词（Prompt）对这些数据进行封装。

我们发布的数据分为三个领域：移动端、桌面端与网页端。

所有标注数据均以JSON格式存储，每个样本包含以下字段：
* `img_filename`：界面截图文件名称
* `instruction`：从辅助树或HTML中提取的人类指令或指代表达式
* `bbox`：与该指令对应的目标元素的边界框

部分数据还包含`data_type`字段，用于记录元素在其结构化信息中的类型（若可获取）。

***

### 移动端数据
该部分数据存储于*mobile_domain*目录下。我们的移动端定位数据由四部分组成。

#### AMEX
Android多标注博览会（AMEX，Android Multi-annotation EXpo）是一个面向通用移动端GUI控制智能体的大规模综合数据集[1]。
标注数据存储于：
-`amex_raw.json`

由于Hugging Face数据集存在单文件大小限制，我们将Amex的图像以zip格式打包并拆分为多个子文件：
- `amex_images_part_aa`
- `amex_images_part_ab`
- `amex_images_part_ac`

你需要先将这些拆分文件合并为原始压缩包，再解压其中内容：

cat amex_images_part_* > amex_images.zip
7z x amex_images.zip -aoa -o/path/to/extract/folder

#### UIBert
UIBert[2]是从Rico数据集[3]扩展而来的数据集，用于两项任务：相似UI组件检索与指代表达式组件检索。
标注数据存储于：
- `uibert_raw.json`
UIBert的图像存储于：
- `UIBert.zip`

#### Widget Captioning与RICOSCA
Widget Captioning数据由文献[4]采集。
RICOSCA是文献[5]中使用Android VH自动标注的数据集。
标注数据存储于：
- `widget_captioning.json`
- `ricosca.json`
Rico的图像存储于：
- `rico_imgs.zip`

#### Android_world_data
该部分数据从用于构建和评测自主计算机控制智能体的Android环境中采样得到[6]。
标注数据存储于：
- `aw_mobile.json`
移动端图像存储于：
- `mobile_images.zip`

***

### 桌面端数据
该部分数据存储于*desktop_domain*目录下。
所有桌面端定位数据均采集自运行不同操作系统的个人计算机真实环境。为增强数据多样性，每张图像被拆分为多个子图像。
我们的桌面端定位数据由三部分组成：Windows、Linux与MacOS。
**各操作系统对应的图像与标注数据分别存储于对应的压缩包与JSON文件中。**

值得注意的是，由于Windows图像数据体积较大，需先合并拆分文件再进行解压：

cat windows_image_part_* > windows_images.zip
7z x windows_images.zip -aoa -o/path/to/extract/folder

***

### 网页端数据
该部分数据存储于*web_domain*目录下。
我们的网页端定位数据由两部分组成。

#### Seeclick web data
来自SeeClick[7]的网页数据从Common Crawl提供的网站中爬取得到，包含超过27万张网页截图与超300万个网页元素。
标注数据存储于：
- `seeclick_web.json`
图像被拆分为多个子文件，需先合并再解压：

cat seeclick_web_image_part_* > seeclick_web_images.zip
7z x seeclick_web_images.zip -aoa -o/path/to/extract/folder

#### Fineweb_crawled_data
该部分数据从Fineweb[8]获取的最新URL对应的网页中爬取得到，Fineweb是一个从Common Crawl中衍生的经过清洗与去重的英文数据集。
由于该部分数据包含至少160万张图像，我们将其压缩为10个zip文件，从`fineweb_3m_s11.zip`到`fineweb_3m_s52.zip`。
请将它们解压至同一目录下。示例如下：

7z x fineweb_3m_s11.zip -aoa -o/same/path/to/extract/fineweb

标注数据存储于：
- `fineweb_3m.json`

***

### 最佳实践
在训练**OS-Atlas-4B**时，我们随机采样预设的提示词对定位数据进行封装。此外，在将元素的相对坐标（范围为[0, 1]）输入模型进行训练前，会将其乘以1000进行缩放。

以下为一个数据条目示例：

{
"conversations": [
{
"from": "human",
"value": "<image>
Using the provided screenshot, I'll describe webpage elements for you to locate (with bbox).
<ref>media-maniacs.org</ref>
<ref>Home</ref>
<ref>Sitemap</ref>
<ref>shop you can have what you choose 2012</ref>"
},
{
"from": "gpt",
"value": "<ref>media-maniacs.org</ref><box>[[70,856,141,871]]</box>
<ref>Home</ref><box>[[21,935,43,951]]</box>
<ref>Sitemap</ref><box>[[21,919,52,934]]</box>
<ref>shop you can have what you choose 2012</ref><box>[[368,839,523,855]]</box>"
}
]
}

**OS-Atlas-7B**

{
"conversations": [
{
"from": "human",
"value": "<image>
Using the provided screenshot, I'll describe webpage elements for you to locate (with bbox).
<|object_ref_start|>Facebook<|object_ref_end|>
<|object_ref_start|>Subscribe<|object_ref_end|>
<|object_ref_start|>Twitter<|object_ref_end|>
<|object_ref_start|>Read More<|object_ref_end|>
<|object_ref_start|>Read More<|object_ref_end|>"
},
{
"from": "gpt",
"value": "<|object_ref_start|>Facebook<|object_ref_end|><|box_start|>(4,955),(36,970)<|box_end|>
<|object_ref_start|>Subscribe<|object_ref_end|><|box_start|>(4,913),(43,932)<|box_end|>
<|object_ref_start|>Twitter<|object_ref_end|><|box_start|>(39,955),(62,970)<|box_end|>
<|object_ref_start|>Read More<|object_ref_end|><|box_start|>(30,138),(73,157)<|box_end|>
<|object_ref_start|>Read More<|object_ref_end|><|box_start|>(30,139),(73,155)<|box_end|>"
}
]
}

我们使用的提示词存储于`prompts.json`文件中。

***

**以下为我们用作数据源的开源数据集，欢迎大家查阅详情并引用这些来源！**
[1] [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://arxiv.org/abs/2407.17490)
[2] [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://arxiv.org/abs/2107.13731)
[3] [Rico: A mobile app dataset for building data-driven design applications](https://dl.acm.org/doi/pdf/10.1145/3126594.3126651)
[4] [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://arxiv.org/pdf/2010.04295.pdf)
[5] [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://arxiv.org/pdf/2005.03776)
[6] [ANDROIDWORLD: A Dynamic Benchmarking Environment for Autonomous Agents](https://arxiv.org/abs/2405.14573)
[7] [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://arxiv.org/abs/2401.10935)
[8] [The fineweb datasets: Decanting the web for the finest text data at scale](https://arxiv.org/abs/2406.17557)