Annotated UI Element Dataset for Desktop Environments

Introducing a specialized dataset containing high-resolution screenshots from various desktop environments, focusing on annotating individual UI components. This dataset is designed to enhance the accuracy of UI element identification and classification within desktop applications, enabling the extraction of hierarchical structures.

Desktop UI Detection Dataset.zip

This resource contains a set of 100 general-purpose screenshots intended for the training set.

Test Desktop UI Detection Dataset.zip

This resource is aimed at evaluating the models trained with the previous screenshots, using a set of captures from a specific business process.

The images have been organized into six groups (G), each representing a unique set of screenshots from the same type of application. These groups are distinguished by their levels of complexity and the depth of their UI hierarchies:

G1: PDF Reader. A combination of web and native applications for interacting with PDF documents.

G2: Public Administration Courses Manager. A web-based application used in the student admission process for public learning programs (these images are not provided due to privacy issues concerning the business process).

G3: Customer Relationship Management (CRM) System. A web-based application for managing business data.

G4: Email Client. A mix of web and native applications used for managing email communications.

G5: File Explorer. A native application for navigating the file system in Windows.

G6: Learning Management System (LMS). A web-based application used to manage courses and students for a given educational institution.

Two folders are provided, each containing all the groups mentioned. These folders represent captures with the application from the corresponding group in fullscreen, or captures with the application from the corresponding group overlapping another application randomly selected from one of the remaining groups (Overlapped).