A Dataset of Annotated Semantic Descriptions of UI Components for Desktop Environments

This dataset is part of the paper "Enriching Process-Related UI Logs via Screenshot-Based Activity Labeling Using Vision-Language Models". It contains two main files: screenshots_&_soms.zip and semantic_labels.csv. The screenshot_&_soms.zip file includes the Screenshots referenced in the different events, alongside their respective "Screen Object Models" (SOM), which includes the different UI Components in the Screenshots and their hierarchical relationship at different levels of depth, ranging from "Screen" or "Application" components, all the way down to "Icon" or "Text", passing through intermediate components such as "Container" or "Sidebar". The contents of dataset sums up to a total of 559 manually labelled UI Elements across 100 different screenshots. Each label represents a semantic description of an UI Element which shortly describes its purpose and meaning (e.g., refresh button). This dataset is used in the original paper to evaluate the capacity of LLMs to extract accurate semantic descriptions of UI Elements with varying techniques and models. More precisely, the semantic_labels.csv presents the data in the form a UI log with independent events containing the following properties: Screenshot: Each event has an associated screenshot which captures the UI at the moment of the user interacting with the user interface. EventType: Represents the user input. In this dataset, it is always "left_click". Coordinates: Click coordinates in the event. This information is used to locate the UI Element with which an user has interacted. Class: Indicates the class of the UI Element the user has interacted with (e.g., Button). This information is included for results analysis purposes. Depth: Indicates the depth of the UI Element the user has interacted with within the SOM of the image. This information is included for results analysis purposes. Density: The screenshots included within the dataset are divided into "Low Density", "High Density", and "Medium Density", depending on the number of UI Elements within the image. Through this property we represent in which of the three groups of images the element is located. This information is included for results analysis purposes. GroundTruth: Human generated activity label, following the same set of instructions provide to the VLMs during their evaluation.本数据集隶属于论文《基于视觉语言模型的截图式活动标注方法丰富流程相关UI日志》（Enriching Process-Related UI Logs via Screenshot-Based Activity Labeling Using Vision-Language Models），包含两个主要文件：screenshots_&_soms.zip与semantic_labels.csv。 screenshots_&_soms.zip压缩包包含不同事件所引用的截图，以及对应的屏幕对象模型（Screen Object Models，SOM）。该模型涵盖截图中的各类UI组件及其不同深度层级的层级关系，覆盖范围从“屏幕”或“应用”级组件，经由“容器（Container）”“侧边栏（Sidebar）”等中间组件，直至“图标（Icon）”或“文本（Text）”级组件。 本数据集总计涵盖100张不同截图中的559个经人工标注的UI元素。每个标注均为UI元素的语义描述，简要说明其用途与含义（例如“刷新按钮”）。本数据集被用于原论文中，以评估大语言模型（Large Language Model，LLM）通过各类技术与模型提取UI元素精准语义描述的能力。 更确切地说，semantic_labels.csv以UI日志的形式呈现数据，其中包含若干独立事件，各事件具备如下属性： 1. Screenshot（截图）：每个事件均关联一张截图，该截图捕获了用户与用户界面（User Interface，UI）交互时刻的UI画面。 2. EventType（事件类型）：代表用户输入操作，本数据集内该字段值恒为"left_click（左键单击）"。 3. Coordinates（坐标）：事件中的点击坐标，该信息用于定位用户交互过的UI元素。 4. Class（类别）：标注用户交互过的UI元素的类别（例如"Button（按钮）"），该信息用于结果分析。 5. Depth（深度）：标注用户交互的UI元素在对应图像的屏幕对象模型中的层级深度，该信息用于结果分析。 6. Density（密度）：根据图像内UI元素的数量，本数据集内的截图被分为"Low Density（低密度）""Medium Density（中密度）"与"High Density（高密度）"三类，该字段用于标识目标元素所属的图像分组，仅用于结果分析。 7. GroundTruth（真实标签）：由人工生成的活动标注，其生成规则与评估期间向视觉语言模型提供的指令集完全一致。