A Dataset of for evaluating Activity label inferencing from Event semantic descriptions

This dataset is part of the paper "Enriching Process-Related UI Logs via Screenshot-Based Activity Labeling Using Vision-Language Models". It contains two main files: screenshots_&_soms.zip and evaluation_data.csv.

The contents of this dataset represent a total of 30 activities, divided in 10 activities with 2 events, 10 activities with 3 events, and 10 activities with 4 events. The purpose of this dataset is to evaluate, with different techniques and models, the capacity of language models to generate accurate activity labels of user performed activities, comparing them to human generated labels with cosine similarity metrics. Each event in the dataset is represented as a row in the evaluation_data.csv log file, with the following properties:

ScreenID: Represents a necessary parameter in the framework proposed in the paper. It serves as an identifier of activities. Each activity has a different ScreenID, and all events within the same activity have the same ScreenID.

Screenshot: Each event has an associated screenshot which captures the UI at the moment of the user interacting with the user interface.

EventType: Represents the user input. In this dataset, it belongs to either "Left Click" or  "Keyboard".

Text: If the EventType is "Keyboard", this property represents the text inputted by the user.

EventDescription: Semantic description of the user action on the screen, generated during the third phase of the proposed framework. It describes what action has the user performed on what UI Element. These descriptions are then processed to generate the activity labels of the user performed activities.

GroundTruth: Human generated activity label, following the same set of instructions provide to the LLMs during their evaluation.

Furthermore, the screenshot_&_soms.zip file includes the Screenshots referenced in the different events, alongside their respective "Screen Object Models", which includes the different UI Components in the Screenshots and their hierarchical relationship at different levels of depth, ranging from "Screen" or "Application" components, all the way down to "Icon" or "Text", passing through intermediate components such as "Container" or "Sidebar".