UI Logs for Screenshot and for Activity Identification

These data correspond to the set of Problems used for the evaluation of the proposal From Screenshots to Process Models: Improving Activity Identification through Screen Text.

Each problem includes a User-Interaction (UI) Log that contains comprehensive process information. Accompanying the UI Log are directories with 58 screenshots from different instances of processes recorded on Odoo or Gmail interfaces and directories containing all on-screen text extracted from each screenshot using OCR. The UI Log references the screenshots and their corresponding OCR files for each row.

Additionally, results for every execution run per Problem are available. The results directories include subdirectories, differentiated by the image/text weight in the execution, containing the expanded base UI Log with columns resulting from feature extraction and clustering. Each Problem has its summary file, providing an overview of all executions and the relevant clustering and evaluation metrics.

A precise breakdown and description of each of the directories and files present in this assessment is provided below:

input/Problem_X-Y/ (where X represents the number of the Problem given in the article and Y its full name).

screenshots/: contains the screenshots from Odoo and Gmail interfaces. The naming pattern is X_img.png where X is the corresponding identifier. Each record contains up to 58 screenshots representing different activities within the instantiation of a business process.

ocr_results/: stores the results of applying OCR techniques to all screenshots for extracting all on-screen text. This directory includes a .txt file for each image, containing the extracted text. The naming pattern is similar to the previous one: X_ocr.txt where X is the identifier. These files are used for executions that use all on-screen text as a source along with the images.

log.csv: .csv file representing the UI Log of the process. It contains the following columns:

id: identifier corresponding to a screenshot within the UI log.

screenshot: path to the image corresponding to the row in the UI log within screenshots/.

timestamp: time marker for defining the sequential order of actions in the UI log

header: text string extracted from the browser’s tab.

header_txt: path to each corresponding text file within ocr_results/.

ground_truth: label assigned to identify and group images by activity. Compared with the activity_label column after algorithm application to determine execution’s evaluation metrics.

output/X_Y/ (where X represents an abbreviation of the Problem chosen for execution and Y specifies whether all on-screen text, browser’s tab text or image hash was used).

X_Y_a_b/ (where a,b refers to image/text weight for each execution). Each directory represents one specific execution from a Problem.

df.csv: .csv file representing the expanded UI Log with the experimentation results after the algorithm has been applied. The new columns are described below:

combined_features: contains the features extracted per image, either individually using hash or CLIP, or, in the latter case, combining them with those from the text source used.

activity_label: represents the group assigned by the algorithm through clustering based on feature comparison.

mapped_prediction: mapping of the cluster assigned by the algorithm to the group distribution established manually by ground_truth.

summary.csv: file which contains a summary of all executions from a Problem. It contains the following information:

exec: number indicating the row identifier.

image_weight: parameter that determines the importance of image features in the model.

text_weight: parameter that determines the importance of text features in the model.

Evaluation metrics:

Precision: the ratio of true positives to the sum of true positives and false positives. It measures the accuracy of not labeling different activities as the same.

Recall: the ratio of true positives to the sum of true positives and false negatives. It measures the ability to identify all the activities properly grouped.

F1-Score: a harmonic mean of precision and recall. It balances the relative contribution of precision and recall equally.

Clustering related metrics. (Calculated for testing purposes; in the article, our approach focuses solely on evaluation metrics).

Silhoutte Coefficient: measures how similar is an object to its cluster compared to other clusters. Values range from -1 to 1, with higher values indicating better-defined clusters.

Davies-Bouldin Index: assesses cluster quality by comparing the distance between clusters to the size of the clusters themselves. Lower values indicate better-defined clusters.

Calinski-Harabasz Index: evaluate cluster quality by comparing the spread within clusters to the spread between clusters. Higher values mean better-defined clusters.