Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla

Dataset description This repository contains datasets collected as part of a study investigating performance regression detection and change point evaluation in software performance monitoring systems. The data originates from controlled annotation tasks and follow-up surveys conducted with Mozilla engineers.The dataset is composed of X main components: Annotation Experiment Data (tasks_anonymized.csv and annotations_anonymized.csv) Demographics Survey Data (demographics_anonymized.csv) Validation Survey Data (survey_anonymized.csv) Timeseries data & attributes (timeseries.zip and timeseries_attributes.json) Annotation Experiment Data This dataset captures the results of a manual annotation experiment where participants analyzed performance datasets and identified potential change points. It consists of two CSV files: Tasks Dataset (tasks_anonymized.csv): The tasks dataset records the annotation tasks assigned to participants and metadata related to their completion. Its columns are: Column Description TaskID Unique identifier of the annotation task DatasetName Identifier of the performance dataset analyzed in the task Difficulty Self-reported difficulty level of the task TimeSpent Time spent completing the task (in seconds) Problem Optional field for reporting issues encountered during the task UserID Identifier of the participant who completed the task Annotations Dataset (annotations_anonymized.csv): The annotations dataset contains the actual change point annotations produced by participants while analyzing the datasets. This dataset is used to construct the ground truth annotations and analyze agreement between annotators. Its columns are: Column Description DatasetName Identifier of the dataset being annotated UserID Identifier of the participant who made the annotation AnnotationIndex Index position of the annotated change point within the time series AnnotationType Type of change detected (mean, variance, and mean_variance) It is worth noting that the variance annotations were dropped upon doing the evaluation of the change point detection methods mentioned in the paper as there was low agreement on this type of annotation points. Demographics Survey Data (demographics_anonymized.csv): This dataset contains background information about participants who took part in the annotation experiment. It is used to describe the background and expertise of participants, allowing the study to contextualize results with respect to experience level and domain knowledge. The columns correspond to the survey questions presented to participants. Validation Survey Data (survey_anonymized.csv): This dataset contains responses to a follow-up validation survey where participants evaluated the results of the study and provided feedback on proposed evaluation strategies. Each column represents one survey question and is labeled using the exact wording presented to participants. Timeseries data & attributes (timeseries.zip and timeseries_attributes.json): They corespond to the time series data used in the annotations process (a total of 174 time series), and their attreibutes which some of them were displayed to annotators upon performing the annotation tasks. Refer to this publication and its replication package here for more context on the time series data.