Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla

Dataset descriptionThis repository contains datasets collected as part of a study investigating performance regression detection and change point evaluation in software performance monitoring systems. The data originates from controlled annotation tasks and follow-up surveys conducted with Mozilla engineers.The dataset is composed of 2 main components: Annotation Experiment Data (tasks_anonymized.csv and annotations_anonymized.csv) Timeseries data & attributes (timeseries.zip and timeseries_attributes.json) Annotation Experiment DataThis dataset captures the results of a manual annotation experiment where participants analyzed performance datasets and identified potential change points. It consists of two CSV files: Tasks Dataset (tasks_anonymized.csv): The tasks dataset records the annotation tasks assigned to participants and metadata related to their completion. Its columns are: Column Description TaskID Unique identifier of the annotation task DatasetName Identifier of the performance dataset analyzed in the task Difficulty Self-reported difficulty level of the task TimeSpent Time spent completing the task (in seconds) Problem Optional field for reporting issues encountered during the task UserID Identifier of the participant who completed the task Annotations Dataset (annotations_anonymized.csv): The annotations dataset contains the actual change point annotations produced by participants while analyzing the datasets. This dataset is used to construct the ground truth annotations and analyze agreement between annotators. Its columns are: Column Description DatasetName Identifier of the dataset being annotated UserID Identifier of the participant who made the annotation AnnotationIndex Index position of the annotated change point within the time series AnnotationType Type of change detected (mean, variance, and mean_variance) It is worth noting that the variance annotations were dropped upon doing the evaluation of the change point detection methods mentioned in the paper as there was low agreement on this type of annotation points. Timeseries data & attributes (timeseries.zip and timeseries_attributes.json): They corespond to the time series data used in the annotations process (a total of 174 time series), and their attreibutes which some of them were displayed to annotators upon performing the annotation tasks. Refer to this publication and its replication package here for more context on the time series data.