Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla

Dataset descriptionThis repository contains datasets collected as part of a study investigating performance regression detection and change point evaluation in software performance monitoring systems. The data originates from controlled annotation tasks and follow-up surveys conducted with Mozilla engineers.The dataset is composed of 2 main components: Annotation Experiment Data (tasks_anonymized.csv and annotations_anonymized.csv) Timeseries data & attributes (timeseries.zip and timeseries_attributes.json) Annotation Experiment DataThis dataset captures the results of a manual annotation experiment where participants analyzed performance datasets and identified potential change points. It consists of two CSV files: Tasks Dataset (tasks_anonymized.csv): The tasks dataset records the annotation tasks assigned to participants and metadata related to their completion. Its columns are: Column Description TaskID Unique identifier of the annotation task DatasetName Identifier of the performance dataset analyzed in the task Difficulty Self-reported difficulty level of the task TimeSpent Time spent completing the task (in seconds) Problem Optional field for reporting issues encountered during the task UserID Identifier of the participant who completed the task Annotations Dataset (annotations_anonymized.csv): The annotations dataset contains the actual change point annotations produced by participants while analyzing the datasets. This dataset is used to construct the ground truth annotations and analyze agreement between annotators. Its columns are: Column Description DatasetName Identifier of the dataset being annotated UserID Identifier of the participant who made the annotation AnnotationIndex Index position of the annotated change point within the time series AnnotationType Type of change detected (mean, variance, and mean_variance) It is worth noting that the variance annotations were dropped upon doing the evaluation of the change point detection methods mentioned in the paper as there was low agreement on this type of annotation points. Timeseries data & attributes (timeseries.zip and timeseries_attributes.json): They corespond to the time series data used in the annotations process (a total of 174 time series), and their attreibutes which some of them were displayed to annotators upon performing the annotation tasks. Refer to this publication and its replication package here for more context on the time series data.数据集说明 本仓库收录的数据集，源自一项针对软件性能监控系统中性能回归检测与变化点评估的研究。数据来源于与Mozilla工程师开展的受控标注任务及后续调研。本数据集包含两大核心组成部分： ### 标注实验数据（tasks_anonymized.csv 与 annotations_anonymized.csv） 本数据集记录了人工标注实验的结果，标注参与者在此实验中分析性能数据集并识别潜在变化点，包含两个CSV文件： 1. **任务数据集（tasks_anonymized.csv）**：该数据集记录了分配给标注参与者的标注任务及其完成相关的元数据，各字段说明如下： | 列名 | 描述 | |--------------|--------------------------| | TaskID | 标注任务的唯一标识符 | | DatasetName | 本次任务分析的性能数据集标识符 | | Difficulty | 标注参与者自我报告的任务难度等级 | | TimeSpent | 完成任务所耗费的时长（单位：秒） | | Problem | 用于记录任务过程中遇到的问题的可选字段 | | UserID | 完成该任务的标注参与者标识符 | 2. **标注数据集（annotations_anonymized.csv）**：该数据集包含标注参与者在分析数据集时生成的实际变化点标注结果，用于构建基准标注并分析标注者间的一致性。各字段说明如下： | 列名 | 描述 | |------------------|----------------------------------------------------------------------| | DatasetName | 待标注数据集的标识符 | | UserID | 生成该标注的标注参与者标识符 | | AnnotationIndex | 标注的变化点在时间序列中的索引位置 | | AnnotationType | 检测到的变化类型（包括均值、方差及均值方差混合型） | 需注意的是，在评估论文中提及的变化点检测方法时，由于此类标注点的标注一致性较低，方差类型的标注已被移除。 ### 时序数据与属性文件（timeseries.zip 与 timeseries_attributes.json） 二者分别对应标注流程中使用的时序数据（共计174条时序序列）及其属性信息，其中部分属性会在标注参与者执行标注任务时向其展示。如需了解时序数据的更多背景信息，请参考该学术论文及其复现仓库。