Investigating Reproducibility in Deep Learning-Based Software Fault Prediction

Over the past few years, increasingly complex machine learning methods have been applied for various Software Engineering (SE) tasks, particularly for the important task of automated fault prediction and localization. It, however, becomes much more difficult for scholars to reproduce the results that are reported in the literature, especially when the applied deep learning models and the evaluation methodology are not properly documented and when code and data are not shared. Given some recent---and very worrying---findings regarding reproducibility and progress in other areas of applied machine learning, this study aims to analyze to what extent the field of software engineering, in particular in the area of software fault prediction, is plagued by similar problems. We have therefore conducted a systematic review of the current literature and examined the level of reproducibility of 56 research articles that were published between 2019 and 2022 in top-tier software engineering conferences. Our analysis revealed that scholars are apparently largely aware of the reproducibility problem, and about two-thirds of the papers provide code for their proposed deep-learning models. However, it turned out that in the vast majority of cases, crucial elements for reproducibility are missing, such as the code of the compared baselines, code for data pre-processing, or code for hyperparameter tuning. In these cases, it, therefore, remains challenging to reproduce the results in the current research literature exactly. Overall, our meta-analysis, therefore, calls for improved research practices to ensure the reproducibility of machine-learning-based research.

近年来，愈发复杂的机器学习方法已被应用于各类软件工程（Software Engineering，SE）任务，其中尤为重要的是自动化故障预测与定位任务。然而，学者们复现文献中报道的研究结果却愈发困难，尤其是当所使用的深度学习模型与评估方法未得到妥善记录，且代码与数据未公开共享时。鉴于近期在应用机器学习其他领域中出现的、令人颇为担忧的可复现性相关研究发现与进展，本研究旨在分析软件工程领域——尤其是软件故障预测方向——在多大程度上受到同类问题的困扰。为此，我们对当前相关文献开展了系统性综述，并考察了2019至2022年发表于顶级软件工程会议的56篇研究论文的可复现性水平。分析结果显示，学者们显然已在很大程度上意识到可复现性问题，约三分之二的论文公开了其所提出的深度学习模型的代码。但研究同时发现，绝大多数案例中，复现所需的关键要素仍存在缺失，例如对比基准模型的代码、数据预处理代码或是超参数调优代码。因此，在这类情形下，精准复现当前研究文献中的结果仍颇具挑战。总体而言，本项元分析呼吁优化研究实践，以确保基于机器学习的研究具备可复现性。