Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning

This repository provides the data used for the experiments of the paper "Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning" by Hazem Fahmy, Fabrizio Pastore, Mojtaba Bagherzadeh, and Lionel Briand appearing in IEEE Transactions on Reliability (doi: 10.1109/TR.2021.3074750) Deep neural networks (DNNs) are increasingly important in safety-critical systems, for example in their perception layer to analyze images. Unfortunately, there is a lack of methods to ensure the functional safety of DNN-based components. We observe three major challenges with existing practices regarding DNNs in safety-critical systems: (1) scenarios that are underrepresented in the test set may lead to serious safety violation risks, but may, however, remain unnoticed; (2) char- acterizing such high-risk scenarios is critical for safety analysis; (3) retraining DNNs to address these risks is poorly supported when causes of violations are difficult to determine. To address these problems in the context of DNNs analyzing images, we propose HUDD, an approach that automatically supports the identification of root causes for DNN errors. HUDD identifies root causes by applying a clustering algorithm to heatmaps capturing the relevance of every DNN neuron on the DNN outcome. Also, HUDD retrains DNNs with images that are automatically selected based on their relatedness to the identified image clusters. We evaluated HUDD with DNNs from the automotive domain. HUDD was able to identify all the distinct root causes of DNN errors, thus supporting safety analysis. Also, our retraining approach has shown to be more effective at improving DNN accuracy than existing approaches.本仓库收录了Hazem Fahmy、Fabrizio Pastore、Mojtaba Bagherzadeh与Lionel Briand发表于《IEEE可靠性汇刊》（IEEE Transactions on Reliability，DOI: 10.1109/TR.2021.3074750）的论文《基于热力图的无监督学习助力深度神经网络安全分析与再训练》（Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning）的实验所用数据。深度神经网络（Deep Neural Network, DNN）在安全关键系统中的应用价值与日俱增，例如在其感知层完成图像分析任务。然而当前尚缺乏有效的方法来保障基于DNN的组件的功能安全性。我们发现安全关键系统中针对DNN的现有实践存在三大核心挑战：其一，测试集覆盖不足的场景可能引发严重的安全违规风险，但此类风险往往难以被及时察觉；其二，对这类高风险场景进行特征刻画是开展安全分析的关键环节；其三，当违规行为的成因难以确定时，现有方法难以支持针对DNN的再训练以修复此类风险。针对图像分析场景下的DNN相关上述问题，我们提出HUDD方法，该方法可自动支持DNN错误根源的识别。HUDD通过将聚类算法应用于热力图来识别错误根源——此类热力图用于表征DNN各神经元对模型输出的相关性程度。此外，HUDD会基于与已识别图像簇的相关性自动筛选图像，并使用这些图像对DNN进行再训练。我们采用汽车领域的DNN对HUDD进行了评估，结果表明HUDD能够识别出DNN错误的所有不同根源，从而有效支持安全分析。同时，相较于现有方法，我们的再训练方案在提升DNN准确率方面表现更优。