Not from scratch: Explainable deep transfer learning fine-tunning with domain adaptation enables trustworthy COVID-19 prediction

Background and Objective: Medical image analysis can help diagnose Coronavirus Disease 2019 (COVID-19) early and save patient lives before the disease worsens. However, there are various limitations to manual inspection of these medical images, such as dependence on physician experience and subjectivity of assessment. To facilitate the rapid and accurate diagnosis of disease, computer-aided diagnostic systems based on deep learning methods, typically convolutional neural networks (CNN) can be used. However, neural networks are usually black-box models that do not provide a clear insight into their prediction outcomes.
Methods: Here, we proposed a framework called explainable deep transfer learning for medical image classification (XDTLMI-Net) that uses four CNNs proficient in handling image data, including GoogLeNet, ResNet18, ResNet50 and ResNet101. This framework uses existing medical domain knowledge to guide transfer learning with COVID-19 CT scan images and CXR images.
Results: XDTLMI-Net performed three tasks of medical image classification of COVID-19 on three benchmark datasets: COVID-19 CT, SARS-COV-2 CT and COVID-19 CXR. It achieved an average classification accuracy of 0.9897, 0.9752 and 0.9397, and an average classification F1-score of 0.9898, 0.9741 and 0.9394, respectively. Moreover, we employed the Shaply Additive exPlanations (SHAP) and gradient-weighted class activation mapping (Grad-CAM) to interpret the COVID-19 predictions and help understand the predictive models’ decision-making process.
Conclusions: A general end-to-end framework called XDTLMI-Net based on CNN and TL was developed, which works on small datasets of medical images, which does not require any segmentation or image preprocessing procedures. XDTLMI-Net outperformed on three datasets in fine-tuning course and gave reasonable importance to each input COVID-19 image, showing its potential for application in different clinical scenarios.

背景与研究目标：医学图像分析可助力早期诊断2019冠状病毒病（Coronavirus Disease 2019, COVID-19），在病情恶化前挽救患者生命。然而手动阅片存在诸多局限，例如依赖医师经验、评估结果带有主观性。为实现疾病的快速精准诊断，可采用基于深度学习方法的计算机辅助诊断系统，这类系统通常以卷积神经网络（Convolutional Neural Networks, CNN）为核心。但神经网络通常属于黑箱模型，无法对其预测结果给出清晰的解释。
研究方法：本研究提出一种用于医学图像分类的可解释性深度迁移学习框架（XDTLMI-Net），该框架采用四种擅长处理图像数据的卷积神经网络，包括GoogLeNet、ResNet18、ResNet50及ResNet101。该框架依托现有医学领域知识，结合COVID-19 CT扫描图像与胸部X线（Chest X-ray, CXR）图像指导迁移学习流程。
研究结果：XDTLMI-Net在三个基准数据集（COVID-19 CT、SARS-COV-2 CT及COVID-19 CXR）上完成了三项COVID-19医学图像分类任务，分别取得0.9897、0.9752、0.9397的平均分类准确率，以及0.9898、0.9741、0.9394的平均分类F1值。此外，本研究采用Shapley可加解释（Shapley Additive exPlanations, SHAP）与梯度加权类激活映射（Gradient-weighted Class Activation Mapping, Grad-CAM）对COVID-19的预测结果进行解释，以帮助理解预测模型的决策过程。
研究结论：本研究开发了一种基于卷积神经网络与迁移学习的通用端到端框架XDTLMI-Net，该框架可适用于小样本医学图像数据集，且无需任何分割或图像预处理步骤。XDTLMI-Net在三个数据集的微调过程中表现更优，并为每幅输入的COVID-19图像赋予合理的权重，展现出其在不同临床场景中应用的潜力。