Not from scratch: Explainable deep transfer learning fine-tunning with domain adaptation enables trustworthy COVID-19 prediction

Background and Objective: Medical image analysis can help diagnose Coronavirus Disease 2019 (COVID-19) early and save patient lives before the disease worsens. However, there are various limitations to manual inspection of these medical images, such as dependence on physician experience and subjectivity of assessment. To facilitate the rapid and accurate diagnosis of disease, computer-aided diagnostic systems based on deep learning methods, typically convolutional neural networks (CNN) can be used. However, neural networks are usually black-box models that do not provide a clear insight into their prediction outcomes. Methods: Here, we proposed a framework called explainable deep transfer learning for medical image classification (XDTLMI-Net) that uses four CNNs proficient in handling image data, including GoogLeNet, ResNet18, ResNet50 and ResNet101. This framework uses existing medical domain knowledge to guide transfer learning with COVID-19 CT scan images and CXR images. Results: XDTLMI-Net performed three tasks of medical image classification of COVID-19 on three benchmark datasets: COVID-19 CT, SARS-COV-2 CT and COVID-19 CXR. It achieved an average classification accuracy of 0.9897, 0.9752 and 0.9397, and an average classification F1-score of 0.9898, 0.9741 and 0.9394, respectively. Moreover, we employed the Shaply Additive exPlanations (SHAP) and gradient-weighted class activation mapping (Grad-CAM) to interpret the COVID-19 predictions and help understand the predictive models’ decision-making process. Conclusions: A general end-to-end framework called XDTLMI-Net based on CNN and TL was developed, which works on small datasets of medical images, which does not require any segmentation or image preprocessing procedures. XDTLMI-Net outperformed on three datasets in fine-tuning course and gave reasonable importance to each input COVID-19 image, showing its potential for application in different clinical scenarios.背景与研究目的：医学图像分析可在新型冠状病毒肺炎（Coronavirus Disease 2019, COVID-19）病情恶化前实现早期诊断，挽救患者生命。然而，人工阅片存在诸多局限，例如依赖医师经验且评估具有主观性。为实现疾病的快速精准诊断，可采用基于深度学习方法（通常为卷积神经网络（Convolutional Neural Networks, CNN））的计算机辅助诊断系统。但神经网络通常属于黑箱模型，无法对其预测结果提供清晰的可解释性。 方法：本文提出了一种用于医学图像分类的可解释性深度迁移学习框架（XDTLMI-Net），该框架采用四种擅长处理图像数据的卷积神经网络，包括GoogLeNet、ResNet18、ResNet50与ResNet101。该框架利用现有医学领域知识，结合新型冠状病毒肺炎CT扫描图像与胸部X线（Chest X-ray, CXR）图像指导迁移学习。 结果：XDTLMI-Net在三个基准数据集上完成了新型冠状病毒肺炎医学图像分类的三项任务，分别为COVID-19 CT、SARS-COV-2 CT与COVID-19 CXR数据集。其平均分类准确率分别为0.9897、0.9752与0.9397，平均分类F1值分别为0.9898、0.9741与0.9394。此外，本文采用夏普利可加解释（Shaply Additive exPlanations, SHAP）与梯度加权类激活映射（Gradient-weighted Class Activation Mapping, Grad-CAM）对新型冠状病毒肺炎的预测结果进行解释，以帮助理解预测模型的决策过程。 结论：本文开发了一种基于卷积神经网络（CNN）与迁移学习（Transfer Learning, TL）的通用端到端框架XDTLMI-Net，该框架可直接处理小型医学图像数据集，无需任何分割或图像预处理步骤。XDTLMI-Net在三个数据集的微调过程中均表现更优，且能为每幅输入的新型冠状病毒肺炎图像分配合理的权重，展现出其在不同临床场景中的应用潜力。