Multimodal deep learning improving the accuracy of pathological diagnoses for membranous nephropathy

Renal biopsy is the gold standard for the diagnosis of glomerular diseases including membranous nephropathy (MN), however, it faces challenges in accuracy, objectivity, and reproducibility of tissue evaluation. This study aims to develop a multimodal pathological diagnosis system to assist pathologists in diagnosing MN in morphology. Using PASM-stained, immunofluorescence, and electron microscopy images from MN patients, we built three deep-learning models to detect lesions. The outputs of these models were combined to provide a comprehensive pathological diagnosis. Our system was compared with pathologists, validated on external test sets, and detected in 138 patients with various kidney diseases. Considering PASM-stained images, our model had a classification accuracy of 91.74%, a recall of 81.97%, and an F1 score of 86.58% for spike identification. For immunofluorescence images, our model had an accuracy rate of 98.97%, a recall rate of 99.65%, and an F1 score of 99.31% for MN classification. Regarding the segmentation of electron-dense deposits, the segmentation model had a Dice coefficient of 85.66% and an IoU of 75.93%. Our model presented superior performance to that of pathologists in fluorescence image classification and segmentation of deposits, achieved high accuracy in spike identification and fluorescence image classification in external test sets, and could be targeted to diagnose MN in a wide range of glomerular diseases. This multimodal pathological diagnosis system can not only assist pathologists in diagnosing MN rapidly and accurately but also lays the foundation to develop diagnostic models for other glomerular diseases.

肾活检是诊断包括膜性肾病（MN）在内的肾小球疾病的金标准，但在组织评估的准确性、客观性和可重复性方面仍面临挑战。本研究旨在开发一种多模态病理诊断系统，以辅助病理学家从形态学角度诊断MN。利用MN患者的PASM染色、免疫荧光及电子显微镜图像，我们构建了三个深度学习模型用于病变检测，并整合这些模型的输出结果以提供全面的病理诊断。我们将该系统与病理学家的诊断结果进行比较，在外部测试集上验证，并对138例患有不同肾脏疾病的患者进行了检测。针对PASM染色图像，模型在钉突识别任务中的分类准确率为91.74%、召回率为81.97%、F1值为86.58%；针对免疫荧光图像，模型在MN分类任务中的准确率为98.97%、召回率为99.65%、F1值为99.31%；在电子致密物沉积分割任务中，分割模型的Dice系数（Dice coefficient）为85.66%，IoU（Intersection over Union）为75.93%。我们的模型在荧光图像分类和沉积物分割任务中的表现优于病理学家，在外部测试集上的钉突识别和荧光图像分类任务中达到了较高准确率，且可针对多种肾小球疾病中的MN进行诊断。该多模态病理诊断系统不仅能辅助病理学家快速准确地诊断MN，还为开发其他肾小球疾病的诊断模型奠定了基础。