Development of a transfer learning-based, multimodal neural network for identifying malignant dermatological lesions from smartphone images

Early skin cancer detection in primary care settings is crucial for prognosis, however primary care physicians often lack relevant training. Machine learning (ML) tools offer a potential solution by enabling systematic triaging of suspicious lesions using images. This study aims to develop a novel neural network for the binary classification of skin lesions into malignant and benign categories using smartphone images and clinical data via a multimodal and transfer learning-based approach.

We used the PAD-UFES-20 dataset, which included 2298 sets of images and associated clinical data. Three neural network models were developed: a clinical data network, an image-based network based on a pre-trained DenseNet-121, and a multimodal network combining clinical and image data. Models were tuned using Bayesian Optimization HyperBand (BOHB) across 5-fold cross-validation.

The neural network models were built using keras 3.3.3 and tensorflow 2.16.1. BOHB tuning was completed using ray 2.30.0.