Dermatologist cooperation with a market-approved convolutional neural network (CNN) may improve skin lesion identification performance, according to recent findings.¹

This study into market-approved CNN use by dermatologists was conducted in order to optimize collaboration in skin cancer screening, especially given the fact that early melanoma diagnosis is essential with regard to patients’ prognosis.

This research was authored by Julia K. Winkler, MD, from the Department of Dermatology at the University of Heidelberg in Germany, and was done to assess a cooperative format as neural networks for skin cancer detection had already been shown to achieve on par or superior performance alone compared to dermatologists.²

“With the present study, we aimed to elucidate the cooperation of dermatologists with a market-approved CNN in a prospective clinical setting,” Winkler and colleagues wrote. “Moreover, we used a validated questionnaire measuring patient acceptance and trust toward the tested CNN.”

Background and Findings

The investigators conducted a prospective, observational, and explorative clinical trial in order to examine the coordination of dermatologists and CNNs that had been approved for use in a clinical setting, assessing potential diagnostic performance changes.

The research team exclusively included melanocytic lesions for statistical reasons, and they performed their research at 2 sites: the University of Heidelberg’s dermatology department and a private dermatology practice in Germany.

Their study utilized dermatologists who had varying levels of dermoscopy experience to perform full-body examinations of potential melanocytic lesions, and the dermatologists were asked to assess the probability of malignancy using a visual analog scale and record their decisions.

Suspected lesions were then assessed by a CNN, which provided a score of malignancy that was shared with the team of dermatologists. In this study, scores ranged from 0 to 1 and the threshold for malignancy was set at a value of ≥0.5. They were then requested to reassess skin lesions and adjust their initial decisions by taking into account the results generated by the CNN model.

The team was then asked by the investigators to revise their diagnoses and management decisions based on the CNN results and provide feedback on the helpfulness and/or reassuring nature of the scores. Patients who had been involved were also provided a questionnaire to determine their trust and acceptance of CNN-based assistant systems. Histopathologic examination was done by experienced board-certified histopathologists.

The primary measures of evaluation were the diagnostic sensitivity and specificity, and the investigators considered accuracy and the area under the receiver operating characteristic curve (ROC AUC) as supplementary measures.

Overall, the investigators involved 22 dermatologists who assessed 228 suspicious melanocytic lesions in 188 patients with a mean age of 53.4 (range 19-91) years and a total of 51.6% male patients. Among the lesions, 190 were nevi, and 38 were found to be melanomas.

The research team found that the use of CNN improved diagnostic sensitivity from 84.2% (95% CI, 69.6% - 92.6%) to 100.0% (95% CI, 90.8% - 100.0%), accuracy from 74.1% (95% CI, 68.1% - 79.4%) to 86.4% (95% CI, 81.3% - 90.3%) and specificity from 72.1% (95% CI, 65.3% - 78.0%) to 83.7% (95% CI, 77.8% - 88.3%).

Additionally, they noted that the mean ROC AUC was 0.895 (95% CI, 0.836 - 0.954) to 0.968 (95% CI, 0.948 - 0.988). The team also reported that the CNN achieved a sensitivity, specificity, and diagnostic accuracy comparable to dermatologists alone in classifying melanocytic lesions.

The investigators also noted that unnecessary excisions of benign nevi were reduced by 19.2% when dermatologists worked with the CNN. The team also found that dermatologists with less dermoscopy experience who then worked with the CNN showed the most diagnostic improvement compared to more experienced dermatologists.

“To our knowledge, we herein present the first prospective diagnostic study investigating the collaboration of dermatologists with a market-approved CNN in a melanoma screening task,” they wrote. “In this study, dermatologists significantly improved their diagnostic performance when cooperating with the tested CNN.”

References

1. ^{Winkler JK, Blum A, Kommoss K, et al. Assessment of Diagnostic Performance of Dermatologists Cooperating With a Convolutional Neural Network in a Prospective Clinical Study: Human With Machine. JAMA Dermatol. Published online May 03, 2023. doi:10.1001/jamadermatol.2023.0905.}
2. ^{Haenssle HA, Fink C, Schneiderbauer R, et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann Oncol. 2018;29(8):1836-1842. doi:10.1093/annonc/mdy166.}