Study Compares Artificial Intelligence Smartphone Applications for Skin Cancer

Credit: Pexels

There are varying levels of agreement between artificial intelligence (AI) smartphone applications designed to detect skin cancer, new findings suggest, with only 2 such applications having a Therapeutic Goods Administration (TGA) class 1 approval.¹

These findings also suggest that many publicly-available apps for skin cancer detection are largely unregulated and additional research may be necessary. Such conclusions were the result of new research comparing different apps and their use of AI for melanoma detection.

The study’s investigators—led by Lara Milliken from the Centre for Health Services Research at the University of Queensland in Brisbane, Australia—decided to use established procedures to analyze a total of 7 available skin cancer detection apps.²

“There has been an increase in consumer apps employing artificial intelligence (AI) to categorize suspicious skin lesions as high or low risk for skin cancer,” Milliken and colleagues wrote. “As the public interest rises, there is potential for enhanced patient care, however, there are also concerns surrounding potential harms.”¹

Study Design

The investigators carried out an observational study, enrolling a convenience sample of adult individuals in the age range of 18 and older. They worked to evaluate the consistency between the most commonly-downloaded skin cancer detection apps in Australia which also provided AI-based risk diagnoses or classifications.

A total of 7 phone apps were chosen by the research team based upon the pre-established criteria. Participants assessed by the team had 1 or 2 lesions on their skin which were photographed by a designated medical student through the use of an iPhone.

The investigators analyzed the images taken on the smartphone using each of the 7 apps, following their specified guidelines. The study did not provide participants with the app results; instead, they were given a skin cancer factsheet and advised to consult a doctor if they had concerns.

They categorized what they assessed as "suspicious," "non-suspicious," or "inconclusive" for the purposes of comparing the usefulness of these apps. A classification of "inconclusive" was utilized if an app was unable to process 1 of the photographed lesions following 10 unsuccessful attempts.

Alignment of Skin Cancer App Results

There were 33 participants who were included in the study. However 2 subjects decided not to remain for the second lesion imaging session, and 64 lesions were involved in the final assessment.

The investigators determined that the apps showed significant variation in their skin cancer evaluations. They noted that a single app labeled 31 as suspicious and another reported no lesions labeled as suspicious.

All smartphone apps analyzed lesions were in agreement within only 9.4% of cases. The research team reported that agreement remained slight, as indicated by the kappa statistic of 0.029. The team concluded that the highest inter-rater alignment occurred between 2 specific apps, adding that the lowest was between a different pair.

An evaluation of the clinical accuracy of these 7 phone apps was not conducted, though the team’s data indicate that the advice app users are given has wide variety depending on which app they decide to implement.

The investigators noted that discrepancies between apps likely result from the different neural networks and datasets utilized. They also highlighted the fact that only 1 app disclosed the type of AI algorithm employed.

Additionally, the research team reported that there was a lack of clarity regarding how much user-provided data is stored within the apps.

“This study is limited by its small sample size,” they wrote. “A medical student captured images for consistency, which may not reflect the public's imaging skillset. We did not provide names of the apps, as this study did not intend to analyse clinical validation or performance, but rather provide an overview of the agreement of commercially available apps.”

References

1. ^{Milliken L, Horsham C, Rodriguez-Acevedo AJ, Jayasinghe D, Soyer HP, Janda M. Comparison of artificial intelligence smartphone applications for skin cancer risk stratification of pigmented lesions: An observational study. J Eur Acad Dermatol Venereol. https://doi.org/10.1111/jdv.20253.}
2. ^{Ngoo A, Finnane A, McMeniman E, Tan JM, Janda M, Soyer HP. Efficacy of smartphone applications in high-risk pigmented lesions. Australas J Dermatol. 2018; 59(3): e175–e182.}