Study Investigators Discuss the Evolving Role of ChatGPT in Ophthalmology

A recent analysis found the popular artificial intelligence (AI) chatbot ChatGPT correctly responded to approximately 84% of multiple-choice practice questions on a common ophthalmic knowledge assessment.

Previous findings by the investigative team from the University of Toronto showed an earlier version of the chatbot correctly answered 46% of the multiple-choice questions in January 2023 and 58% in February 2023, indicating the chatbot’s increasing knowledge over time.

To better understand the evolving role of AI in ophthalmology, 3 of the study investigators joined HCPLive to share their thoughts on the study’s findings and where they see the future role of AI.

• Rajeev Muni, MD, MSc, a vitreoretinal surgeon in the department of ophthalmology at St. Michael’s Hospital and Unity Health Toronto
• Marko M. Popovic, MD, MPH, a resident physician in the department of ophthalmology and vision sciences at the University of Toronto
• Andrew Mihalache, a medical student at the University of Toronto

“We can see almost in real time how this AI chatbot has evolved in terms of its ophthalmic knowledge and the gains in the performance of the chatbot, we’re seeing in virtually every subspecialty area of ophthalmology, from cornea to glaucoma and retina,” Popovic said. “There’s still work to be done, but is ChatGPT 4 approaching a point where it can start to offer assistance in preparing for board certification? It’s still not at 100%. But the dramatic advances in such a short period of time lends me to believe there could be potential in the future.”

The investigators pointed to the improved knowledge corpus of the chatbot as the primary reason for the drastic increase in correct responses on ophthalmic knowledge, noting other specialties may observe similar improvements over time if studied.

“You know, 100% is hard to achieve on any test by any machine or individual, but if you look at the pace of improvement, it's pretty remarkable,” Muni said. “And I think there's going to have to be continuous assessment of this. As newer versions come out and different chatbots are available, we'll compare the different ones and see what the positives and negatives are, but it certainly seems like the pace is quite dramatic.”

The experts discussed not just the raw ability of the chatbot, but the evolving place of artificial intelligence in medicine and its potential impact on clinical practice. Muni suggested the benefit of chatbots in training for board certification, but also the increasing role of AI in providing reinforcement to clinical decision-making based on imaging parameters.

“When you have AI algorithms that have been trained to look at imaging, and perhaps using biomarkers that we may not see with the naked eye, there’s a potential for AI to allow for decision support that may be even better than what can be done by humans,” he said. “I think this is going to involve a lot of work validating these algorithms for use with our common imaging modalities and I think many centers around the world are working intensely in this area.”

But the investigators also discussed the limitations of AI, issues with misinformation and the patient experience, and the potential for medical-legal issues. Noting young physicians early into their careers may rely on ChatGPT, they suggested the chatbot could provide inaccurate or incomplete responses and lead to suboptimal care.

“I think the bottom line here is that at the end of the day, in the diagnosis and treatment of patients, the AI chatbot cannot be held accountable for what it provides, the physician is ultimately accountable,” Popovic said. “And that’s particularly challenging in the situation where you ask ChatGPT, what are the symptoms of X condition and it provides 10 symptoms and only 9 of them are correct.”

For more insight from the investigators into the role of artificial intelligence and ChatGPT in ophthalmology, watch the full interview.

Relevant disclosures for Dr. Muni include Abbvie, Bausch+Lomb, Novartis, Roche, and others.

References

1. Iapoce C. Artificial Intelligence Chatbot appears to improve on Ophthalmic Knowledge Assessment. HCP Live. July 18, 2023. Accessed August 11, 2023. https://www.hcplive.com/view/artificial-intelligence-chatbot-appears-improve-ophthalmic-knowledge-assessment.
2. Mihalache A, Huang RS, Popovic MM, Muni RH. Performance of an Upgraded Artificial Intelligence Chatbot for Ophthalmic Knowledge Assessment. JAMA Ophthalmol. Published online July 13, 2023. doi:10.1001/jamaophthalmol.2023.2754