Comparative Analysis of AI Chatbots ChatGPT, Gemini, and Copilot’s Answers to Common Cataract Questions

Doi: 10.36351/pjo.v40i4.1887

Abstract

Purpose:  To compare the readability  and quality of the answers given by artificial intelligence (AI) chatbots to the twenty-five most frequent searches about cataracts on Google.

Study Design:  Cross sectional comparative study.

Place and Duration of study:  Serdivan State Hospital's Ophthalmology Department from March 2024 to April 2024.

Methods:  The word 'Cataract' was entered into Google Trends, and 25 trending searches made worldwide since 2004 were identified. These 25 trending searches were entered separately into AI programs. The answers were examined for quality using the Ensuring Quality Information for Patients (EQIP) test, while the readability was evaluated using the Flesch-Kincaid Reading Ease (FKRE) and Flesch-Kincaid Grade Level (FKGL).

Results:  The keywords searched most frequently were ‘Cataract surgery,’‘Eye cataract and 'After Cataract’. The EQIP category of all three AI chatbots was found to be 'serious quality issues.' When EQIP scores were compared, ChatGPT had a lower median value than Gemini and Copilot (p:0.001, p:0.007, respectively), while there was no significant difference between Gemini and Copilot (p:0.098). When FKRE values were compared, ChatGPT had a lower median value than Gemini and Copilot (p:0.001, p:0.001, respectively), with no significant difference between Gemini and Copilot (p:0.557). When FKGL values were compared, ChatGPT had a higher median value than Gemini and Copilot (p:0.003, p:0.001, respectively), with no significant difference between Gemini and Copilot(p:0.245).

Conclusion:  All three AI chatbots had an EQIP category of 'serious issues with quality .'The readability of all three was not at the recommended level.