Conversational Guide for Cataract Surgery Complications: A Comparative Study of Surgeons versus Large Language Model-Based Chatbot Generated Instructions for Patient Interaction

It is difficult to explain the complications of surgery to patients. Care has to be taken to convey the facts clearly and objectively while expressing concern for their wellbeing. This study compared responses from surgeons with responses from a large language model (LLM)-based chatbot. We presented 10 common scenarios of cataract surgery complications to seven senior surgeons and a chatbot. The responses were graded by two independent graders for comprehension, readability, and complexity of language using previously validated indices. The responses were analyzed for accuracy and completeness. Honesty and empathy were graded for both groups. Scores were averaged and tabulated. The readability scores for the surgeons (10.64) were significantly less complex than the chatbot (12.54) (<i>p</i> &lt; 0.001). The responses from the surgeons were shorter, whereas the chatbot tended to give more detailed answers. The average accuracy and completeness score of chatbot-generated conversations was 2.36 (0.55), which was similar to the surgeons’ score of 2.58 (0.36) (<i>p</i> = 0.164). The responses from the chatbot were more generalized, lacking specific alternative measures. While empathy scores were higher for surgeons (1.81 vs. 1.20, <i>p</i> = 0.041), honesty scores showed no significant difference. The LLM-based chatbot gave a detailed description of the complication but was less specific about the alternative measures. The surgeons had a more in-depth understanding of the situation. The chatbot showed complete honesty but scored less for empathy. With more training using complex real-world scenarios and specialized ophthalmologic data, the chatbots could be used to assist the surgeons in counselling patients for postoperative complications.