Comparison of Human and Large Language Model Coded Themes in Patient Comments About Their Care Experience

The themes that arise in patient verbatim comments regarding experience can guide clinician and unit improvements, but traditional qualitative analysis is time-consuming. This study evaluated whether large language models (LLMs) can efficiently identify themes in patient feedback. We analyzed 77 transcribed comments from post-visit telephone surveys in musculoskeletal specialty care grouped by rating: 0, 1-2, and 3-6 on a 0 to 10-point scale. An LLM first generated themes, after which human coders refined and applied these themes to all comments. We compared the themes and theme counts generated by an LLM and by humans using the LLM as a tool. The LLM themes and subthemes that varied by rating category. Human coders consolidated these into three consistent themes across rating categories: trust and relationship, lack of direction, and logistics. The lowest recommendation rating was dominated by relationship concerns, whereas mid-range scores mostly addressed logistical issues. LLM theme counts approximated human counts across groups, suggesting that LLMs can streamline qualitative review while preserving human judgment in refinement.