<p>Underlying data used for analysis.</p>

Current evaluations of large language models (LLMs) in healthcare have largely emphasized theoretical benchmarks and clinician oversight, with limited exploration of real-world physician-AI interaction. In this two-stage prospective study, we assessed physician satisfaction with LLM-generated responses to real clinical queries. This study did not evaluate clinical accuracy, patient outcomes, or patient safety. In the first unblinded stage, physicians used three models - a general-purpose model (GPT-4o), a reasoning-focused model (GPT-o1), and a healthcare-specific model (OpenEvidence) - to address 25 clinical dilemmas - and rated the quality of the responses. In the second blinded stage, the same physicians evaluated responses generated either by an LLM or by a human alone, without knowledge of the source. Across 100 real-world medical responses, median physician scores on a 5-point Likert scale were comparable between unblinded and blinded evaluations (p = 0.90). Satisfaction was not associated with physicians’ resistance to change, nor did it correlate with the accuracy or relevance of cited literature. These findings suggest that physicians did not favor information generated by LLMs over externally provided responses, and that clinician satisfaction alone may not serve as a reliable proxy for validating decision support tools.

当前针对医疗领域大语言模型（LLMs）的评估，大多聚焦于理论基准测试与临床医师监管层面，针对真实场景中医师与人工智能交互的探索仍较为有限。本研究为两阶段前瞻性研究，旨在评估医师对大语言模型针对真实临床问询生成回复的满意度。本研究未对临床准确性、患者结局或患者安全进行评估。第一阶段为非盲评估：医师使用三类模型——通用型模型（GPT-4o）、聚焦推理的模型（GPT-o1）以及医疗专用模型（OpenEvidence）——处理25项临床困境，并对回复质量进行评分。第二阶段为盲法评估：同一批医师在不知晓回复来源的情况下，对大语言模型或人类单独生成的回复进行评估。针对100份真实医疗回复，采用5级李克特量表的医师评分中位数在非盲与盲法评估中无显著差异（p=0.90）。医师满意度与医师的变革抵触情绪无关，也与引用文献的准确性和相关性无关联。本研究结果表明，医师并未更青睐大语言模型生成的信息而非外部提供的回复，且仅以临床医师满意度或许无法作为验证临床决策支持工具的可靠替代指标。