Expert evaluations of LLM responses on clinical questions

Real clinicians with years of services of post qualification over 10 years were invited to act as expert evaluators in their respective specialties on model responses (answers) generated by LLMs for the real clinical questions submitted to them.Each clinical expert on the evaluator panel independently rated each LLM response based on the following physician-equivalence scale based on the Chinese medical training system: 0, no knowledge; 0.5-1.5, general knowledge; 2-3.5, fresh medical graduate; 4-5.5, junior medical resident (&lt;3 years after graduation); 6-6.5, senior medical resident, fellow, or physician (3-6 years after graduation); 7-8, associate consultant or associate specialist (6-10 years after graduation); 8.5-9, consultants (&gt;10 years after graduation); and \geq9.5, leading experts. A mapping of the physician career stages in different medical training systems is provided in Supplementary Methods.The json file contains 685 questions (each with 8-letter IDs), each with 7 responses (with 6 letter IDs and model suffices). Each model response in term has between 1 and 3 evaluation results, each with these four keys: "user", "strength", "hallucination", "halluText"; corresponding to the evaluator ID, plausibility score (0-10) received, hallucination score (0-5) received, and the contents of the hallucination allegations.<b>Figure2.modelCoefficients</b> contains fitted coefficients used in plotting Figure 2A, 2B, and 2C.<b>hallu-sept-4-2025.xlsx</b> contains the 40 alleged hallucinations.<br>