Impact of reference panel composition on scores of script concordance test assessing basic nephrology knowledge in undergraduate medical education

In the assessment of basic medical knowledge, the composition of the reference panel between specialists and primary care (PC) physicians is a contentious issue. We assessed the effect of panel composition on the scores of undergraduate medical students in a script concordance test (SCT). The scale of an SCT on basic nephrology knowledge was set by a panel of nephrologists or a mixed panel of nephrologists and PC physicians. The results of the SCTs were compared with ANOVA for repeated measurements. Concordance was assessed with Bland and Altman plots. Forty-five students completed the SCT. Their scores differed according to panel composition: 65.6 ± 9.73/100 points for nephrologists, and 70.27 ± 8.82 for the mixed panel, <i>p</i> &lt; 0.001. Concordance between the scores was low with a bias of −4.27 ± 2.19 and a 95% limit of agreement of −8.96 to −0.38. Panel composition led to a change in the ranking of 71% of students (mean 3.6 ± 2.6 places). The composition of the reference panel, either specialist or mixed, for SCT assessment of basic knowledge has an impact on test results and student rankings.

在基础医学知识测评领域，由专科医师与基层医疗（Primary Care, PC）医师构成的参考评分专家组的组成方案，是一项颇具争议的议题。本研究针对脚本一致性测试（Script Concordance Test, SCT）中参考专家组的组成方案对医学生得分的影响展开了评估。针对基础肾脏病学知识的两款SCT测评量表，分别由肾脏病专科医师专家组，以及肾脏病专科医师与PC医师混合组成的专家组独立制定。采用重复测量方差分析（Analysis of Variance, ANOVA）对两款SCT的测评结果进行比较，并通过Bland-Altman图评估得分间的一致性。共有45名医学生完成了本次测评。不同量表对应的学生得分存在显著差异：纯肾脏病专家组制定的量表得分均值为65.6±9.73/100，混合专家组制定的量表得分均值为70.27±8.82/100，组间差异具有统计学意义（p<0.001）。两款量表的得分一致性较低，偏差为-4.27±2.19，95%一致性界限为-8.96至-0.38。专家组组成方案导致71%的学生排名发生变动，平均排名变化幅度为3.6±2.6个位次。用于基础医学知识SCT测评的参考专家组组成方案（纯专科专家组或混合专家组），会对测评结果及学生排名产生显著影响。