Evidence to support test interpretation: Evaluating the content validity of a new item set for the Transition Pragmatics Interview

There is little consensus on evidence-based practice guidelines for the selection of criterion-referenced assessments. Having confidence in scores from criterion-referenced assessments requires evidence that items align with their intended constructs. The purposes of these studies were to demonstrate evidence of content validity for the revised item set of a developing social communication assessment and to provide clinicians with a model of content validity evaluations that can be generalised to the review of other assessments. In Study 1, 10 experts rated 25 newly-developed items for how well they represented the intended construct. In Study 2, seven participants ages 14–20 were administered the Three Step Test Interview to assess their cognitive processes for responding to new items. Examinee responses were coded for construct-relevant and construct-irrelevant factors. Twenty-three of the 25 newly-developed items were deemed representative of the intended construct by experts and elicited construct-relevant response processes from examinees. The integration of expert review and examinee cognitive interviewing provides a more complete evaluation of the alignment of the items to their intended construct. Transparent reports of the methods and findings of content validity studies strengthen the ability of clinicians to select criterion-referenced assessments that support valid decisions.

目前学界针对标准参照评估（criterion-referenced assessments）遴选工作的循证实践指南尚未达成广泛共识。若要对标准参照评估的得分建立信心，需有证据证明评估条目与其预设构念（constructs）相符。本系列研究旨在为一款尚在研发中的社会沟通评估工具的修订条目集提供内容效度（content validity）证据，同时为临床医师提供一套可推广至其他评估工具审查的内容效度评价范式。在研究1中，10名专家对25条新研发的评估条目进行评分，以判断其对预设构念的契合程度。在研究2中，研究人员对7名年龄介于14至20岁的受试者实施了三步测试访谈（Three Step Test Interview），以考察其作答新条目时的认知过程。研究人员会对受试者的作答进行编码，以区分与构念相关及无关的影响因素。25条新研发条目中的23条被专家认定契合预设构念，且能引发受试者产生与构念相关的作答过程。将专家评审与受试者认知访谈相结合的方式，可更全面地评估评估条目与其预设构念的契合度。对内容效度研究的方法与结果进行透明化报告，可提升临床医师遴选可支撑有效决策的标准参照评估工具的能力。