Evidence to support test interpretation: Evaluating the content validity of a new item set for the Transition Pragmatics Interview

There is little consensus on evidence-based practice guidelines for the selection of criterion-referenced assessments. Having confidence in scores from criterion-referenced assessments requires evidence that items align with their intended constructs. The purposes of these studies were to demonstrate evidence of content validity for the revised item set of a developing social communication assessment and to provide clinicians with a model of content validity evaluations that can be generalised to the review of other assessments. In Study 1, 10 experts rated 25 newly-developed items for how well they represented the intended construct. In Study 2, seven participants ages 14–20 were administered the Three Step Test Interview to assess their cognitive processes for responding to new items. Examinee responses were coded for construct-relevant and construct-irrelevant factors. Twenty-three of the 25 newly-developed items were deemed representative of the intended construct by experts and elicited construct-relevant response processes from examinees. The integration of expert review and examinee cognitive interviewing provides a more complete evaluation of the alignment of the items to their intended construct. Transparent reports of the methods and findings of content validity studies strengthen the ability of clinicians to select criterion-referenced assessments that support valid decisions.

目前针对标准参照评估（criterion-referenced assessments）的遴选，尚未形成统一的循证实践指南共识。若要确保标准参照评估得分的可信度，需依托相关证据证明测试条目与其预设构念（construct）相契合。本系列研究旨在达成两个目标：一是为某款在研社交沟通评估工具的修订条目集提供内容效度（content validity）证据；二是为临床工作者提供一套可推广至其他评估工具审核的内容效度评估范式。在研究1中，10名专家对25条新开发的测试条目进行评分，以评估其对预设构念的代表性程度。在研究2中，招募7名年龄介于14至20岁的受试者，采用三步测试访谈（Three Step Test Interview）评估其应对新条目时的认知加工过程。研究人员对受试者的作答进行编码，区分构念相关因素与构念无关因素。25条新开发的测试条目当中，有23条被专家认定为符合预设构念的代表性条目，且能够引发受试者产生与构念相关的作答过程。将专家评审与受试者认知访谈相结合的评估路径，可更全面地验证测试条目与预设构念的契合度。透明化呈现内容效度研究的方法与研究结果，能够增强临床工作者遴选可支撑有效决策的标准参照评估工具的能力。