Data underlying the publication: “It Should Be Relevant, Reliable and Feasible”: Introducing Face, an Instrument for Assessing the Face Validity of Choice Experiments

Face validity indicates to what extent participants are engaged in making choices; and understand and interpret the information presented to them in the study as intended by its designer. It is an important but often overlooked aspect of the overall validity of choice experiments and no comprehensive instruments for assessing it are available. Improving the design of choice experiments potentially improves the quality of participants’ responses, which increases the relevance and usability for policy and practice. In this study we developed and tested an instrument to assess the FAce validity of Choice Experiments (FACE) in a uniform, systematic manner. The instrument is based on nine components that are used to define face validity identified from literature: acceptance, clarity, completeness, familiarity, feasibility, legibility, relevance, sensitivity, and transparency. FACE covers these components in 14 questions with 5-point Likert scales on which participants can indicate their level of agreement. <br>1,020 participants completed the instrument following a discrete choice experiment on COVID-19 pandemic preparedness measures in the Netherlands. To evaluate and validate FACE, we used a four-step approach. First, we evaluated the internal consistency of the instrument using Cronbach’s Alpha. Second, we checked how suitable the data was for applying factor analysis. We evaluated the correlation matrix and its determinant, Barlett’s test of Sphericity and Kaiser-Meyer-Olkin’s (KMO) test. Third, we performed exploratory factor analysis to investigate how the components included in the instruments are, using principal component analysis with promax oblique rotations to account for expected correlation among factors. We used the Eigenvalue rule of λ ≥ 1.0 to retain factors. Finally, we converted the Likert scale answer categories into scores of 1 (fully disagree) to 5 (fully agree). Using these scores, we computed factor scores and factor-based scales for each participant, indicating how they related to each factor. We standardized the scores across all components, weighted these by score coefficients and summed them based on the regression method. Finally, we assessed the relationship between face validity scores and socio-demographic groups of participants using linear regression analyses with a dummy dependent variable that takes the value of 1 if the factor-based scale has a value below 25% of the distribution (i.e., lower face validity) and takes the value of 0 otherwise. We related it to respondents’ socio-demographic characteristics, experiment performance and attitude towards the presentation of the decision problem. <br>This first application of FACE showed that the face validity of a choice experiment was determined by whether participants considered its study design to be relevant, reliable and feasible. Moreover, we found that relevance and reliability were most strongly related to characteristics of the survey design, while feasibility was most strongly related to participants’ socio-demographic characteristics. Face validity was assessed high(er) by participants who were younger, male, lower educated, vaccinated against COVID-19, supportive of policy responses to a pandemic situation and sufficiently engaged in the experiment.

表面效度（Face validity）指参与者在研究中开展选择时的投入程度，以及能否按照研究设计者的意图，理解并解读实验中呈现的信息。该维度是选择实验整体效度中一项重要却常被忽视的内容，目前尚无通用工具可用于其评估。优化选择实验的设计，有望提升参与者的应答质量，进而增强其在政策制定与实践场景中的相关性与可用性。
本研究开发并测试了一套统一、系统的选择实验表面效度（Choice Experiments Face Validity，简称FACE）评估工具。该工具基于文献中明确的9项表面效度构成要素：接受度、清晰度、完整性、熟悉度、可行性、易读性、相关性、敏感性与透明度。FACE通过14道配备5点李克特（Likert）量表的问题覆盖上述要素，参与者可借此表明自身的认同水平。
共有1020名参与者在完成荷兰新冠疫情防控措施的离散选择实验（discrete choice experiment）后填写了该工具。为评估并验证FACE，我们采用了四步研究流程：首先，通过克朗巴哈α系数（Cronbach’s Alpha）评估工具的内部一致性；其次，检验数据是否适用于因子分析，具体包括对相关矩阵及其行列式、巴特利特球形检验（Barlett’s Test of Sphericity）与凯泽-迈耶-奥尔金检验（Kaiser-Meyer-Olkin，简称KMO）的评估；第三，采用带普罗马克斯斜交旋转（promax oblique rotations）的主成分分析（principal component analysis），以考量因子间的预期相关性，开展探索性因子分析以探究工具中各构成要素的结构，我们以特征值λ≥1.0作为保留因子的标准；最后，将李克特量表的作答类别转换为1（完全不同意）至5（完全同意）的分值。基于上述分值，我们为每位参与者计算因子得分与基于因子的量表得分，以体现其与各因子的关联程度。我们对所有要素的得分进行标准化处理，通过得分系数加权后基于回归法求和。最终，我们采用线性回归分析（linear regression analyses），以因子量表得分低于分布25%分位（即表面效度较低）时赋值为1、其余情况赋值为0的虚拟因变量，探究表面效度得分与参与者社会人口学分组之间的关联，并将该关联与受访者的社会人口学特征、实验参与表现以及对决策问题呈现方式的态度相联系。
FACE的首次应用结果显示，选择实验的表面效度取决于参与者是否认为该实验设计具备相关性、可靠性与可行性。此外，我们发现相关性与可靠性与问卷设计特征的关联最为紧密，而可行性则与参与者的社会人口学特征关联最强。年轻、男性、受教育程度较低、已接种新冠疫苗、支持疫情防控政策且充分参与实验的参与者，对表面效度的评分更高。