Reinforced Designs: Multiple Instruments Plus Control Groups as Evidence Factors in an Observational Study of the Effectiveness of Catholic Schools

Absent randomization, causal conclusions gain strength if several independent evidence factors concur. We develop a method for constructing evidence factors from several instruments plus a direct comparison of treated and control groups, and we evaluate the methods performance in terms of design sensitivity and simulation. In the application, we consider the effectiveness of Catholic versus public high schools, constructing three evidence factors from three past strategies for studying this question, namely: (i) having nearby access to a Catholic school as an instrument, (ii) being Catholic as an instrument for attending Catholic school, and (iii) a direct comparison of students in Catholic and public high schools. Although these three analyses use the same data, we: (i) construct three essentially independent statistical tests of no effect that require very different assumptions, (ii) study the sensitivity of each test to the assumptions underlying that test, (iii) examine the degree to which independent tests dependent upon different assumptions concur, (iv) pool evidence across independent factors. In the application, we conclude that the ostensible benefit of Catholic education depends critically on the validity of one instrument, and is therefore quite fragile. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

在未实施随机化的场景中，若多个独立证据因子（evidence factors）达成一致，则因果推断结论的说服力将得以增强。我们提出了一种从多个工具变量（instrument）以及处理组与对照组的直接对比中构建证据因子的方法，并从设计敏感性与模拟实验两个维度对该方法的性能进行了评估。在实证应用中，我们聚焦于天主教学校与公立高中的办学效果差异，基于此前用于研究该问题的三种策略构建了三个证据因子，具体为：(i) 以就近就读天主教学校作为工具变量；(ii) 以自身宗教信仰为天主教徒作为就读天主教学校的工具变量；(iii) 对天主教学校与公立高中的学生进行直接对比分析。尽管这三项分析使用了同一数据集，但我们完成了以下工作：(i) 构建了三个本质独立的无效应统计检验，且各检验所需的假设前提差异显著；(ii) 探究了每项检验对其自身假设前提的敏感性；(iii) 考察了基于不同假设的独立检验之间的一致程度；(iv) 对不同独立证据因子的证据进行合并。在本次实证应用中，我们得出结论：天主教教育的表面收益关键依赖于某一工具变量的有效性，因此该结论的稳定性较差。本文的补充材料（包括可用于复现研究的标准化材料说明）可作为在线补充材料获取。