How Transparency and Reproducibility Can Increase Credibility in Policy Analysis: A Case Study of the Minimum Wage Policy Estimate

The analysis of public policies, even when performed by the best non-partisan
agencies, often lacks credibility (Manski, 2013). This allows policy makers to cherrypick between reports, or within a specific report, to select estimates that better match their beliefs. For example, in 2014 the Congressional Budget Office (CBO) produced a report on the effects of raising the minimum wage that was cited both by opponents and supporters of the policy, with each side accepting as credible only partial elements of the report. Lack of transparency and reproducibility (TR) in a policy report implies that its credibility relies on the reputation of the authors, and their organizations, instead of on a critical appraisal of the analysis.

This dissertation translates to policy analysis solutions developed to address the
lack of credibility in a different setting: the reproducibility crisis in science. I adapt the Transparency and Openness Promotion (TOP) guidelines (Nosek et al, 2015) to the policy analysis setting. The highest standards from the adapted guidelines involve the use of two key tools: dynamic documents that combine all elements of an analysis in one place, and open source version control (git). I then implement these high standards in a case study of the CBO report mentioned above, and present the complete analysis in the form of an open-source dynamic document. In addition to increasing the credibility of the case study analysis, this methodology brings attention to several components of the policy analysis that have been traditionally overlooked in academic research, for example the distribution of the losses used to pay for the increase in wages. Increasing our knowledge in these overlooked areas may prove most valuable to an evidence-based policy debate.

公共政策分析即便由最优秀的无党派机构进行，亦常显不足其可信度（Manski，2013）。此状况使得政策制定者得以在报告之间，或某一特定报告中，挑选与其信念更相吻合的估算值。例如，2014年，国会预算办公室（CBO）发布了一份关于提高最低工资影响的报告，该报告被政策反对者和支持者双方引用，而各自仅将报告中部分内容视为可信。政策报告中缺乏透明度和可重复性（TR）意味着其可信度依赖于作者及其所在机构的声誉，而非对该分析进行批判性评估。本篇论文将针对解决不同场景下可信度缺失的政策分析解决方案进行翻译：即科学领域中的可重复性危机。本研究将Nosek等人（2015）提出的透明度和开放性促进（TOP）指南（Nosek et al., 2015）应用于政策分析环境。该指南的最高标准涉及两种关键工具的使用：将分析的所有元素集中在一起的动态文档，以及开源版本控制（git）。随后，本研究将上述高标准应用于CBO报告的案例研究中，并以开源动态文档的形式呈现完整的分析。除了提高案例研究分析的可信度外，此方法还引起了学术研究中传统上被忽视的政策分析几个方面的关注，例如用于支付工资增加所造成的损失分布。对这些被忽视领域的知识增加可能对基于证据的政策辩论具有极大的价值。