Data_Sheet_2_Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis.CSV

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products.
Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments.
Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability.
Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.

背景：9·11事件与2002年10月发布的《伊拉克大规模杀伤性武器持续项目国家情报评估》（National Intelligence Estimate）促成了美国情报界的根本性变革。作为此次改革的组成部分，情报分析实务标准被修订并编纂为政策文件《情报共同体指令（Intelligence Community Directive, ICD）203》，同时在新设立的国家情报总监办公室中任命了一名分析监察员，以确保整个情报界的合规性。本研究旨在验证一项尚未经检验的假设：即ICD203标准可用于对情报分析成果开展可靠评估。
方法：15名独立评分者采用基于ICD203标准的评分细则，对64份针对假想情报问题撰写的分析报告的推理质量进行评估。本研究计算了单个评分与分组汇总评分的组内相关系数（intra-class correlation coefficients, ICC）。
结果：尽管经过了统一培训与评分者校准，单个评分的可靠性仍较差；但汇总后的评分则展现出良好至极佳的可靠性。
结论：考虑到实际情报问题相较本研究的假想案例更为复杂棘手，本研究建议，若要对情报成果实施可靠的质量管控流程，至少需要3名评分者组成评分小组。本研究明确了评估可靠性的边界，并为进一步评估符合分析实务标准的情报报告的预测效度提供了研究基础。