Evaluation synthesis analysis can be accelerated through text mining, searching, and highlighting: A case-study on data extraction from 631 UNICEF evaluation reports

Abstract Background: The United Nations Children's Fund (UNICEF) is the United Nations agency dedicated to promoting and advocating for the protection of children's rights, meeting their basic needs, and expanding their opportunities to reach their full potential. They achieve this by working with governments, communities, and other partners via programmes that safeguard children from violence, provide access to quality education, ensure that children survive and thrive, provide access to water, sanitation and hygiene, and provide life-saving support in emergency contexts. Programmes are evaluated as part of UNICEF Evaluation Policy , and the publicly available reports include a wealth of information on results, recommendations, and lessons learned. Objective: To critically explore UNICEF’s impact, a systematic synthesis of evaluations was conducted to provide a summary of UNICEF main achievements and areas where they could improve, as a reflection of key recommendations, lessons learned, enablers, and barriers to achieving their goals and to steer its future direction and strategy. Since the evaluations are extensive, manual analysis was not feasible, so a semi-automated approach was taken. Methods: This paper examines the automation techniques used to try and increase the feasibility of undertaking broad evaluation syntheses analyses. Our semi-automated human-in-the-loop methods supported data extraction of data for 64 outcomes across 631 evaluation reports; each of which comprised hundreds of pages of text. The outcomes are derived from the five goal areas within UNICEF 2022-2025 Strategic Plan. For text pre-processing we implemented PDF-to-text extraction, section parsing, and sentence mining via a neural network. Data extraction was supported by a freely available text-mining workbench, SWIFT-Review. Here, we describe using comprehensive adjacency-search-based queries to rapidly filter reports by outcomes and to highlight relevant sections of text to expedite data extraction. Results: While the methods used were not expected to produce 100% complete results for each outcome, they present useful automation methods for researchers facing otherwise non-feasible evaluation syntheses tasks. We reduced the text volume down to 8% using deep learning (recall 0.93) and rapidly identified relevant evaluations across outcomes with a median precision of 0.6. All code is available and open-source. Conclusions: When the classic approach of systematically extracting information from all outcomes across all texts exceeds available resources, the proposed automation methods can be employed to speed up the process while retaining scientific rigour and reproducibility.

**研究背景**
联合国儿童基金会（UNICEF）是联合国下属专门机构，致力于推动并倡导保护儿童权利、满足儿童基本需求，拓展儿童充分发挥自身潜能的机会。该机构通过与各国政府、社区及其他合作伙伴携手开展各类项目实现上述目标：项目内容涵盖保护儿童免遭暴力侵害、保障儿童获得优质教育、确保儿童存活并健康成长、为儿童提供水、环境卫生与个人卫生服务，以及在紧急情境中提供挽救生命的支持。根据《联合国儿童基金会评估政策》（UNICEF Evaluation Policy），所有项目均需接受评估，而公开的评估报告中蕴含了大量关于项目成果、建议及经验教训的信息。

**研究目标**
本研究旨在批判性探究联合国儿童基金会的工作成效，通过对评估报告开展系统性综合分析，总结其主要成就与可改进方向，同时梳理实现其目标过程中的关键建议、经验教训、推动因素与阻碍因素，为其未来发展方向与战略规划提供参考。鉴于评估报告体量庞大，人工分析不具备可行性，因此本研究采用半自动化研究方法。

**研究方法**
本文探讨了为提升大规模评估综合分析可行性所采用的自动化技术。我们采用的半自动化人机协同方法，支持对631份评估报告中涵盖的64项成果进行数据提取——每份报告均包含数百页文本。这些成果源自《联合国儿童基金会2022-2025年战略计划》中的五大目标领域。文本预处理环节中，我们实现了PDF转文本提取、章节解析，并通过神经网络开展语句挖掘。数据提取工作依托一款免费开源的文本挖掘工作台SWIFT-Review完成。本文还介绍了如何利用基于邻接搜索的综合查询，按成果类型快速筛选报告，并高亮显示相关文本段落，以加快数据提取进度。

**研究结果**
尽管所采用的方法无法确保每项成果的提取结果达到100%完整，但它们为面临近乎无法完成的评估综合分析任务的研究人员提供了实用的自动化方案。通过深度学习（deep learning）技术，我们将文本体量缩减至原有的8%（召回率（recall）0.93），并快速识别出与各成果相关的评估报告，其精准度（precision）中位数达0.6。所有代码均已开源发布。

**研究结论**
当传统的系统性提取所有文本中所有成果信息的方法超出可用资源承载范围时，本研究提出的自动化方法可用于加快分析进程，同时保留科学严谨性与可重复性。