SyROCCo dataset

The peer-reviewed publication for this dataset has been published in Data & Policy, and can be accessed here: https://arxiv.org/abs/2406.16527 Please cite this when using the dataset. This dataset has been produced as a result of the “Systematic Review of Outcomes Contracts using Machine Learning” (SyROCCo) project. The goal of the project was to apply machine learning techniques to a systematic review process of outcomes-based contracting (OBC). The purpose of the systematic review was to gather and curate, for the first time, all of the existing evidence on OBC. We aimed to map the current state of the evidence, synthesise key findings from across the published studies, and provide accessible insights to our policymaker and practitioner audiences. OBC is a model for the provision of public services wherein a service provider receives payment, in-part or in-full, only upon the achievement of pre-agreed outcomes. The data used to conduct the review consists of 1,952 individual studies of OBC. They include peer reviewed journal articles, book chapters, doctoral dissertations, and assorted ‘grey literature’ - that is, reports and evaluations produced outside of traditional academic publications. Those studies were manually filtered by experts on the topic from an initial search of over 11,000 results. The full text of the articles was obtained from their PDF versions and preprocessed. This involved text format normalisation, removing acknowledgements and bibliographic references. The corpus was then connected to the INDIGO Impact Bond Dataset. Projects and organisations mentioned in this latter dataset were searched for in the article’s corpus to relate both datasets. Other types of information that were identified in the texts were 1) financial mechanisms (type of outcomes-based instrument); using a list of terms related to those financial mechanisms based on prior discussions with a policy advisory group (Picker et al., 2021); 2) references to the 17 Sustainable Development Goals (SDGs) defined by the United Nations General Assembly in the 2030 Agenda; 3) country names mentioned in each article and income levels related to the countries; according to the World Classification of Income Levels 2022 by the World Bank. Three machine learning techniques were applied to the corpus: Policy areas identification. A query-driven topic model (QDTM) (Fang et al., 2021) was used to determine the probability of an article belonging to different policy areas (health, education, homelessness, criminal justice, employment and training, child and family welfare, and agriculture and environment), using all text of the article as input. The QDTM is a semi-supervised machine learning algorithm that allows users to specify their prior knowledge in the form of simple queries in words or phrases and return query-related topics. Named Entity Recognition. Three named entity recognition models were applied: “en_core_web_lg” and “en_core_web_trf” models from the python package ‘spaCy’ and the “ner-ontonotes-large” English model from ‘Flair’. “en_core_web_trf” is based on the RoBERTa-base transformer model. ‘Flair’ is a bi-LSTM character-based model. All models were trained on the “OntoNotes 5” data source (Marcus et al., 2011) and are able to identify geographical locations, organisation names, and laws and regulations. An ensemble method was adopted, considering the entities that appear simultaneously in the results of any two models as the correct entities. Semantic text similarity. We calculated the similarity score between articles. The 10,000 most frequently mentioned words were first extracted from all the articles’ titles and abstracts and the text vectorization technique TF*IDF was applied to convert each article’s abstract into an importance score vector based on these words. Using these numerical vectors, the cosine similarity between different articles was calculated. The SyROCCo Dataset includes references to the 1952 studies of OBCs mentioned above and the results of the previous processing steps and techniques. Each entry of the dataset contains the following information. The basic information of each document is its title, abstract, authors, published years, DOI and Article ID: Title: Title of the document. Abstract: Text of the abstract. Authors: Authors of a study. Published Years: Published Years of a study. DOI: DOI link of a study. Article ID: ID of the document selected during the screening process. The probability of a study belonging to each policy area: policy_sector_health: The probability of a study belongs to the policy sector “health”. policy_sector_education: The probability of a study belongs to the policy sector “education”. policy_sector_homelessness: The probability of a study belongs to the policy sector “homelessness”. policy_sector_criminal: The probability of a study belongs to the policy sector “criminal” policy_sector_employment: The probability of a study belongs to the policy sector “employment” policy_sector_child: The probability of a study belongs to the policy sector “child”. policy_sector_environment: The probability of a study belongs to the policy sector “environment”. Other types of information such as financial mechanisms, Sustainable Development Goals, and different types of named entities: financial_mechanisms: Financial mechanisms mentioned in a study. top_financial_mechanisms: The financial mechanisms mentioned in a study are listed in descending order according to the number of times they are mentioned, and include the corresponding context of the mentions. top_sgds: Sustainable Development Goals mentioned in a study are listed in descending order according to the number of times they are mentioned, and include the corresponding context of the mentions. top_countries: Country names mentioned in a study are listed in descending order according to the number of times they are mentioned, and include the corresponding context of the mentions. This entry is also used to determine the income level of the mentioned counties. top_Project: Indigo projects mentioned in a study are listed in descending order according to the number of times they are mentioned, and include the corresponding context of the mentions. top_GPE: Geographical locations mentioned in a study are listed in descending order according to the number of times they are mentioned, and include the corresponding context of the mentions. top_LAW: Relevant laws and regulations mentioned in a study are listed in descending order according to the number of times they are mentioned, and include the corresponding context of the mentions. top_ORG: Organisations mentioned in a study are listed in descending order according to the number of times they are mentioned, and include the corresponding context of the mentions.本数据集的同行评审论文已发表于期刊《Data & Policy》，原文可通过链接 https://arxiv.org/abs/2406.16527 查阅。使用本数据集时，请务必引用该文献。 本数据集源自"基于机器学习的成果导向合同系统综述"（Systematic Review of Outcomes Contracts using Machine Learning, SyROCCo）项目。该项目旨在将机器学习技术应用于成果导向合同（Outcomes-Based Contracting, OBC）的系统综述流程。本次系统综述的核心目标为：首次系统性收集并整理所有已公开的OBC相关研究证据，梳理现有证据的整体研究现状，整合已发表研究的关键结论，并为政策制定者与实务工作者群体提供易于理解的研究洞察。 OBC是一种公共服务提供模式，服务提供商仅在达成预先约定的服务成果后，才能获得全部或部分报酬。 本次综述所用的数据包含1952项OBC相关独立研究，涵盖同行评议期刊论文、图书章节、博士学位论文，以及各类"灰色文献"（即传统学术出版物之外产出的报告与评估材料）。研究团队先通过初始检索获得超11000条结果，再由该领域专家人工筛选出最终纳入的研究。 研究人员从PDF版本中获取所有文章的全文，并进行预处理：包括文本格式标准化、剔除致谢与参考文献内容。 随后，本语料库与INDIGO影响力债券数据集（INDIGO Impact Bond Dataset）进行关联：通过在文章语料库中检索该数据集提及的项目与机构，实现两个数据集的关联匹配。 研究人员从文本中提取了三类其他信息：1）金融机制（即成果导向工具类型）：参考与政策咨询小组前期研讨形成的金融机制相关术语清单（Picker等，2021）；2）联合国大会在《2030年可持续发展议程》中定义的17项可持续发展目标（Sustainable Development Goals, SDGs）相关提及内容；3）每篇文章中提及的国家名称，以及依据世界银行《2022年世界收入分类标准》划分的对应国家收入层级。 研究团队针对该语料库应用了三种机器学习技术： 1. 政策领域识别：采用查询驱动主题模型（Query-Driven Topic Model, QDTM）（Fang等，2021），以文章全文作为输入，计算文章归属至不同政策领域（医疗、教育、无家可归者救助、刑事司法、就业与培训、儿童与家庭福利、农业与环境）的概率。QDTM是一种半监督机器学习算法，允许用户以单词或短语构成的简单查询形式指定先验知识，并返回与查询相关的主题。 2. 命名实体识别（Named Entity Recognition, NER）：研究团队应用了三款命名实体识别模型：来自Python包spaCy的"en_core_web_lg"与"en_core_web_trf"模型，以及来自Flair的"ner-ontonotes-large"英文模型。其中"en_core_web_trf"基于RoBERTa-base Transformer模型构建，Flair则是一款基于字符的双向长短期记忆网络（bi-LSTM）模型。所有模型均基于OntoNotes 5数据集（Marcus等，2011）训练，可识别地理位置、机构名称以及法律法规内容。研究采用集成方法，将同时出现在任意两款模型识别结果中的实体认定为正确实体。 3. 语义文本相似度：研究团队计算了文章间的相似度得分。首先从所有文章的标题与摘要中提取出现频率最高的10000个词汇，再采用文本向量化技术TF*IDF，将每篇文章的摘要转换为基于这些词汇的重要性得分向量。基于这些数值向量，研究人员计算了不同文章间的余弦相似度。 SyROCCo数据集包含上述1952项OBC研究的相关信息，以及前述预处理步骤与机器学习技术的产出结果。数据集的每条记录包含以下信息： 1. 文档基础信息：包括文档标题、摘要、作者、发表年份、DOI以及筛选过程中分配的文档ID： - 标题：文档的标题 - 摘要：文档的摘要文本 - 作者：研究的作者列表 - 发表年份：研究的发表年份 - DOI：研究的DOI链接 - 文档ID：筛选过程中为该文档分配的唯一标识 2. 研究归属各政策领域的概率： - policy_sector_health：研究归属"医疗"政策领域的概率 - policy_sector_education：研究归属"教育"政策领域的概率 - policy_sector_homelessness：研究归属"无家可归者救助"政策领域的概率 - policy_sector_criminal：研究归属"刑事司法"政策领域的概率 - policy_sector_employment：研究归属"就业与培训"政策领域的概率 - policy_sector_child：研究归属"儿童与家庭福利"政策领域的概率 - policy_sector_environment：研究归属"农业与环境"政策领域的概率 3. 其他类型信息，包括金融机制、可持续发展目标以及各类命名实体： - financial_mechanisms：研究中提及的金融机制 - top_financial_mechanisms：按提及频次降序排列的研究中提及的金融机制，同时包含对应提及上下文 - top_sgds：按提及频次降序排列的研究中提及的可持续发展目标，同时包含对应提及上下文 - top_countries：按提及频次降序排列的研究中提及的国家名称，同时包含对应提及上下文。该字段同时用于确定所提及国家的收入层级 - top_Project：按提及频次降序排列的研究中提及的INDIGO项目，同时包含对应提及上下文 - top_GPE：按提及频次降序排列的研究中提及的地理位置，同时包含对应提及上下文 - top_LAW：按提及频次降序排列的研究中提及的相关法律法规，同时包含对应提及上下文 - top_ORG：按提及频次降序排列的研究中提及的机构名称，同时包含对应提及上下文