UKPLab/scicoqa

SciCoQA是一个旨在检测科学出版物与其代码库之间差异的数据集，以确保代码忠实实现论文中描述的方法，解决AI和计算科学中的可重复性危机。数据集共包含611个论文-代码差异实例，包括81个真实世界实例和530个合成实例，涵盖AI、物理学、定量生物学等多个计算科学学科。数据集由达姆施塔特工业大学UKP实验室的Tim Baumgärtner和Iryna Gurevych策划，主要用于评估大型语言模型在科学自动化质量保证任务中的表现。

SciCoQA is a dataset designed to detect discrepancies between scientific publications and their codebases to ensure faithful implementations. The dataset addresses the challenge of ensuring that the code faithfully implements the methods reported in the scientific paper, a critical aspect of solving the reproducibility crisis in AI and computational sciences. The dataset contains a total of 611 paper-code discrepancies, consisting of 81 real-world instances and 530 synthetic instances. These discrepancies span diverse computational science disciplines, including AI, Physics, Quantitative Biology, and others. Curated by Tim Baumgärtner and Iryna Gurevych (Ubiquitous Knowledge Processing Lab (UKP Lab), Technical University of Darmstadt), the dataset is primarily used to benchmark Large Language Models (LLMs) on the task of automated quality assurance in science.