OpenBookQA

“OpenBookQA 是一种新的问答数据集，它以开卷考试为模型，用于评估人类对学科的理解。它由 5,957 个多项选择的初级科学问题（4,957 个训练，500 个开发，500 个测试）组成，它探讨了对 1,326 个核心科学事实的小“书”的理解以及这些事实在新情况中的应用。对于训练，数据集包括从每个问题到它旨在探索的核心科学事实的映射。回答 OpenBookQA 问题需要书中未包含的其他广泛的常识。这些问题在设计上会被基于检索的算法和单词共现算法错误地回答。此外，数据集包括 5,167 个众包常识的集合事实，以及训练/开发/测试问题的扩展版本，其中每个问题都与其原始核心事实、人类准确性分数、清晰度分数和匿名人群相关联rker ID。”

OpenBookQA is a novel question answering dataset modeled after open-book exams, designed to evaluate human understanding of academic subjects. It comprises 5,957 multiple-choice middle school science questions, split into 4,957 training instances, 500 development instances, and 500 test instances. This dataset explores the comprehension of a small "book" of 1,326 core scientific facts and the application of these facts in novel scenarios. For the training split, the dataset includes mappings from each question to the core scientific fact it is intended to explore. Answering OpenBookQA questions requires extensive additional common knowledge not contained within this book. These questions are intentionally designed to be incorrectly answered by retrieval-based algorithms and word co-occurrence based models. Additionally, the dataset includes a collection of 5,167 crowdsourced common knowledge facts, as well as extended versions of the training, development and test questions, where each question is associated with its original core fact, human accuracy score, clarity score, and anonymous crowdworker ID.