CLUTRR

CLUTRR数据集是由加拿大麦吉尔大学计算机科学学院的研究团队创建，专注于评估自然语言理解（NLU）系统在文本关系推理任务中的系统性泛化能力和鲁棒性。该数据集包含6016个独特的重述，这些重述被用于生成一系列半合成的故事，旨在测试模型在处理未见过的逻辑规则组合时的推理能力。CLUTRR数据集通过精确控制推理的复杂性，允许研究人员生成多样化的测试案例，从而更深入地了解NLU系统的性能。此外，数据集还允许通过添加不同类型的噪声事实来评估模型的鲁棒性，这些噪声事实包括无关事实、支持性事实和断开连接的事实，从而全面测试模型在复杂环境下的表现。

CLUTRR Dataset was created by a research team from the School of Computer Science, McGill University, Canada, focusing on evaluating the systematic generalization ability and robustness of natural language understanding (NLU) systems in textual relational reasoning tasks. This dataset contains 6,016 unique paraphrases, which are used to generate a series of semi-synthetic stories designed to test models' reasoning capabilities when handling unseen combinations of logical rules. By precisely controlling the complexity of reasoning, the CLUTRR dataset allows researchers to generate diverse test cases, enabling a deeper understanding of the performance of NLU systems. Additionally, the dataset enables evaluation of model robustness by adding different types of noisy facts, including irrelevant facts, supportive facts, and disconnected facts, thereby comprehensively testing model performance in complex environments.