CREAK

CREAK是一个旨在评估自然语言处理模型在实体理解和常识推理方面能力的数据集。该数据集由德克萨斯大学奥斯汀分校的研究人员创建，包含13,000个人工编写的关于实体的英语陈述，这些陈述被标记为真或假。数据集的创建过程涉及众包工作者基于维基百科实体生成陈述，鼓励从零开始创造性编写，以确保陈述结合了实体知识和常识。CREAK数据集的应用领域包括测试模型对实体知识的检索能力和未明确说明的常识知识，旨在解决模型在处理涉及实体的常识推理时的挑战。

CREAK is a dataset aimed at evaluating the capabilities of natural language processing models in entity understanding and commonsense reasoning. Developed by researchers at The University of Texas at Austin, it contains 13,000 manually written English statements about entities, each labeled as either true or false. During the dataset construction process, crowdworkers generate statements based on Wikipedia entities and are encouraged to conduct creative writing from scratch to ensure that the statements integrate both entity-specific knowledge and commonsense knowledge. Application areas of the CREAK dataset include testing models' abilities to retrieve entity knowledge and utilize unstated commonsense knowledge, with the goal of tackling the challenges faced by models when handling commonsense reasoning tasks involving entities.