ALCUNE
收藏资源简介：
ALCUNA是由北京大学王选计算机研究所、数据科学中心和计算语言学教育部重点实验室联合开发的基准数据集，旨在评估大型语言模型（LLMs）面对新知识时的知识理解、区分和关联能力。该数据集基于EOL（生命百科全书）数据库构建，通过KnowGen方法生成人工生物实体及其属性和关系，包含84351个问题，涵盖3554个人工实体，平均每个实体具有11.75个属性三元组和25.39个兄弟实体。数据集通过合理改变现有实体属性和关系生成新知识，模拟真实世界中模型遭遇新知识的场景，可用于评估LLMs在新知识场景下的性能，帮助推动模型在新知识处理方面的发展。数据集已发布，可访问相关链接获取更多信息。
ALCUNA is a benchmark dataset jointly developed by the Wangxuan Computer Research Institute, Data Science Center, and Key Laboratory of Computational Linguistics of the Ministry of Education at Peking University. It is designed to evaluate the knowledge understanding, distinction, and association capabilities of large language models (LLMs) in the face of new knowledge. The dataset is constructed based on the EOL (Encyclopedia of Life) database and generated artificial biological entities, their attributes, and relationships using the KnowGen method. It contains 84,351 questions, covering 3,554 artificial entities, with an average of 11.75 attribute triples and 25.39 sibling entities per entity. The dataset generates new knowledge by rationally altering existing entity attributes and relationships, simulating real-world scenarios where models encounter new knowledge. It can be used to assess the performance of LLMs in new knowledge scenarios and help promote the development of models in the processing of new knowledge. The dataset has been released, and more information can be obtained through the provided link.
ALCUNA数据集概述
基本信息
- 数据集名称：ALCUNA: Large Language Models Meet New Knowledge
- 论文链接：https://arxiv.org/abs/2310.14820v1
- 数据集下载地址：https://drive.google.com/drive/folders/1P2Yt4XM-uSzfJoec4psIhpk-mfm-K3R1?usp=share_link
数据集结构
元数据 (dataset/meta_data.jsonl)
-
每行对应一个人工实体的元数据信息：
artificial_entity：人工实体的信息name：人工实体的名称id：父实体ID的负值rank：生物分类学中的等级property：属性结构列表
parent_entity：父实体的信息difference：生成人工实体属性的差异extension：来自其他实体的属性结构列表variation：元组列表(old_property, new_property)heredity：从父实体继承的属性结构列表dropout：未继承到人工实体的父实体属性结构列表
-
属性结构：
name：当前属性的名称type：当前属性的类型（attribute或relation）values：当前属性的有效值列表
问题数据集 (dataset/id2question.json)
- 包含从人工实体ID到对应问题的字典映射：
question：问题文本answers：所有有效答案列表form：问题形式（boolean、fill-in-blank、multi-choice）type：问题子集类型（Knowledge Understanding、Knowledge Differentiation、Knowledge Association）meta_data：related_property：与问题相关的人工实体属性difference：related_property的差异类型（extension、variation、heredity、dropout）hop_triplets（可选）：多跳问题对应的关系三元组链（仅限Knowledge Association数据集）
引用信息
bibtex @misc{yin2023alcuna, title={ALCUNA: Large Language Models Meet New Knowledge}, author={Xunjian Yin and Baizhou Huang and Xiaojun Wan}, year={2023}, eprint={2310.14820}, archivePrefix={arXiv}, primaryClass={cs.CL} }
