LeandraFichtel/KAMEL

---

# Dataset Card for KAMEL: Knowledge Analysis with Multitoken Entities in Language Models
## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)
## Dataset Description
- **Homepage:**
https://github.com/JanKalo/KAMEL
- **Repository:**
https://github.com/JanKalo/KAMEL
- **Paper:**
@inproceedings{kalo2022kamel,
title={KAMEL: Knowledge Analysis with Multitoken Entities in Language Models},
author={Kalo, Jan-Christoph and Fichtel, Leandra},
booktitle={Automated Knowledge Base Construction},
year={2022}
}
### Dataset Summary
This dataset provides the data for KAMEL, a probing dataset for language models that contains factual knowledge
from Wikidata and Wikipedia.

See the paper for more details. For more information, also see:
https://github.com/JanKalo/KAMEL
### Languages
en
## Dataset Structure
### Data Instances

### Data Fields
KAMEL has the following fields:
* index: the id
* sub_label: a label for the subject
* obj_uri: Wikidata uri for the object
* obj_labels: multiple labels for the object
* chosen_label: the preferred label
* rel_uri: Wikidata uri for the relation
* rel_label: a label for the relation

### Data Splits
The dataset is split into a training, validation, and test dataset.
It contains 234 Wikidata relations.
For each relation there exist 200 training, 100 validation,
and 100 test instances.

## Dataset Creation
### Curation Rationale
This dataset was gathered and created to explore what knowledge graph facts are memorized by large language models.
### Source Data
#### Initial Data Collection and Normalization
See the reaserch paper and website for more detail. The dataset was
created from Wikidata and Wikipedia.
### Annotations
#### Annotation process
There is no human annotation, but only automatic linking from Wikidata facts to Wikipedia articles.
The details about the process can be found in the paper.
#### Who are the annotators?
Machine Annotations
### Personal and Sensitive Information
Unkown, but likely information about famous people mentioned in the English Wikipedia.
## Considerations for Using the Data
### Social Impact of Dataset
The goal for the work is to probe the understanding of language models.
### Discussion of Biases
Since the data is created from Wikipedia and Wikidata, the existing biases from these two data sources may also be reflected in KAMEL.
## Additional Information
### Dataset Curators
The authors of KAMEL at Vrije Universiteit Amsterdam and Technische Universität Braunschweig.
### Licensing Information
The Creative Commons Attribution-Noncommercial 4.0 International License. see https://github.com/facebookresearch/LAMA/blob/master/LICENSE
### Citation Information
@inproceedings{kalo2022kamel,
title={KAMEL: Knowledge Analysis with Multitoken Entities in Language Models},
author={Kalo, Jan-Christoph and Fichtel, Leandra},
booktitle={Automated Knowledge Base Construction},
year={2022}
}

原始信息汇总

数据集概述

数据集描述

• 名称: KAMEL: Knowledge Analysis with Multitoken Entities in Language Models
• 摘要: 该数据集为KAMEL项目提供数据，是一个用于语言模型的探测数据集，包含来自Wikidata和Wikipedia的事实知识。
• 语言: 英语 (en)

数据集结构

• 数据实例: 数据集分为训练、验证和测试集，包含234个Wikidata关系，每个关系对应200个训练实例、100个验证实例和100个测试实例。
• 数据字段:
  • index: 标识符
  • sub_label: 主体标签
  • obj_uri: 对象的Wikidata URI
  • obj_labels: 对象的多个标签
  • chosen_label: 首选标签
  • rel_uri: 关系的Wikidata URI
  • rel_label: 关系标签

数据集创建

• 采集与规范化: 数据集来源于Wikidata和Wikipedia，详细过程见研究论文和网站。
• 注释过程: 无人工注释，通过自动链接Wikidata事实到Wikipedia文章。
• 个人和敏感信息: 可能包含英语Wikipedia中提及的名人信息。

使用数据集的考虑

• 社会影响: 旨在探测语言模型的理解能力。
• 偏见讨论: 数据源自Wikipedia和Wikidata，可能反映这两个数据源的现有偏见。

附加信息

• 数据集维护者: Vrije Universiteit Amsterdam和Technische Universität Braunschweig的KAMEL作者。
• 许可信息: 遵循Creative Commons Attribution-Noncommercial 4.0 International License。
• 引用信息: 见数据集详情页中的引用格式。