alexandrainst/dane
收藏Hugging Face2024-01-18 更新2024-05-25 收录
下载链接：
https://hf-mirror.com/datasets/alexandrainst/dane
下载链接
链接失效反馈官方服务：
资源简介：
---
annotations_creators:
- expert-generated
language_creators:
- found
language:
- da
license:
- cc-by-sa-4.0
multilinguality:
- monolingual
size_categories:
- 1K<n<10K
source_datasets:
- extended|other-Danish-Universal-Dependencies-treebank
task_categories:
- token-classification
task_ids:
- named-entity-recognition
- part-of-speech
paperswithcode_id: dane
pretty_name: DaNE
dataset_info:
features:
- name: sent_id
dtype: string
- name: text
dtype: string
- name: tok_ids
sequence: int64
- name: tokens
sequence: string
- name: lemmas
sequence: string
- name: pos_tags
sequence:
class_label:
names:
'0': NUM
'1': CCONJ
'2': PRON
'3': VERB
'4': INTJ
'5': AUX
'6': ADJ
'7': PROPN
'8': PART
'9': ADV
'10': PUNCT
'11': ADP
'12': NOUN
'13': X
'14': DET
'15': SYM
'16': SCONJ
- name: morph_tags
sequence: string
- name: dep_ids
sequence: int64
- name: dep_labels
sequence:
class_label:
names:
'0': parataxis
'1': mark
'2': nummod
'3': discourse
'4': compound:prt
'5': reparandum
'6': vocative
'7': list
'8': obj
'9': dep
'10': det
'11': obl:loc
'12': flat
'13': iobj
'14': cop
'15': expl
'16': obl
'17': conj
'18': nmod
'19': root
'20': acl:relcl
'21': goeswith
'22': appos
'23': fixed
'24': obl:tmod
'25': xcomp
'26': advmod
'27': nmod:poss
'28': aux
'29': ccomp
'30': amod
'31': cc
'32': advcl
'33': nsubj
'34': punct
'35': case
- name: ner_tags
sequence:
class_label:
names:
'0': O
'1': B-PER
'2': I-PER
'3': B-ORG
'4': I-ORG
'5': B-LOC
'6': I-LOC
'7': B-MISC
'8': I-MISC
splits:
- name: train
num_bytes: 7311212
num_examples: 4383
- name: test
num_bytes: 909699
num_examples: 565
- name: validation
num_bytes: 940413
num_examples: 564
download_size: 1209710
dataset_size: 9161324
---
# Dataset Card for DaNE
## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)
## Dataset Description
- **Homepage:** [DaNE homepage](https://danlp-alexandra.readthedocs.io/en/latest/docs/datasets.html#dane)
- **Repository:** [Github](https://github.com/alexandrainst/danlp)
- **Paper:** [Aclweb](https://www.aclweb.org/anthology/2020.lrec-1.565)
- **Leaderboard:**
- **Point of Contact:**
### Dataset Summary
The Danish Dependency Treebank (DaNE) is a named entity annotation for the Danish Universal Dependencies treebank using the CoNLL-2003 annotation scheme.
The Danish UD treebank (Johannsen et al., 2015, UD-DDT) is a conversion of the Danish Dependency Treebank (Buch-Kromann et al. 2003) based on texts from Parole (Britt, 1998). UD-DDT has annotations for dependency parsing and part-of-speech (POS) tagging. The dataset was annotated with Named Entities for PER, ORG, and LOC by the Alexandra Institute in the DaNE dataset (Hvingelby et al. 2020).
### Supported Tasks and Leaderboards
Parts-of-speech tagging, dependency parsing and named entitity recognition.
### Languages
Danish
## Dataset Structure
### Data Instances
This is an example in the "train" split:
```python
{
'sent_id': 'train-v2-0\n',
'lemmas': ['på', 'fredag', 'have', 'SiD', 'invitere', 'til', 'reception', 'i', 'SID-hus', 'i', 'anledning', 'af', 'at', 'formand', 'Kjeld', 'Christensen', 'gå', 'ind', 'i', 'den', 'glad', 'tresser', '.'],
'dep_labels': [35, 16, 28, 33, 19, 35, 16, 35, 18, 35, 18, 1, 1, 33, 22, 12, 32, 11, 35, 10, 30, 16, 34],
'ner_tags': [0, 0, 0, 3, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0],
'morph_tags': ['AdpType=Prep', 'Definite=Ind|Gender=Com|Number=Sing', 'Mood=Ind|Tense=Pres|VerbForm=Fin|Voice=Act', '_', 'Definite=Ind|Number=Sing|Tense=Past|VerbForm=Part', 'AdpType=Prep', 'Definite=Ind|Gender=Com|Number=Sing', 'AdpType=Prep', 'Definite=Def|Gender=Neut|Number=Sing', 'AdpType=Prep', 'Definite=Ind|Gender=Com|Number=Sing', 'AdpType=Prep', '_', 'Definite=Def|Gender=Com|Number=Sing', '_', '_', 'Mood=Ind|Tense=Pres|VerbForm=Fin|Voice=Act', '_', 'AdpType=Prep', 'Number=Plur|PronType=Dem', 'Degree=Pos|Number=Plur', 'Definite=Ind|Gender=Com|Number=Plur', '_'],
'dep_ids': [2, 5, 5, 5, 0, 7, 5, 9, 7, 11, 7, 17, 17, 17, 14, 15, 11, 17, 22, 22, 22, 18, 5],
'pos_tags': [11, 12, 5, 7, 3, 11, 12, 11, 12, 11, 12, 11, 16, 12, 7, 7, 3, 9, 11, 14, 6, 12, 10],
'text': 'På fredag har SID inviteret til reception i SID-huset i anledning af at formanden Kjeld Christensen går ind i de glade tressere.\n',
'tokens': ['På', 'fredag', 'har', 'SID', 'inviteret', 'til', 'reception', 'i', 'SID-huset', 'i', 'anledning', 'af', 'at', 'formanden', 'Kjeld', 'Christensen', 'går', 'ind', 'i', 'de', 'glade', 'tressere', '.'],
'tok_ids': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
}
```
### Data Fields
Data Fields:
- q_id: a string question identifier for each example, corresponding to its ID in the Pushshift.io Reddit submission dumps.
- subreddit: One of explainlikeimfive, askscience, or AskHistorians, indicating which subreddit the question came from
- title: title of the question, with URLs extracted and replaced by URL_n tokens
- title_urls: list of the extracted URLs, the nth element of the list was replaced by URL_n
- sent_id: a string identifier for each example
- text: a string, the original sentence (not tokenized)
- tok_ids: a list of ids (int), one for each token
- tokens: a list of strings, the tokens
- lemmas: a list of strings, the lemmas of the tokens
- pos_tags: a list of strings, the part-of-speech tags of the tokens
- morph_tags: a list of strings, the morphological tags of the tokens
- dep_ids: a list of ids (int), the id of the head of the incoming dependency for each token
- dep_labels: a list of strings, the dependency labels
- ner_tags: a list of strings, the named entity tags (BIO format)
### Data Splits
| | train | validation | test |
|-------------|-------:|-----------:|-------:|
| # sentences | 4383 | 564 | 565 |
| # tokens | 80 378 | 10 322 | 10 023 |
## Dataset Creation
### Curation Rationale
[More Information Needed]
### Source Data
#### Initial Data Collection and Normalization
[More Information Needed]
#### Who are the source language producers?
[More Information Needed]
### Annotations
#### Annotation process
[More Information Needed]
#### Who are the annotators?
[More Information Needed]
### Personal and Sensitive Information
[More Information Needed]
## Considerations for Using the Data
### Social Impact of Dataset
[More Information Needed]
### Discussion of Biases
[More Information Needed]
### Other Known Limitations
[More Information Needed]
## Additional Information
### Dataset Curators
[More Information Needed]
### Licensing Information
[More Information Needed]
### Citation Information
[More Information Needed]
### Citation Information
```
@inproceedings{hvingelby-etal-2020-dane,
title = "{D}a{NE}: A Named Entity Resource for {D}anish",
author = "Hvingelby, Rasmus and
Pauli, Amalie Brogaard and
Barrett, Maria and
Rosted, Christina and
Lidegaard, Lasse Malm and
S{\o}gaard, Anders",
booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference",
month = may,
year = "2020",
address = "Marseille, France",
publisher = "European Language Resources Association",
url = "https://aclanthology.org/2020.lrec-1.565",
pages = "4597--4604",
abstract = "We present a named entity annotation for the Danish Universal Dependencies treebank using the CoNLL-2003 annotation scheme: DaNE. It is the largest publicly available, Danish named entity gold annotation. We evaluate the quality of our annotations intrinsically by double annotating the entire treebank and extrinsically by comparing our annotations to a recently released named entity annotation of the validation and test sections of the Danish Universal Dependencies treebank. We benchmark the new resource by training and evaluating competitive architectures for supervised named entity recognition (NER), including FLAIR, monolingual (Danish) BERT and multilingual BERT. We explore cross-lingual transfer in multilingual BERT from five related languages in zero-shot and direct transfer setups, and we show that even with our modestly-sized training set, we improve Danish NER over a recent cross-lingual approach, as well as over zero-shot transfer from five related languages. Using multilingual BERT, we achieve higher performance by fine-tuning on both DaNE and a larger Bokm{\aa}l (Norwegian) training set compared to only using DaNE. However, the highest performance isachieved by using a Danish BERT fine-tuned on DaNE. Our dataset enables improvements and applicability for Danish NER beyond cross-lingual methods. We employ a thorough error analysis of the predictions of the best models for seen and unseen entities, as well as their robustness on un-capitalized text. The annotated dataset and all the trained models are made publicly available.",
language = "English",
ISBN = "979-10-95546-34-4",
}
```
### Contributions
Thanks to [@ophelielacroix](https://github.com/ophelielacroix), [@lhoestq](https://github.com/lhoestq) for adding this dataset.
提供机构：
alexandrainst原始信息汇总
数据集概述
名称: DaNE (Danish Named Entity)
语言: 丹麦语 (da)
许可证: CC-BY-SA-4.0
数据集大小: 1K<n<10K 实例
多语言性: 单语种
任务类型:
- 词性标注 (part-of-speech)
- 依存句法分析 (dependency parsing)
- 命名实体识别 (named-entity-recognition)
数据集结构:
-
特征:
- sent_id: 字符串，句子标识符
- text: 字符串，原始句子
- tok_ids: 整数序列，每个词的标识符
- tokens: 字符串序列，词列表
- lemmas: 字符串序列，词干列表
- pos_tags: 字符串序列，词性标签列表
- morph_tags: 字符串序列，形态标签列表
- dep_ids: 整数序列，依存关系头部的标识符
- dep_labels: 字符串序列，依存关系标签列表
- ner_tags: 字符串序列，命名实体标签列表（BIO格式）
-
数据分割:
- train: 4383 实例
- test: 565 实例
- validation: 564 实例
数据集创建:
- 注释创建者: 专家生成
- 语言创建者: 发现
- 源数据集: 扩展自 Danish-Universal-Dependencies-treebank
数据集详细信息
- 特征详细信息:
- pos_tags:
- 0: NUM
- 1: CCONJ
- 2: PRON
- 3: VERB
- 4: INTJ
- 5: AUX
- 6: ADJ
- 7: PROPN
- 8: PART
- 9: ADV
- 10: PUNCT
- 11: ADP
- 12: NOUN
- 13: X
- 14: DET
- 15: SYM
- 16: SCONJ
- dep_labels:
- 0: parataxis
- 1: mark
- 2: nummod
- 3: discourse
- 4: compound:prt
- 5: reparandum
- 6: vocative
- 7: list
- 8: obj
- 9: dep
- 10: det
- 11: obl:loc
- 12: flat
- 13: iobj
- 14: cop
- 15: expl
- 16: obl
- 17: conj
- 18: nmod
- 19: root
- 20: acl:relcl
- 21: goeswith
- 22: appos
- 23: fixed
- 24: obl:tmod
- 25: xcomp
- 26: advmod
- 27: nmod:poss
- 28: aux
- 29: ccomp
- 30: amod
- 31: cc
- 32: advcl
- 33: nsubj
- 34: punct
- 35: case
- ner_tags:
- 0: O
- 1: B-PER
- 2: I-PER
- 3: B-ORG
- 4: I-ORG
- 5: B-LOC
- 6: I-LOC
- 7: B-MISC
- 8: I-MISC
- pos_tags:
数据集使用注意事项
- 许可证: 使用本数据集需遵守 CC-BY-SA-4.0 许可证。
- 数据敏感性: 数据集可能包含敏感信息，使用时需谨慎处理。
- 数据偏见: 数据集可能存在偏见，使用时应考虑其对模型的影响。
搜集汇总
数据集介绍
构建方式
丹麦语依存树库（DaNE）是基于丹麦语通用依存树库（UD-DDT）构建的命名实体标注资源。UD-DDT源自丹麦语依存树库（Buch-Kromann等人，2003），其文本来源于Parole语料库（Britt，1998），已具备依存句法分析和词性标注信息。Alexandra研究所在此基础上，采用CoNLL-2003标注方案，针对人名（PER）、组织（ORG）和地点（LOC）三类实体进行了人工标注。整个树库经由双重标注以确保质量，最终形成了包含训练集（4383句）、验证集（564句）和测试集（565句）的标准化语料。
特点
DaNE是当前规模最大且公开可用的丹麦语命名实体金标准标注数据集，兼具词性标注、依存句法分析和命名实体识别三重任务能力。其数据字段丰富，涵盖词形、词元、形态标签、依存关系和BIO格式的实体标签，支持细粒度语言分析。数据集以丹麦语为单一语言，采用CC-BY-SA-4.0许可协议发布，确保了学术和商业应用的广泛兼容性。
使用方法
该数据集可通过HuggingFace Datasets库直接加载，适用于训练和评估命名实体识别、词性标注及依存句法分析模型。用户可调用`load_dataset("alexandrainst/dane")`获取预划分的训练、验证和测试集。在模型应用上，研究者可利用其标注信息微调丹麦语BERT或多语言BERT等预训练模型，或进行跨语言迁移学习实验，如结合挪威语数据进行联合训练以提升性能。
背景与挑战
背景概述
在自然语言处理领域，高质量标注语料库是推动句法分析与命名实体识别研究的关键基石。DaNE（Danish Dependency Treebank with Named Entities）数据集由丹麦奥胡斯大学附属的Alexandra Institute于2020年发布，主要研究人员包括Rasmus Hvingelby、Amalie Brogaard Pauli、Maria Barrett等。该数据集基于已有的丹麦语通用依存树库（UD-DDT），采用CoNLL-2003标注方案，为其中约5500个句子添加了人名、地名、组织名等命名实体标签。作为目前最大的公开丹麦语命名实体黄金标注资源，DaNE不仅填补了低资源语言在细粒度语义标注方面的空白，更为跨语言迁移学习与丹麦语自然语言理解研究提供了标准化基准。其影响力体现在：通过对比单语BERT、多语BERT及FLAIR等架构的性能，揭示了预训练语言模型在非英语场景下的适应能力，并验证了从挪威语等亲属语言迁移标注的有效性，从而拓展了低资源语言NER研究的范式。
当前挑战
DaNE所解决的领域问题聚焦于丹麦语命名实体识别这一长期被忽视的任务，其核心挑战包括：低资源环境下标注数据匮乏导致的模型泛化能力不足，以及丹麦语特有的形态复杂性和灵活词序对序列标注精度的干扰。在构建过程中，团队面临多重困难：首先，原始树库来源于Parole语料库，语料年代较早且领域分布不均，需要人工校验文本的当代适用性；其次，采用双重标注策略确保实体边界与类别的一致性，但标注者间分歧（如机构名与地名的模糊界限）要求设计严格的仲裁流程；此外，CoNLL-2003标注体系对丹麦语复合词、缩写等语言现象的适配性不足，需额外制定细粒度规则。最终，数据集仅包含约1万句，训练集规模（4383句）有限，使得模型在长尾实体和未登录词上的表现成为持续挑战。
常用场景
经典使用场景
DaNE（Danish Dependency Treebank with Named Entities）作为丹麦语自然语言处理领域的重要资源，其经典使用场景聚焦于三大核心任务：词性标注、依存句法分析和命名实体识别。该数据集基于丹麦语通用依存树库（UD-DDT），通过CoNLL-2003标注体系，为每个句子提供了包括词元、词性标签、形态特征、依存关系及命名实体标签在内的丰富标注信息。研究者常利用DaNE训练和评估序列标注模型，如基于FLAIR、丹麦语BERT或多语言BERT的架构，在细粒度词级别任务上检验模型对丹麦语语法和语义的理解能力。数据集的划分（训练集4383句、验证集565句、测试集564句）确保了实验的可重复性与基准可比性，使其成为丹麦语自然语言处理研究的标杆性评测平台。
解决学术问题
在学术研究层面，DaNE有效解决了低资源语言丹麦语在结构化信息抽取领域缺乏高质量标注数据的困境。此前，丹麦语的命名实体识别主要依赖跨语言迁移方法，但性能受限于语言间的差异。该数据集通过提供专家标注的黄金标准语料，使研究者能够系统评估监督学习方法在丹麦语上的表现，显著提升了NER的准确性。此外，DaNE支持对未大写文本的鲁棒性分析，揭示了模型在非规范输入下的局限性，为后续研究工作提供了重要方向。其发布推动了丹麦语自然语言处理从依赖跨语言迁移向自主模型训练的转变，为斯堪的纳维亚语言家族的资源建设树立了典范。
衍生相关工作
DaNE的发布催生了一系列衍生研究，推动了丹麦语自然语言处理领域的创新。其中，Hvingelby等人（2020）在LREC上发表的论文首次系统评估了FLAIR、丹麦语BERT和多语言BERT在DaNE上的表现，并探索了从挪威语等相近语言进行跨语言迁移学习的方法，证明了联合微调能提升低资源语言性能。后续工作包括基于DaNE的丹麦语NER模型鲁棒性分析，以及将其与丹麦语Universal Dependencies树库结合进行多任务学习的研究。此外，DaNE还被用作基准数据集，在丹麦语词嵌入评估和结构化预测模型比较中发挥重要作用，促进了针对斯堪的纳维亚语言的预训练语言模型（如丹麦语BERT）的优化与改进。
以上内容由遇见数据集搜集并总结生成
