bltlab/ParaNames

---
dataset_info:
features:
- name: wikidata_id
dtype: string
- name: eng
dtype: string
- name: label
dtype: string
- name: language
dtype: string
- name: type
dtype: string
splits:
- name: train
num_bytes: 9154111537
num_examples: 140140300
download_size: 1554666017
dataset_size: 9154111537
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
license: mit
task_categories:
- token-classification
tags:
- wikipedia
pretty_name: ParaNames
size_categories:
- 1M<n<10M
language:
- nia
- avk
- bjn
- anp
- si
- bcc
- sma
- syl
- nso
- lez
- qug
- ks
- rmc
- khw
- szy
- kk
- uz
- sro
- tg
- jbo
- gld
- cu
- nb
- gom
- zgh
- iu
- th
- lv
- eo
- aln
- te
- nrm
- udm
- bcl
- ku
- rup
- ga
- es
- bm
- crh
- na
- yi
- mh
- grc
- lmo
- ff
- fi
- ty
- ay
- pi
- ii
- kea
- zh
- ce
- km
- vep
- ch
- br
- rw
- prg
- rue
- bxr
- hu
- kcg
- be
- tly
- sr
- dag
- rm
- ve
- nah
- hr
- dv
- mrj
- ko
- as
- ext
- liv
- ho
- fa
- pam
- en
- mnw
- abs
- mnc
- fy
- sq
- xal
- ia
- nv
- mdf
- pms
- av
- vec
- koi
- sco
- nl
- pfl
- loz
- xmf
- ng
- scn
- war
- ka
- da
- el
- nn
- cs
- li
- pdc
- no
- ki
- so
- krl
- rn
- nds
- lbe
- it
- gu
- mai
- mni
- sh
- frr
- uk
- qu
- am
- io
- ms
- sg
- mos
- pap
- ksh
- gn
- gor
- fon
- mus
- pnb
- nan
- sat
- kbp
- niu
- co
- fur
- glk
- lo
- om
- shi
- haw
- tw
- ab
- ro
- ota
- os
- ur
- kl
- cho
- hi
- jut
- ik
- ky
- sli
- sn
- lij
- ht
- pt
- wal
- ba
- gd
- ban
- tum
- fj
- arz
- dty
- ta
- my
- srq
- ts
- lb
- sl
- tok
- pa
- dsb
- aa
- kg
- lt
- shn
- rmy
- csb
- tyv
- bho
- sa
- cdo
- kn
- pl
- tet
- gag
- bqi
- got
- bg
- bs
- kus
- rif
- ml
- mn
- inh
- pcm
- smj
- sm
- mt
- ang
- lld
- ti
- arq
- cnh
- srn
- cr
- sc
- mg
- ak
- fr
- tt
- mhr
- kbd
- pag
- sk
- gv
- ee
- ss
- eu
- vi
- de
- bn
- ig
- mzn
- sw
- ny
- oc
- kw
- mk
- lad
- la
- ln
- st
- smn
- bug
- mo
- als
- gl
- ca
- lfn
- su
- tl
- kiu
- ne
- lus
- zu
- lg
- is
- hyw
- brh
- szl
- za
- agq
- to
- he
- vot
- sdc
- gsw
- vmw
- mwl
- bgn
- mi
- or
- ckb
- nqo
- arn
- chy
- dz
- awa
- chr
- stq
- id
- kr
- et
- ltg
- dtp
- hil
- sei
- skr
- sms
- myv
- yo
- ru
- ceb
- kaa
- gan
- mad
- sah
- sd
- frp
- ami
- bar
- bew
- ady
- bbc
- bo
- wa
- pih
- wuu
- tr
- bpy
- nap
- kab
- lrc
- mr
- rgn
- an
- krc
- min
- hak
- diq
- new
- ksw
- vmf
- wo
- yue
- ja
- sv
- zea
- jv
- ast
- ar
- ug
- fit
- ses
- tn
- eml
- mrh
- dga
- ilo
- arc
- fat
- az
- ie
- nov
- din
- kv
- krj
- ary
- hsb
- cv
- bi
- hz
- hy
- ase
- hsn
- atj
- vo
- jam
- lki
- cy
- ha
- sty
- hif
- ps
- ace
- tk
- gaa
- se
- alt
- af
- tpi
- xh
- lzh
- kri
- vro
- fo
---

# Dataset Card for Dataset Name

<!-- Provide a quick summary of the dataset. -->

This dataset card aims to be a base template for new datasets. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md?plain=1).

## Dataset Details

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->

- **Curated by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]

### Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the dataset is intended to be used. -->

### Direct Use

<!-- This section describes suitable use cases for the dataset. -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

[More Information Needed]

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

[More Information Needed]

## Dataset Creation

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->

[More Information Needed]

### Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

#### Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

[More Information Needed]

#### Who are the source data producers?

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

[More Information Needed]

### Annotations [optional]

<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->

#### Annotation process

<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->

[More Information Needed]

#### Who are the annotators?

<!-- This section describes the people or systems who created the annotations. -->

[More Information Needed]

#### Personal and Sensitive Information

<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

## Citation

```bibtex
@misc{sälevä2024paranames,
title={ParaNames 1.0: Creating an Entity Name Corpus for 400+ Languages using Wikidata},
author={Jonne Sälevä and Constantine Lignos},
year={2024},
eprint={2405.09496},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

原始信息汇总

数据集概述

数据集信息

特征

• wikidata_id: 字符串类型
• eng: 字符串类型
• label: 字符串类型
• language: 字符串类型
• type: 字符串类型

数据分割

• train:
  • 字节数: 9154111537
  • 样本数: 140140300

数据大小

• 下载大小: 1554666017
• 数据集大小: 9154111537

配置

• config_name: default
  • data_files:
    • split: train
    • path: data/train-*

许可证

• license: MIT

任务类别

• task_categories:
  • token-classification

标签

• tags:
  • wikipedia

数据集名称

• pretty_name: ParaNames

数据集大小类别

• size_categories:
  • 1M<n<10M

语言

• language:
  • nia, avk, bjn, anp, si, bcc, sma, syl, nso, lez, qug, ks, rmc, khw, szy, kk, uz, sro, tg, jbo, gld, cu, nb, gom, zgh, iu, th, lv, eo, aln, te, nrm, udm, bcl, ku, rup, ga, es, bm, crh, na, yi, mh, grc, lmo, ff, fi, ty, ay, pi, ii, kea, zh, ce, km, vep, ch, br, rw, prg, rue, bxr, hu, kcg, be, tly, sr, dag, rm, ve, nah, hr, dv, mrj, ko, as, ext, liv, ho, fa, pam, en, mnw, abs, mnc, fy, sq, xal, ia, nv, mdf, pms, av, vec, koi, sco, nl, pfl, loz, xmf, ng, scn, war, ka, da, el, nn, cs, li, pdc, no, ki, so, krl, rn, nds, lbe, it, gu, mai, mni, sh, frr, uk, qu, am, io, ms, sg, mos, pap, ksh, gn, gor, fon, mus, pnb, nan, sat, kbp, niu, co, fur, glk, lo, om, shi, haw, tw, ab, ro, ota, os, ur, kl, cho, hi, jut, ik, ky, sli, sn, lij, ht, pt, wal, ba, gd, ban, tum, fj, arz, dty, ta, my, srq, ts, lb, sl, tok, pa, dsb, aa, kg, lt, shn, rmy, csb, tyv, bho, sa, cdo, kn, pl, tet, gag, bqi, got, bg, bs, kus, rif, ml, mn, inh, pcm, smj, sm, mt, ang, lld, ti, arq, cnh, srn, cr, sc, mg, ak, fr, tt, mhr, kbd, pag, sk, gv, ee, ss, eu, vi, de, bn, ig, mzn, sw, ny, oc, kw, mk, lad, la, ln, st, smn, bug, mo, als, gl, ca, lfn, su, tl, kiu, ne, lus, zu, lg, is, hyw, brh, szl, za, agq, to, he, vot, sdc, gsw, vmw, mwl, bgn, mi, or, ckb, nqo, arn, chy, dz, awa, chr, stq, id, kr, et, ltg, dtp, hil, sei, skr, sms, myv, yo, ru, ceb, kaa, gan, mad, sah, sd, frp, ami, bar, bew, ady, bbc, bo, wa, pih, wuu, tr, bpy, nap, kab, lrc, mr, rgn, an, krc, min, hak, diq, new, ksw, vmf, wo, yue, ja, sv, zea, jv, ast, ar, ug, fit, ses, tn, eml, mrh, dga, ilo, arc, fat, az, ie, nov, din, kv, krj, ary, hsb, cv, bi, hz, hy, ase, hsn, atj, vo, jam, lki, cy, ha, sty, hif, ps, ace, tk, gaa, se, alt, af, tpi, xh, lzh, kri, vro, fo

引用

bibtex @misc{sälevä2024paranames, title={ParaNames 1.0: Creating an Entity Name Corpus for 400+ Languages using Wikidata}, author={Jonne Sälevä and Constantine Lignos}, year={2024}, eprint={2405.09496}, archivePrefix={arXiv}, primaryClass={cs.CL} }

搜集汇总

数据集介绍

构建方式

ParaNames数据集由bltlab团队构建，旨在通过Wikidata知识库为超过400种语言提供实体名称资源。其构建过程充分利用了Wikidata中结构化实体的多语言标签信息，系统性地提取了每个实体在不同语言下的名称表述。具体而言，数据集以Wikidata标识符（wikidata_id）为锚点，关联对应的英文名称（eng）及各语言的本地化标签（label），同时标注语言代码（language）与实体类型（type），从而形成跨语言对齐的平行名称对。该流程通过自动化脚本完成大规模数据采集与清洗，最终生成了包含约1.4亿个样本的单一训练分割（train），覆盖从尼亚语到祖鲁语等数百种语言，为多语言自然语言处理任务奠定了宽覆盖、高密度的数据基础。

特点

ParaNames数据集的核心特点在于其极致的语言多样性与规模。它涵盖了超过400种语言，包括大量低资源语言（如阿瓦尔语、科米语等），显著拓展了实体名称资源的语言边界。每个样本均包含Wikidata ID、英文名称、本地化标签、语言代码及实体类型五维结构，支持细粒度的跨语言实体链接与名称对齐研究。数据集以单一训练集形式发布，无预设验证或测试分割，便于用户根据任务自定义划分。此外，其采用MIT开源协议发布，降低了使用门槛，而基于Wikidata的构建来源确保了数据的权威性与持续更新潜力，为多语言命名实体识别、机器翻译及知识图谱补全等任务提供了高质量、可扩展的语料支撑。

使用方法

ParaNames数据集适用于多种自然语言处理场景。用户可直接加载HuggingFace数据集库中的'train'分割，利用'wikidata_id'字段进行跨语言实体对齐，或通过'language'与'label'字段构建多语言名称词典。在命名实体识别任务中，可结合'type'标签（如人物、地点）进行监督学习；在机器翻译领域，'eng'与'label'的对应关系可作为平行语料微调模型。建议将数据集按语言或实体类型过滤后使用，例如筛选特定低资源语言子集以缓解数据稀疏问题。由于数据规模庞大（约9GB），推荐使用流式加载（streaming）模式以优化内存管理。研究人员亦可基于其结构化格式，结合外部知识库进行实体链接或跨语言信息检索的实验设计。

背景与挑战

背景概述

在自然语言处理领域，跨语言实体识别与链接任务长期受限于多语言命名实体资源的匮乏，尤其是对低资源语言的覆盖严重不足。为应对这一瓶颈，Jonne Sälevä与Constantine Lignos于2024年发布了ParaNames数据集，该数据集依托Wikidata的海量结构化知识，系统性地构建了覆盖400余种语言的实体名称语料库。其核心研究问题在于如何利用统一的知识图谱自动生成高质量、多语言对齐的实体名称对，从而为跨语言信息抽取、机器翻译及知识库补全等下游任务提供基础支撑。ParaNames的问世显著拓展了多语言NLP的研究边界，尤其为低资源语言研究提供了关键数据基础，对推动语言技术普惠化具有里程碑意义。

当前挑战

ParaNames数据集面临的核心挑战在于多语言覆盖的均衡性与实体名称的语义保真度。首先，尽管数据集囊括400余种语言，但不同语言间实体名称的丰富度差异悬殊，高资源语言（如英语、中文）与低资源语言（如阿留申语、米克马克语）之间的数据分布严重失衡，可能引发模型在低资源场景下的性能退化。其次，构建过程中依赖Wikidata的自动映射机制，难以完全规避实体名称的歧义问题，例如同一实体在不同语言中可能对应多个异义名称，或存在拼写变体与转写误差。此外，跨语言实体对齐的准确性受限于Wikidata自身的数据质量，部分低频实体的名称缺失或错误标注可能引入噪声，进而影响下游任务的有效性。

常用场景

经典使用场景

ParaNames数据集在自然语言处理领域，尤其是多语言知识图谱构建与跨语言实体链接任务中，扮演着基础性资源角色。该数据集基于维基数据（Wikidata）构建，覆盖400余种语言，提供了海量实体的多语言名称对，为研究多语言命名实体识别（NER）、实体规范化及跨语言信息检索提供了标准化训练与评测基准。研究者常将其用于训练统一的多语言实体表示模型，或作为评估跨语言知识迁移能力的黄金标准语料。

解决学术问题

该数据集有效解决了多语言环境下实体名称异质性带来的学术挑战。传统资源往往局限于高资源语言，而ParaNames通过系统化整合维基数据中的多语言标签，突破了低资源语言实体名称稀疏的瓶颈，为低资源语言的自然语言处理研究提供了关键数据支撑。其意义在于推动了多语言实体对齐、零样本跨语言迁移学习等前沿方向的进展，显著降低了多语言知识工程中数据获取的成本与复杂度。

衍生相关工作

ParaNames衍生了一系列重要工作，包括基于其构建的多语言实体嵌入模型（如MultiBERT）、跨语言实体链接系统（如MEL）以及低资源语言NER增强方法。研究者还将其与维基百科多语言语料结合，提出了实体名称的语义相似度计算框架，并催生了面向400+语言的名称变体归一化任务（Name Variant Normalization）的公开评测基准，进一步推动了多语言NLP领域的标准化评估体系发展。