imvladikon/paranames

<img src="data/paranames_banner.png"></img>

# ParaNames: A multilingual resource for parallel names

This repository contains releases for the ParaNames corpus, consisting of parallel names of over 12 million named entities in over 400 languages.

ParaNames was introduced in [Sälevä, J. and Lignos, C., 2022. ParaNames: A Massively Multilingual Entity Name Corpus. arXiv preprint arXiv:2202.14035](https://arxiv.org/abs/2202.14035).

Please cite as:
```
@article{saleva2022paranames,
title={ParaNames: A Massively Multilingual Entity Name Corpus},
author={S{\"a}lev{\"a}, Jonne and Lignos, Constantine},
journal={arXiv preprint arXiv:2202.14035},
year={2022}
}
```

See the [Releases page](https://github.com/bltlab/paranames/releases) for the downloadable release.

# Using the data release

## Release format

The corpus is released as a gzipped TSV file which is produced by the pipeline included in this repository.

## Release notes

### Repeated entities

In current releases, any entity that is associated with multiple named entity types (PER, LOC, ORG) in the Wikidata type hierarchy will appear multiple times in the output, once with each type. This affects less than 3% of the entities in the data.

If you want a unique set of entities, you should deduplicate the data using the `wikidata_id` field.

If you only want to use entities that are associated with a single named entity type, you should remove any `wikidata_id` that appears in multiple rows.

# Using the code

First, install the following non-Python dependencies:

- MongoDB
- [xsv](https://github.com/BurntSushi/xsv)
- ICU support for your computer (e.g. `libicu-dev`)

Next, install ParaNames and its Python dependencies by running `pip install -e .`.

It is recommended that you use a Conda environment for package management.

## Creating the ParaNames corpus

To create a corpus following our approach, follow the steps below:

1. Download the latest Wikidata dump from the [Wikimedia page](https://dumps.wikimedia.org/wikidatawiki/entities/) and extract it. Note that this may take up several TB of disk space.
2. Use `recipes/paranames_pipeline.sh` which ingests the Wikidata JSON to MongoDB and then dumps and postprocesses it to our final TSV resource.

The call to `recipes/paranames_pipeline.sh` works as follows:

```
recipes/paranames_pipeline.sh <path_to_extracted_json_dump> <output_folder> <n_workers>
```

Set the number of workers based on the number of CPUs your machine has.
By default, only 1 CPU is used.

The output folder will contain one subfolder per language, inside of which `paranames_<language_code>.tsv` can be found.
The entire resource is located in `<output_folder>/combined/paranames.tsv`.

### Notes

ParaNames offers several options for customization:

- If your MongoDB instance uses a non-standard port, you should change the value of [`mongodb_port`](https://github.com/bltlab/paranames/blob/main/recipes/paranames_pipeline.sh#L13) accordingly inside `paranames_pipeline.sh`.

- Setting [`should_collapse_languages=yes`](https://github.com/bltlab/paranames/blob/main/recipes/dump.sh#L17) will cause Wikimedia language codes to be "collapsed" to the top-level Wikimedia language code, i.e. `kk-cyrl` will be converted to `kk`, `en-ca` to `en` etc.

- Setting [`should_keep_intermediate_files=yes`](https://github.com/bltlab/paranames/blob/main/recipes/dump.sh#L18) will cause intermediate files to be deleted. This includes the raw per-type TSV dumps (`{PER,LOC,ORG}.tsv`) from MongoDB, as well as outputs of `postprocess.py`.

- Within [`recipes/dump.sh`](https://github.com/bltlab/paranames/blob/main/recipes/dump.sh), it is also possible to define languages to be excluded and whether entity types should be disambiguated. By default, no languages are excluded and no disambiguation is done.

- After the pipeline completes, `<output_folder>` will contain one folder per language, inside of which is a TSV file containing the subset of names in that language. Combined TSVs with names in all languages are available in the `combined` folder.

<img src="data/paranames_banner.png"></img>

# ParaNames：面向并行命名实体的多语言语料资源

本仓库托管ParaNames语料库的发布版本，该语料库包含覆盖400余种语言的1200余万个命名实体的并行名称对。

ParaNames 由 Sälevä, J. 与 Lignos, C. 于2022年提出，相关论文为《ParaNames: A Massively Multilingual Entity Name Corpus》，预印本编号arXiv:2202.14035，可访问https://arxiv.org/abs/2202.14035查看。

引用格式如下：

@article{saleva2022paranames,
title={ParaNames: A Massively Multilingual Entity Name Corpus},
author={S{"a}lev{"a}, Jonne and Lignos, Constantine},
journal={arXiv preprint arXiv:2202.14035},
year={2022}
}

可访问[发布页面](https://github.com/bltlab/paranames/releases)获取可下载的语料包。

# 数据发布包使用说明

## 发布包格式
本语料库以gzip压缩的TSV文件格式发布，该文件由本仓库内置的处理流水线生成。

## 发布说明
### 重复实体说明
在当前发布版本中，若某实体在Wikidata类型层级中关联了多种命名实体类型（PER（人物）、LOC（地点）、ORG（组织）），则该实体将在输出中多次出现，每种类型对应一条记录。此类实体占总数据的比例不足3%。

若需获取唯一实体集合，可通过`wikidata_id`字段对数据进行去重。

若仅需使用仅关联单一种命名实体类型的实体，应删除所有在多行中出现的`wikidata_id`。

# 代码使用指南
首先，请安装以下非Python依赖项：
- MongoDB
- [xsv](https://github.com/BurntSushi/xsv)
- 适配您操作系统的ICU库（例如`libicu-dev`）

随后，执行`pip install -e .`命令安装ParaNames及其Python依赖包。

建议使用Conda环境进行包管理。

## 构建ParaNames语料库
若需按照本项目的流程构建语料库，请执行以下步骤：
1. 从[Wikimedia官方页面](https://dumps.wikimedia.org/wikidatawiki/entities/)下载最新的Wikidata数据转储文件并解压。请注意，该文件将占用数TB的磁盘空间。
2. 执行`recipes/paranames_pipeline.sh`脚本，该脚本会将Wikidata的JSON数据导入MongoDB，随后将数据导出并进行后处理，最终生成目标TSV格式的语料资源。

执行`recipes/paranames_pipeline.sh`的命令格式如下：

recipes/paranames_pipeline.sh <path_to_extracted_json_dump> <output_folder> <n_workers>

可根据主机的CPU核心数设置工作线程数，默认仅使用1个CPU核心。

输出文件夹将为每种语言创建一个子文件夹，子文件夹内包含`paranames_<language_code>.tsv`文件。完整的语料资源存放在`<output_folder>/combined/paranames.tsv`路径下。

### 注意事项
ParaNames 提供多项自定义配置选项：
- 若您的MongoDB实例使用非标准端口，请在`paranames_pipeline.sh`中修改[`mongodb_port`](https://github.com/bltlab/paranames/blob/main/recipes/paranames_pipeline.sh#L13)的对应配置值。
- 将[`should_collapse_languages`](https://github.com/bltlab/paranames/blob/main/recipes/dump.sh#L17)设置为`yes`后，Wikimedia语言代码将被折叠至顶级语言代码，例如`kk-cyrl`将被转换为`kk`，`en-ca`将被转换为`en`等。
- 将[`should_keep_intermediate_files`](https://github.com/bltlab/paranames/blob/main/recipes/dump.sh#L18)设置为`yes`将保留中间文件，否则中间文件将被删除。中间文件包括从MongoDB导出的按类型划分的原始TSV转储文件（`{PER,LOC,ORG}.tsv`）以及`postprocess.py`的输出结果。
- 在[`recipes/dump.sh`](https://github.com/bltlab/paranames/blob/main/recipes/dump.sh)中，还可配置需要排除的语言列表以及是否对实体类型进行消歧。默认情况下，不排除任何语言，也不执行实体类型消歧。
- 流水线执行完成后，`<output_folder>`将为每种语言生成一个子文件夹，文件夹内包含对应语言的名称子集TSV文件。包含所有语言名称的合并TSV文件存放在`combined`文件夹中。

原始信息汇总

数据集概述

数据集名称

ParaNames: A multilingual resource for parallel names

数据集描述

包含超过1200万命名实体的平行名称，涵盖超过400种语言。

数据集来源

介绍于Sälevä, J. and Lignos, C., 2022. ParaNames: A Massively Multilingual Entity Name Corpus. arXiv preprint arXiv:2202.14035。

引用信息

@article{saleva2022paranames, title={ParaNames: A Massively Multilingual Entity Name Corpus}, author={S{"a}lev{"a}, Jonne and Lignos, Constantine}, journal={arXiv preprint arXiv:2202.14035}, year={2022} }

数据集格式

以gzipped TSV文件形式发布，由本仓库中的管道生成。

数据集使用注意事项

• 重复实体：当前版本中，任何与多个命名实体类型（PER, LOC, ORG）关联的实体将在输出中多次出现，每次对应一个类型。这影响不到3%的数据集实体。
• 去重：若需唯一实体集，应使用wikidata_id字段进行数据去重。
• 单类型实体：若仅需使用与单一命名实体类型关联的实体，应移除出现在多行的wikidata_id。

数据集创建指南

1. 从Wikimedia page下载最新的Wikidata转储并提取。
2. 使用recipes/paranames_pipeline.sh脚本处理数据，该脚本将Wikidata JSON导入MongoDB，然后转储并后处理为最终的TSV资源。

数据集定制选项

• MongoDB端口自定义。
• 语言代码折叠选项。
• 保留或删除中间文件。
• 定义排除语言和实体类型是否应进行消歧。