Nos_CorpusNOS-GL: Galician Macrocorpus for LLM training

Update notice

(EN)

An updated version of CorpusNÓS is available on Hugging Face at: https://huggingface.co/datasets/proxectonos/corpusnos

Future updates, corrections, and extended releases of the corpus will be primarily made available through the Hugging Face repository. This Zenodo record is kept for archival and citation purposes, but users are encouraged to consult the Hugging Face version for the most recent release.

(GL)

Unha versión actualizada de CorpusNÓS está dispoñible en Hugging Face: https://huggingface.co/datasets/proxectonos/corpusnos

As futuras actualizacións, correccións e versións ampliadas do corpus publicaranse principalmente a través do repositorio de Hugging Face. Este rexistro de Zenodo mantense con fins de arquivo e citación, mais recoméndase ás persoas usuarias consultar a versión de Hugging Face para acceder á publicación máis recente.

-----------------

CorpusNÓS is a massive Galician corpus made up of 2.1B words primarily devised for training large language models. The corpus sources are varied and represent a relatively wide range of genres. 

------------------

We happily announce that we are introducing a new version of the CorpusNÓS. After improving our text cleaning and processing methods in our cleaning pipeline, we have decided to release this new version of the corpus, which reflects those enhancements. 

This new version contains the same files as the previous one and holds the same distribution of the data, however, we decided to change the format from plain text (*.txt) to JSONL (*.jsonl) so future cleaning processes can be performed easily, and relevant metadata can be included. As of now, some examples of entries from the CorpusNós have the following structure: 

{"id": 0, "text": "Abades: Parroquia do concello de Baltar baixo a advocación de san Paio.", "num_words": 12} 

{"id": 581, "text": "Feliz 2008 a tódolos nosos lectores\nAgora que remata 2007, un ano cheo de novidades tecnolóxicas que difundimos a través deste espazo dixital, queremos desexar a tódolos que non seguen con fidelidade unha boa despedida do ano e un feliz aninovo.\nNós volveremos o mércores, 2 de xaneiro, á nosa actividade ordinaria, cumprindo coa nosa labor informativa para que as novas tecnolóxicas de Galicia e en galego cheguen ós nosos lectores puntualmente.", "num_words": 72, "pyplexity_score": 717.7585757844212, "lang": "gl"} 

In the plain text version, the delimiter between different documents was constituted by two newlines (\n\n). In the JSONL version, each document is a JSON object with their corresponding id, but it also includes the number of words of each document, and, in some cases, the pyplexity score and the language tag. 

This new version of CorpusNós has undergone a heavier process of deduplication than the previous one. This means that more exact match duplications as well as partial duplications have been removed from the corpus and, therefore, the number of documents and tokens in this version has decreased and the current statistics are: 

Subcorpus: 

Data obtained via transfer agreement 

Genre 

Nº tokens 

Nº documents 

Books 

7.217.626 

103 

Research articles 

2.638.537 

635 

Press 

92.661.350 

161.760 

Governmental 

221.565.059 

527.699 

Web contents 

15.471.132 

41.276 

Encyclopedic 

4.799.214 

47.396 

Subtotal 

332.721.231 

777.583 

Subcorpus: 

Public data 

Genre 

Nº tokens 

Nº documents 

Press and blogs 

142.238.181 

598.375 

Encyclopedic 

48.260.708 

148.560 

Web crawls 

1.205.699.835 

2.850.604 

Translation corpora 

106.555.883 

3.544.026 

Subtotal 

1.502.754.607 

7.141.565 

Total 

1.835.475.838 

7.919.148 

The TXT version is still available under the corpusnos_v1_txt zip file and it mantains the same structure as before (documents are divided by two newlines '\n\n') but this version hasn't gone through the improved cleaning process mentioned above. 

Please, note that if you want to download or use the newest version you have to download the corpusnos_v2_jsonl.

Note: Some of the files referenced may be missing in this version of the corpus due to pending transfer agreements and they will be included in a future version of the corpus as soon as they are available for publishing.  Note: Please, note that the following subcorpora have different licenses which correspond to their original licenses as specified in the paper: TED2020 (CC BY–NC–ND 4.0), mC4 (Apache License 2.0), OSCAR (CC0).  

Please refer to our paper for more details, CorpusNÓS: A massive Galician corpus for training large language models. 

If you use this data in your work, please cite: 

de-Dios-Flores, Iria, Silvia Paniagua Suárez, Cristina Carbajal Pérez, Daniel Bardanca Outeiriño, Marcos Garcia and Pablo Gamallo. 2024. CorpusNÓS: A massive Galician corpus for training large language models. Proceedings of the 16th International Conference on Computational Processing of Portuguese - ACL Anthology (Volume 1), 593-599. 

Funding

This corpus was compiled/development.... within the Nós Project, funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the [project ILENIA] (https://proyectoilenia.es/) with reference 2022/TL22/00215336.

更新通知

当前CorpusNÓS语料库的更新版本已上传至Hugging Face平台：https://huggingface.co/datasets/proxectonos/corpusnos。本语料库的后续更新、修正与扩展版本将主要通过Hugging Face仓库发布。本Zenodo存档仅用于留存与引用，建议用户优先查阅Hugging Face平台上的最新版本。

CorpusNÓS是一款超大规模加利西亚语语料库，总词量达21亿，主要用于训练大语言模型（Large Language Model）。该语料库的数据源多元，涵盖了相对广泛的文本体裁。

我们很高兴地宣布推出CorpusNÓS语料库的新版本。在优化了文本清洗与处理流程后，我们正式发布此次更新版本，以体现这些改进成果。

新版本与旧版本包含相同的文件与数据分布，但我们将文件格式从纯文本（*.txt）调整为JSONL（*.jsonl）格式，以便于后续开展清洗工作，并支持添加相关元数据。目前，CorpusNÓS语料库的条目示例结构如下：

{"id": 0, "text": "Abades: Parroquia do concello de Baltar baixo a advocación de san Paio.", "num_words": 12}

{"id": 581, "text": "Feliz 2008 a tódolos nosos lectores
Agora que remata 2007, un ano cheo de novidades tecnolóxicas que difundimos a través deste espazo dixital, queremos desexar a tódolos que non seguen con fidelidade unha boa despedida do ano e un feliz aninovo.
Nós volveremos o mércores, 2 de xaneiro, á nosa actividade ordinaria, cumprindo coa nosa labor informativa para que as novas tecnolóxicas de Galicia e en galego cheguen ós nosos lectores puntualmente.", "num_words": 72, "pyplexity_score": 717.7585757844212, "lang": "gl"}

旧版纯文本格式中，不同文档之间的分隔符为两个换行符（

）。新版JSONL格式中，每份文档为一个JSON对象，包含对应的id，同时新增了文档词数统计，部分条目还包含困惑度评分（pyplexity_score）与语言标签。

新版本的CorpusNÓS语料库相比旧版执行了更严格的去重流程，移除了精确匹配重复与部分重复内容，因此语料库的文档数与Token数均有所减少。当前版本的统计数据如下：

#### 子语料库：通过传输协议获取的数据
| 文本体裁 | Token数量 | 文档数量 |
|----------------|-----------------|------------|
| 图书 | 7,217,626 | 103 |
| 研究论文 | 2,638,537 | 635 |
| 新闻媒体 | 92,661,350 | 161,760 |
| 政府文献 | 221,565,059 | 527,699 |
| 网页内容 | 15,471,132 | 41,276 |
| 百科类内容 | 4,799,214 | 47,396 |
| **小计** | **332,721,231** | **777,583** |

#### 子语料库：公开数据
| 文本体裁 | Token数量 | 文档数量 |
|--------------------|------------------|---------------|
| 新闻与博客 | 142,238,181 | 598,375 |
| 百科类内容 | 48,260,708 | 148,560 |
| 网页爬取数据 | 1,205,699,835 | 2,850,604 |
| 翻译语料库 | 106,555,883 | 3,544,026 |
| **小计** | **1,502,754,607** | **7,141,565** |

**总计**：Token总量1,835,475,838，文档总数7,919,148。

纯文本格式的旧版语料库仍以corpusnos_v1_txt压缩包形式提供，其文档分隔规则与旧版一致（以两个换行符`

`分隔文档），但未经过上述优化后的清洗流程。

请注意，若需下载或使用最新版本，请获取corpusnos_v2_jsonl压缩包。

备注1：由于部分传输协议尚未完成，本版本语料库中部分引用文件暂未包含，将在后续版本中随可用情况补充发布。
备注2：请注意，以下子语料库遵循其原始授权协议，详情请参阅相关论文：TED2020（CC BY–NC–ND 4.0）、mC4（Apache License 2.0）、OSCAR（CC0）。

如需了解更多细节，请参阅我们的论文《CorpusNÓS: A massive Galician corpus for training large language models》。

若您在研究中使用本语料库，请引用如下文献：
de-Dios-Flores, Iria, Silvia Paniagua Suárez, Cristina Carbajal Pérez, Daniel Bardanca Outeiriño, Marcos Garcia and Pablo Gamallo. 2024. CorpusNÓS: A massive Galician corpus for training large language models. Proceedings of the 16th International Conference on Computational Processing of Portuguese - ACL Anthology (Volume 1), 593-599.

### 资助说明
本语料库的编译与开发依托Nós项目，该项目由西班牙数字化转型与公共职能部资助，并依托欧盟下一代欧盟（NextGenerationEU）框架下的ILENIA项目（https://proyectoilenia.es/）实施，项目编号为2022/TL22/00215336。