common_corpus

# Common Corpus

<p align="center">
<a href="https://iclr.cc/virtual/2026/poster/10011885"><b>Full paper - ICLR 2026 oral</b></a>
</p>

Common Corpus is the largest open and permissible licensed text dataset, comprising 2.27 trillion tokens (2,267,302,720,836 tokens). It is a diverse dataset, consisting of books, newspapers, scientific articles, government and legal documents, code, and more. Common Corpus has been created by Pleias in association with several partners.

Common Corpus differs from existing open datasets in that it is:
* **Truly Open**: contains only data that is either uncopyrighted or permissively licensed
* **Traceable**: each individual document is associated with documented contextual information, including licensed use or lack of copyright.
* **Multilingual**: mostly representing English and French data, but contains data for 8 languages with more than 10 billion tokens (German, Spanish, Italian, Polish, Greek, Latin) and 33 languages with more than 1 billion tokens.
* **Diverse**: consisting of scientific articles, government and legal documents, code, and cultural heritage data, including books and newspapers
* **Extensively Curated**: spelling and formatting has been corrected from digitized texts, harmful and toxic content has been removed, and content with low educational content has also been removed.

The dataset in its entirety meets the requirements of the Code of Conduct of the AI Act and goes further than the current requirements for data transparency. It aims to set a new standard of openness in AI, showing that detailed provenance at a granular document level is a realistic objective, even at the scale of 2.3 trillion tokens.

Common Corpus makes it possible to train model compatible with [the Open Source Initiative’s definition](https://opensource.org/ai/open-source-ai-definition#:~:text=An%20Open%20Source%20AI%20is,including%20to%20change%20its%20output.) of open-source AI, which includes openness of use, meaning use is permitted for “any purpose and without having to ask for permission". Based on the available licensing information Common Corpus can be filtered to only include public domain works or a subset of free licenses (like attribution only).

# About Common Corpus

Common Corpus is made of six carefully curated collections:
* **OpenCulture**: our largest collection at 967,018,390,906 tokens, featuring public domain books, newspapers from cultural heritage repositories and open projets like Wikisource ad Gutenberg. We're developing innovative tools of OCR correction based on Pleias Models to correct historical digitization errors, while implementing advanced toxicity filtering to ensure content meets modern ethical standards.
* **OpenGovernment**: 579,150,518,908 tokens of financial and legal documents, including Finance Commons (from sources like SEC and WTO) and Legal Commons (including Europarl, Caselaw Access Project, Chinese Case Law), providing enterprise-grade training data from regulatory bodies and administrative sources.
* **OpenSource**: 283,227,402,898 tokens of high-quality code in open source from GitHub, filtered using ArmoRM to ensure only the top 80% of submissions by quality rating are included.
* **OpenScience**: 281,193,563,789 tokens of academic content from Open Alex and other open science reposiories, processed using vision-language models to preserve crucial document structure and formatting.
* **OpenWeb**: 88,517,032,065 tokens from Wikipedia (official releases from the [Wikimedia Foundation](https://huggingface.co/datasets/wikimedia/wikipedia) on Huggingface), YouTube Commons and Stack-Exchange.
* **Open Semantic**: 67,958,671,827 tokens from Wikidata (official releases from the [Wikimedia Foundation](https://huggingface.co/datasets/wikimedia/wikipedia) on Huggingface). The data has been reprocessed thanks to support and help of Wikidata and Wikimedia Germany. It includes the transcriptions of all the semantic triplets into natural language statements in over 300 languages.

| Collection | Domain | Sources |
|----------------|--------------------------|-------------------------------------------------------------------------------------------|
| OpenGovernment | legal and administrative | [Finance Commons](https://huggingface.co/collections/PleIAs/finance-commons-66925e1095c7fa6e6828e26c) (e.g. SEC, WTO) and Legal Commons (e.g. Europarl, Caselaw Access Project, Chinese CaseLaw) |
| OpenCulture | cultural heritage | public domain books and newspapers, Wikisource |
| OpenScience | academic | OpenAlex |
| OpenWeb | web text | [YouTube Commons](https://huggingface.co/datasets/PleIAs/YouTube-Commons), MOSEL, Stack Exchange, CCCC |
| OpenSource | code | GitHub |
| OpenSemantic | Semantic data | Wikidata |

The first version of [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus) was released in November of 2024. The second version added Wikidata and detailed document-level information, including licensing and other core metadata whenever available. The third ongoing version dramatically expand the language coverage of Common Corpus beyond the US and Europe with the integration of large collection of documents in Chinese, Japanese, Arabic, Korean and Hindi.

The dataset release is accompanied by a comprehensive technical report (ICRL 2026 - oral) detailing our methodologies and data sources will accompany the release, ensuring full transparency and reproducibility.

## Dataset Structure

<details >
<summary>Data Fields</summary>

* `identifier`: unique text identifier. In many cases, this is also the link to the original resources.
* `collection`: name of one of the XX sub-collections curated for Common corpus.
* `open type`: one of the six leading collection groupings:
* `license`: sharing rights for the content either uncopyrighted (public domain, US federal public domain, CC0 on Wikidata) or various free licenses (Creative Commons, MIT, French Licence ouverte, etc.)
* `date`: date of creation of the resource where known. Due to the significance of public domain and other cultural heritage content, more than half of Common Corpus predates the 21st century.
* `title`: title of the resource when known or alternatively the filename.
* `creator`: institution publishing/collecting/curating the resource.
* `language`: automatically identified language.
* `word_count`: number of space delimited words.
* `token_count`: number of tokens as calculated by Pleias official tokenizer and Gemma-3 tokenizer for Chinese, Japanese, Arabic, Korean and few additional non-Western languages.
* `text`: full text, without formatting.

</details >
<br />

## Provenance

The provenance of the datasets that make up Refined Common Corpus is detailed in the technical report [link]. Additionally, the original source URL is available in the metadata for each document for most of the dataset.

## How to Use

### Considerations for Using the Data

All data in Common Corpus are permissibly licensed and may be used for both commercial and non-commercial purposes.

The dataset is multilingual. The language text is included in the metadata, so data can be filtered by language. Additionally, some of the text data are historical. The year each text is written is included in the metadata, therefore it is possible to construct a dataset with a custom date cutoff if desired.

### Discussion of Bias

Some of the dataset sources contain biased and toxic content, such as stereotypes about certain minoritized groups. We have removed texts which had high toxicity scores according to our toxicity classifier, [Celadon](https://huggingface.co/PleIAs/celadon), or which contain offensive terms and slurs. See our [preprint](https://arxiv.org/pdf/2410.22587) for more details.

### Personal and Sensitive Information

We have attempted to remove personally identifiable information (PII). We primarily use [Microsoft Presidio](https://microsoft.github.io/presidio/), but make additional modifications to account for language- and country-specific considerations, such as European phone number formats.

Some small parts of the French administrative common crawl have been entirely dropped using our unreleased small reasoning model for GDPR-filtering, due to the heightened risk of transmitting identifiable indirect personal information.

## Using Common Corpus
from datasets import load_dataset
data = load_dataset('PleIAs/common_corpus')

# Acknowledgements

The Corpus was built up with the support and concerted efforts of the AI Alliance, the French Ministry of Culture as part of the prefiguration of the service offering of the Alliance for Language technologies EDIC (ALT-EDIC).

This dataset was also made in partnership with Wikimedia Enterprise and Wikidata/Wikimedia Germany. We're also thankful to our partner Libraries Without Borders for continuous assistance on extending low resource language support.

The corpus was stored and processed with the generous support of the AI Alliance, Jean Zay (Eviden, Idris), Tracto AI, Mozilla. Generation of OCR correction at scale were performed using HPC resources from two GENCI–IDRIS grants: 2023-AD011014736 and GC011015451.

Some parts of the corpus have been built on top of other similar open science LLM community initiatives such as German-Commons, MOSEL, kl3m, AI4Bharat, Creative Commons Common Crawl. We included a new curator field to properly acknowledge this data work.

<div style="text-align: center;">
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/ai_alliance.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/logo-genci-header.svg" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/Nvidia_(logo).svg.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/tractoAI.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/mozilla.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/wikimedia_logo.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
</div>

# 通用语料库（Common Corpus）

<p align="center">
<a href="https://iclr.cc/virtual/2026/poster/10011885"><b>完整论文 - ICLR 2026 口头报告论文</b></a>
</p>

通用语料库（Common Corpus）是目前规模最大的开放且获得许可授权的文本数据集，总计包含2.27万亿个词元（Token），具体为2,267,302,720,836个词元。该数据集涵盖类型广泛，包含图书、报纸、学术论文、政府与法律文书、代码等多种内容。通用语料库由Pleias联合多家合作方共同构建。

通用语料库与现有开放数据集的区别在于其具备以下核心特性：
* **真正开放**：仅包含无版权或获得宽松许可授权的数据
* **可溯源**：每份独立文档均关联有可查询的上下文信息，包括使用许可或版权状态说明。
* **多语言覆盖**：以英语和法语数据为主，同时包含8种词元（Token）量超100亿的语言数据（德语、西班牙语、意大利语、波兰语、希腊语、拉丁语），以及33种词元（Token）量超10亿的语言数据。
* **类型多元**：涵盖学术论文、政府与法律文书、代码及文化遗产数据（含图书与报纸）
* **精细化清洗**：已对数字化文本的拼写与格式进行校正，移除有害、有毒内容及教育价值较低的文本。

该数据集整体符合《人工智能法案（AI Act）》行为准则的要求，且在数据透明度方面超出了当前的标准要求。其目标是为人工智能领域的开放化树立新标杆，证明即便在2.3万亿词元（Token）的规模下，实现文档粒度的详细溯源也是可行的。

通用语料库可用于训练符合[开放源代码倡议组织（Open Source Initiative）定义](https://opensource.org/ai/open-source-ai-definition#:~:text=An%20Open%20Source%20AI%20is,including%20to%20change%20its%20output.)的开源人工智能模型，该定义包含使用开放性，即允许“任何用途且无需申请许可”。根据现有许可信息，通用语料库可被过滤为仅包含公有领域作品或部分宽松自由许可（如仅署名许可）的子集。

# 关于通用语料库

通用语料库由6个经过精心筛选的子数据集组成：
* **开放文化数据集（OpenCulture）**：规模最大的子数据集，包含967,018,390,906个词元（Token），涵盖公有领域图书、来自文化遗产库的报纸，以及Wikisource、古腾堡计划（Gutenberg）等开放项目的内容。团队正在基于Pleias模型开发创新的光学字符识别（OCR）校正工具，以修正历史数字化过程中的错误，同时采用高级有毒内容过滤机制，确保内容符合现代伦理标准。
* **开放政务数据集（OpenGovernment）**：包含579,150,518,908个词元（Token）的金融与法律文书数据，涵盖金融公共库（Finance Commons，来源包括美国证券交易委员会SEC、世界贸易组织WTO）及法律公共库（Legal Commons，包含欧洲议会平行语料库Europarl、判例法访问项目Caselaw Access Project、中国判例法数据），可提供来自监管机构与行政来源的企业级训练数据。
* **开放源代码数据集（OpenSource）**：包含283,227,402,898个词元（Token）的高质量开源代码数据，来源为GitHub，通过ArmoRM进行过滤，仅保留质量评分前80%的提交内容。
* **开放科学数据集（OpenScience）**：包含281,193,563,789个词元（Token）的学术内容数据，来源为OpenAlex及其他开放科学库，通过多模态视觉语言模型进行处理，以保留关键的文档结构与格式。
* **开放网络数据集（OpenWeb）**：包含88,517,032,065个词元（Token）的数据，来源包括维基百科（Hugging Face平台上[维基媒体基金会（Wikimedia Foundation）](https://huggingface.co/datasets/wikimedia/wikipedia)的官方发布版本）、YouTube公共库及Stack Exchange。
* **开放语义数据集（Open Semantic）**：包含67,958,671,827个词元（Token）的数据，来源为Wikidata（Hugging Face平台上维基媒体基金会的官方发布版本）。在Wikidata与德国维基媒体的支持下，该数据已完成重新处理，将所有语义三元组转换为300余种语言的自然语言语句。

| 子数据集名称 | 领域 | 来源 |
|----------------|--------------------------|-------------------------------------------------------------------------------------------|
| OpenGovernment | 法律与行政 | [金融公共库](https://huggingface.co/collections/PleIAs/finance-commons-66925e1095c7fa6e6828e26c)（例如美国证券交易委员会SEC、世界贸易组织WTO）及法律公共库（例如欧洲议会平行语料库Europarl、判例法访问项目、中国判例法） |
| OpenCulture | 文化遗产 | 公有领域图书与报纸、Wikisource |
| OpenScience | 学术领域 | OpenAlex |
| OpenWeb | 网络文本 | [YouTube公共库](https://huggingface.co/datasets/PleIAs/YouTube-Commons)、MOSEL、Stack Exchange、CCCC |
| OpenSource | 代码领域 | GitHub |
| OpenSemantic | 语义数据 | Wikidata |

[通用语料库（Common Corpus）](https://huggingface.co/datasets/PleIAs/common_corpus)的首个版本于2024年11月发布。第二版新增了Wikidata数据及详细的文档级元数据，包括可用的许可信息与其他核心元数据。目前正在开发的第三版大幅拓展了通用语料库的语言覆盖范围，不再局限于欧美地区，集成了中文、日语、阿拉伯语、韩语及印地语的大量文档数据。

本次数据集发布将同步附带一份详细的技术报告（ICLR 2026 口头报告论文），其中详述了我们的研究方法与数据来源，以确保研究的完全透明性与可复现性。

## 数据集结构

<details >
<summary>数据字段说明</summary>

* `标识符（identifier）`：唯一文本标识符，多数情况下同时指向原始资源的链接。
* `子数据集名称（collection）`：通用语料库所包含的子数据集之一的名称。
* `开放类型（open type）`：六大子数据集分组之一。
* `许可协议（license）`：内容的共享权限，包括无版权（公有领域、美国联邦公有领域、Wikidata上的CC0协议）或各类宽松自由许可（知识共享协议Creative Commons、MIT协议、法国开放许可Licence ouverte等）。
* `创建日期（date）`：资源的创建日期（如可查询）。由于公有领域及其他文化遗产内容占比较高，通用语料库中超过一半的文本创作于21世纪之前。
* `标题（title）`：资源的标题（如可查询），若无则使用文件名。
* `创作者（creator）`：发布、收集或整理该资源的机构。
* `语言（language）`：自动识别的文本语言。
* `词数（word_count）`：以空格分隔的单词数量。
* `词元数（token_count）`：词元（Token）数量，由Pleias官方词元分词器计算；对于中文、日语、阿拉伯语、韩语及其他少数非西方语言，则使用Gemma-3分词器计算。
* `文本（text）`：无格式的完整文本内容。

</details >
<br />

## 溯源信息

构成精炼版通用语料库的各子数据集的溯源信息已在技术报告[链接]中详细说明。此外，多数数据集的每份文档的元数据中均包含原始来源URL。

## 使用指南

### 数据使用注意事项
通用语料库中的所有数据均持有宽松许可授权，可用于商业与非商业用途。

该数据集支持多语言，元数据中包含文本语言信息，因此可按语言对数据进行筛选。此外，部分文本为历史文献，元数据中包含每份文本的创作年份，因此可根据需求自定义时间截断阈值来构建数据集。

### 偏差说明
部分数据集来源包含偏见性与有毒内容，例如针对特定少数群体的刻板印象。团队已根据毒性分类器[Celadon](https://huggingface.co/PleIAs/celadon)的评分，移除了毒性得分较高的文本，以及包含冒犯性词汇与诽谤性内容的文本。详细信息可参阅我们的[预印本论文](https://arxiv.org/pdf/2410.22587)。

### 个人与敏感信息处理
团队已尽力移除个人可识别信息（PII）。我们主要使用[Microsoft Presidio](https://microsoft.github.io/presidio/)工具，并针对不同语言与地区的特性进行了额外调整，例如适配欧洲电话号码格式。

由于存在泄露间接可识别个人信息的较高风险，团队使用未公开的小型推理模型对GDPR合规要求进行过滤，已完全移除了部分法国行政网络爬取数据。

## 通用语料库使用示例
python
from datasets import load_dataset
data = load_dataset('PleIAs/common_corpus')

## 致谢
本语料库的构建得到了人工智能联盟（AI Alliance）的支持，以及法国文化部的协同参与，该合作属于语言技术联盟EDIC（ALT-EDIC）服务方案的预研项目。

本数据集的开发还得到了维基媒体企业项目（Wikimedia Enterprise）与Wikidata/德国维基媒体的合作支持。同时感谢合作方无国界图书馆（Libraries Without Borders）在拓展低资源语言支持方面提供的持续协助。

本语料库的存储与处理工作得到了人工智能联盟、Jean Zay（Eviden、Idris）、Tracto AI、Mozilla的慷慨支持。大规模OCR校正任务的执行使用了两项GENCI-IDRIS项目的高性能计算资源：2023-AD011014736与GC011015451。

本语料库的部分内容基于其他类似的开放科学大语言模型（Large Language Model，LLM）社区项目构建，例如German-Commons、MOSEL、kl3m、AI4Bharat、知识共享公共爬虫（Creative Commons Common Crawl）。我们新增了整理者字段，以明确致谢这些数据工作的贡献者。

<div style="text-align: center;">
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/ai_alliance.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/logo-genci-header.svg" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/Nvidia_(logo).svg.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/tractoAI.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/mozilla.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
<img src="https://huggingface.co/datasets/PleIAs/common_corpus/resolve/main/logo/wikimedia_logo.png" style="width: 33%; margin: 0 auto; display: inline-block;"/>
</div>