bertin-project/mc4-es-sampled

---
annotations_creators:
- no-annotation
language_creators:
- found
language:
- es
license:
- odc-by
size_categories:
- n<1K
- 1K<n<10K
- 10K<n<100K
- 100K<n<1M
- 1M<n<10M
- 10M<n<100M
- 100M<n<1B
source_datasets:
- mc4
- bertin-project/mc4-sampling
task_categories:
- text-generation
- fill-mask
task_ids:
- language-modeling
pretty_name: mC4-es-sampled
---

# Dataset Card for mC4-es-sampled

## Table of Contents

- [Dataset Card for mC4-es-sampled](#dataset-card-for-mc4-es-sampled)
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://huggingface.co/datasets/allenai/c4
- **Paper:** https://arxiv.org/abs/1910.10683

### Dataset Summary

This dataset is the result of applying perplexity sampling to the Spanish portion of mC4 using [`mc4-sampling`](https://huggingface.co/datasets/bertin-project/mc4-sampling/). Please, refer to [BERTIN Project](https://huggingface.co/bertin-project/bertin-roberta-base-spanish).

You can load the mC4 Spanish sampled like this:

```python
from datasets import load_dataset

for config in ("random", "stepwise", "gaussian"):
mc4es = load_dataset(
"bertin-project/mc4-es-sampled",
config,
split="train",
streaming=True
).shuffle(buffer_size=1000)
for sample in mc4es:
print(config, sample)
break
```
Alternatively, you can bypass the `datasets` library and quickly download (\~1.5hrs, depending on connection) a specific config in the same order used to pre-train BERTIN models in a massive (\~200GB) JSON-lines files:

```python
import io
import gzip
import json
import sys

import requests
from tqdm import tqdm

_DATA_URL_TRAIN = "https://huggingface.co/datasets/bertin-project/mc4-es-sampled/resolve/main/mc4-es-train-50M-{config}-shard-{index:04d}-of-{n_shards:04d}.json.gz"

def main(config="stepwise"):
data_urls = [
_DATA_URL_TRAIN.format(
config=config,
index=index + 1,
n_shards=1024,
)
for index in range(1024)
]
with open(f"mc4-es-train-50M-{config}.jsonl", "w") as f:
for dara_url in tqdm(data_urls):
response = requests.get(dara_url)
bio = io.BytesIO(response.content)
with gzip.open(bio, "rt", encoding="utf8") as g:
for line in g:
json_line = json.loads(line.strip())
f.write(json.dumps(json_line) + "\
")

if __name__ == "__main__":
main(sys.argv[1])
```

### Supported Tasks and Leaderboards

mC4-es-sampled is mainly intended for reproducibility purposes of the BERTIN Project and to pretrain language models and word representations on medium budgets.

### Languages

The dataset only supports the Spanish language.

## Dataset Structure

### Data Instances

An example form the `Gaussian` config:

```python
{'timestamp': '2018-10-20T06:20:53Z', 'text': 'Ortho HyaluroTop 200 aporta el colágeno y ácido hialurónico que, con la edad, se producen en menor cantidad. La vitamina C promueve la producción de colágeno para mantener la piel sana y protege a las células contra los radicales libres causados ??por la contaminación ambiental y los rayos UV.', 'url': 'https://www.farmaciagaleno.com/orthonat-hyalurotop-200-30-capsulas'}
```

### Data Fields

The data have several fields:

- `url`: url of the source as a string
- `text`: text content as a string
- `timestamp`: timestamp as a string

### Data Splits

The resulting mC4 subsets for Spanish are reported in this table:

| config | train |
|:---------|:--------|
| stepwise | 50M |
| random | 50M |
| gaussian | 50M |

The split `validation` is exactly the same as the original `mc4` dataset.

## Dataset Creation

### Curation Rationale

This dataset was built from the original [`mc4`](https://huggingface.co/datasets/mc4) by applying perplexity-sampling via [`mc4-sampling`](https://huggingface.co/datasets/bertin-project/mc4-sampling) for Spanish.

## Additional Information

### Dataset Curators

Original data by [Common Crawl](https://commoncrawl.org/).

### Licensing Information

AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset.

### Citation Information

To cite this dataset ([arXiv](https://arxiv.org/abs/2207.06814)):
```bibtex
@article{BERTIN,
author = {Javier De la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury},
title = {{BERTIN}: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling},
journal = {Procesamiento del Lenguaje Natural},
volume = {68},
number = {0},
year = {2022},
keywords = {},
abstract = {The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pretraining sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name perplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget.},
issn = {1989-7553},
url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6403},
pages = {13--23}
}
```

If you use this dataset, we would love to hear about it! Reach out on twitter, GitHub, Discord, or shoot us an email.

To cite the original `mc4` dataset:
```
@article{2019t5,
author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
journal = {arXiv e-prints},
year = {2019},
archivePrefix = {arXiv},
eprint = {1910.10683},
}
```

### Contributions

Dataset contributed by [@versae](https://github.com/versae) for BERTIN Project.

Thanks to [@dirkgr](https://github.com/dirkgr) and [@lhoestq](https://github.com/lhoestq) for adding the original mC4 dataset.

原始信息汇总

数据集概述

数据集名称

• 名称: mC4-es-sampled
• 别名: mC4 Spanish sampled

数据集属性

• 语言: 仅支持西班牙语（es）
• 许可证: ODC-BY
• 数据大小: 分为多个类别，从小于1K到小于1B不等

数据来源

• 源数据集:
  • mc4
  • bertin-project/mc4-sampling

任务类型

• 任务类别:
  • text-generation
  • fill-mask
• 任务ID: language-modeling

数据集结构

• 数据实例示例: python {timestamp: 2018-10-20T06:20:53Z, text: Ortho HyaluroTop 200 aporta el colágeno y ácido hialurónico que, con la edad, se producen en menor cantidad. La vitamina C promueve la producción de colágeno para mantener la piel sana y protege a las células contra los radicales libres causados ??por la contaminación ambiental y los rayos UV., url: https://www.farmaciagaleno.com/orthonat-hyalurotop-200-30-capsulas}

• 数据字段:

  • url: 源网址，字符串类型
  • text: 文本内容，字符串类型
  • timestamp: 时间戳，字符串类型

数据分割

• 分割详情:

  config	train
  stepwise	50M
  random	50M
  gaussian	50M

数据集创建

• 创建理由: 从原始的mc4数据集通过应用perplexity采样方法构建，旨在提高西班牙语语言模型预训练的效率和质量。

许可证信息

• 许可证详情: 本数据集根据ODC-BY许可证发布，使用时需遵守Common Crawl的使用条款。

引用信息

• 引用格式: bibtex @article{BERTIN, author = {Javier De la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury}, title = {{BERTIN}: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling}, journal = {Procesamiento del Lenguaje Natural}, volume = {68}, number = {0}, year = {2022}, keywords = {}, abstract = {The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pretraining sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name perplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget.}, issn = {1989-7553}, url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6403}, pages = {13--23} }

贡献者

• 主要贡献者: @versae for BERTIN Project.
• 原始数据集贡献者: @dirkgr and @lhoestq for adding the original mC4 dataset.

搜集汇总

数据集介绍

构建方式

在自然语言处理领域，大规模语料库的质量直接决定了预训练语言模型的性能。mC4-es-sampled数据集正是基于这一认知，从Common Crawl爬取的西班牙语文本中精心构建而成。该数据集源自原始的mC4西班牙语子集，通过引入基于困惑度（perplexity）的采样技术，利用bertin-project/mc4-sampling工具对文本进行筛选与重采样。具体而言，研究团队设计了三种采样配置：随机采样（random）、逐步采样（stepwise）和高斯采样（gaussian），每种配置均生成包含5000万条样本的训练集，从而在保留语料多样性的同时，显著提升了数据的质量密度。

特点

mC4-es-sampled数据集的核心特点在于其独特的数据筛选机制与高效的数据利用方式。通过困惑度采样，该数据集能够自动识别并优先保留那些对语言模型训练更具价值的文本片段，有效过滤了低质量或噪声内容。每个数据实例均包含文本内容、来源URL及时间戳三个字段，结构简洁而信息完整。数据集提供了三种不同采样策略的配置选项，为研究者提供了灵活的对比实验基础。值得注意的是，该数据集在保持与原始mC4验证集一致的前提下，将训练集规模精简至5000万条，却能达到甚至超越使用完整数据集训练的语言模型性能，充分体现了数据质量胜于数量的设计理念。

使用方法

研究者可通过HuggingFace的datasets库便捷地加载mC4-es-sampled数据集，支持随机、逐步和高斯三种配置的流式读取，并允许设置缓存大小以优化内存使用。对于需要原始JSON格式数据的用户，数据集还提供了通过HTTP请求直接下载压缩分片的方式，每个配置包含1024个分片文件，总数据量约200GB，下载完成后可合并为单行JSON文件。该数据集主要面向西班牙语语言模型的预训练任务，特别适用于中等计算资源条件下的模型训练，研究者可根据实验需求选择合适的采样配置，通过对比不同配置下的模型性能，深入探究数据采样策略对语言模型学习效果的影响机制。

背景与挑战

背景概述

在大规模语言模型预训练领域，数据质量与计算效率的平衡始终是核心议题。由BERTIN项目团队于2022年创建的mC4-es-sampled数据集，旨在解决西班牙语自然语言处理中资源受限的困境。该数据集基于Common Crawl的mC4西班牙语子集，通过创新的困惑度采样技术进行筛选，由Javier De la Rosa、Eduardo G. Ponferrada等研究人员主导开发。其核心研究问题在于：如何在不依赖海量数据和高昂算力的前提下，高效预训练出性能优异的西班牙语语言模型。这一工作显著降低了小规模团队参与大模型训练的门槛，为低资源语言模型的发展开辟了新路径，在西班牙语NLP领域具有里程碑式的影响力。

当前挑战

该数据集面临的挑战主要体现在两个层面。在领域问题层面，原始mC4西班牙语语料包含大量噪声（如机器翻译文本、低质量网页内容），直接用于预训练会导致模型学习效率低下且泛化能力不足，亟需一种能够自动识别并保留高质量文本的采样策略。在构建过程中，团队需解决三个具体挑战：首先，如何设计合理的困惑度采样算法（如随机采样、逐步采样、高斯采样）以平衡数据多样性与质量；其次，需要处理超过200GB的庞大数据量，在有限计算预算下完成分片下载与压缩存储；最后，必须确保采样后的50M样本子集在不同配置下的可复现性，为后续研究提供标准化基准。

常用场景

经典使用场景

mC4-es-sampled 数据集的核心应用在于为西班牙语语言模型的预训练提供高质量、低噪声的语料资源。该数据集通过对原始 mC4 西班牙语子集应用困惑度采样（perplexity sampling）技术，筛选出更具语言连贯性和信息密度的文本片段，从而显著提升模型在有限计算资源下的训练效率。经典使用场景包括基于 Transformer 架构的掩码语言模型（如 RoBERTa）的预训练，以及作为文本生成任务（如因果语言建模）的训练数据。研究者可依据随机（random）、逐步（stepwise）或高斯（gaussian）三种采样配置灵活选择数据分布，以适应不同规模与目标的实验需求。

衍生相关工作

mC4-es-sampled 的提出催生了一系列围绕数据质量与高效预训练的研究工作。其核心创新——困惑度采样方法被后续研究广泛借鉴，例如应用于其他低资源语言的语料筛选，或与课程学习（curriculum learning）策略结合以进一步优化训练动态。BERTIN 项目本身基于该数据集训练的 RoBERTa 模型成为西班牙语 NLP 的基准之一，推动了多个下游任务的性能提升。此外，该工作启发了对数据分布与模型泛化能力之间关系的深入探讨，相关成果发表于《Procesamiento del Lenguaje Natural》等期刊，并成为数据高效预训练领域的重要参考案例。

数据集最近研究

最新研究方向

在自然语言处理领域，大规模预训练语言模型的资源消耗问题日益凸显。mC4-es-sampled数据集通过引入基于困惑度的采样技术（perplexity sampling），对西班牙语子集进行高效筛选，显著降低了预训练所需的数据量和计算成本。这一前沿方向聚焦于数据中心的模型优化策略，即通过高质量子集的选择性训练，在保证模型性能的前提下，实现资源利用的极大化。BERTIN项目的研究表明，仅使用原始数据五分之一的样本，即可在约一半的训练步数内达到与当前最优模型相当甚至更优的效果，为资源受限的研究团队提供了可行的替代路径。该数据集不仅推动了低资源语言模型的高效训练，也呼应了当前对可持续AI发展的关注，其方法可推广至其他语种，具有重要的方法论意义和应用前景。