bertin-project/mc4-sampling
收藏Hugging Face2024-10-31 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/bertin-project/mc4-sampling
下载链接
链接失效反馈官方服务：
资源简介：
mC4-sampling数据集是一个多语言的巨型清洗版Common Crawl网络爬虫语料库，专为文本生成和填充掩码任务设计。该数据集支持108种语言，并提供了随机、高斯和逐步等多种基于困惑度的数据过滤和采样方法。它适用于预算有限的语言模型预训练，并根据ODC-BY许可发布。
提供机构：
bertin-project
原始信息汇总
数据集概述
数据集名称
- 名称: mC4-sampling
- 别名: mC4
数据集描述
- 概述: 该数据集基于AllenAI版本的原始mC4，并增加了采样方法以实时进行困惑度过滤。
- 原始数据集: mC4，多语言巨型清洗版Common Crawl网络爬虫语料库。
- 语言: 支持108种语言。
数据集结构
- 数据实例: 包含
url、text和timestamp字段。 - 数据字段:
url: 源URL，字符串类型。text: 文本内容，字符串类型。timestamp: 时间戳，字符串类型。
采样方法
- 随机采样: 基于概率阈值
factor进行文档保留。 - 高斯采样: 根据困惑度分布对文档进行采样，参数包括
factor和width。 - 步进采样: 根据困惑度分布的中心四分位数进行反比采样。
支持的任务
- 主要任务: 预训练语言模型和词表示。
许可证
- 许可证类型: ODC-BY
引用信息
- 引用格式: bibtex @article{BERTIN, author = {Javier De la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury}, title = {{BERTIN}: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling}, journal = {Procesamiento del Lenguaje Natural}, volume = {68}, number = {0}, year = {2022}, keywords = {}, abstract = {The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pretraining sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name perplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget.}, issn = {1989-7553}, url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6403}, pages = {13--23} }
