iNeil77/pseudo-mini-pile

---
dataset_info:
- config_name: all
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 360187653412.6177
num_examples: 56194997
download_size: 199030076349
dataset_size: 360187653412.6177
- config_name: c4_realnews
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 31597106256.723488
num_examples: 11427438
download_size: 19889880484
dataset_size: 31597106256.723488
- config_name: openwebtext
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 30974178275.039234
num_examples: 6474479
download_size: 19069709415
dataset_size: 30974178275.039234
- config_name: peS2o
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 221900508006.5479
num_examples: 32612199
download_size: 116217303065
dataset_size: 221900508006.5479
- config_name: redpajama_books
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 49246538575.26426
num_examples: 107443
download_size: 29612204926
dataset_size: 49246538575.26426
- config_name: stackexchange
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 2034535930.2150385
num_examples: 716532
download_size: 1222605537
dataset_size: 2034535930.2150385
- config_name: uspto
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 14755999149.910166
num_examples: 3247716
download_size: 7058272149
dataset_size: 14755999149.910166
- config_name: wiki
features:
- name: content
dtype: string
splits:
- name: train
num_bytes: 7528525537.163156
num_examples: 1609190
download_size: 4593971902
dataset_size: 7528525537.163156
configs:
- config_name: all
data_files:
- split: train
path: all/train-*
- config_name: c4_realnews
data_files:
- split: train
path: c4_realnews/train-*
- config_name: openwebtext
data_files:
- split: train
path: openwebtext/train-*
- config_name: peS2o
data_files:
- split: train
path: peS2o/train-*
- config_name: redpajama_books
data_files:
- split: train
path: redpajama_books/train-*
- config_name: stackexchange
data_files:
- split: train
path: stackexchange/train-*
- config_name: uspto
data_files:
- split: train
path: uspto/train-*
- config_name: wiki
data_files:
- split: train
path: wiki/train-*
task_categories:
- text-generation
language:
- en
size_categories:
- 10M<n<100M
---

A small, aggressively cleaned and de-duped pre-training corpus for academic settings. It aims to recreate something akin to [The Pile](https://huggingface.co/datasets/EleutherAI/pile) but prioritizes quality for the constrained token budget academic researchers live with.

It has seven config subsets and an eighth `all` subset that combines them for a total of ~91B tokens (GPT2 Tokenizer estimate). These splits are as follows:

1. `c4_realnews`: The RealNews domain subset of the C4 dataset containing news articles.
2. `openwebtext`: The OpenWebText dataset containing the contents of the links mentioned in Reddit posts with at least 3 upvotes.
3. `peS2o`: The PeS2o dataset containing academic articles from Semantic Scholar.
4. `redpajama_books`: The books subset of RedPajama V1.
5. `stackexchange`: The EN StackExchange non-code subset of the BigScience ROOTs dataset.
6. `uspto`: The EN USPTO patent applications contents' subset of the BigScience ROOTs dataset.
7. `wiki`: The EN Wiki subset of the BigScience ROOTs dataset.

The following processing and filtering steps have been applied:

1. Removed citation text and bibliography information for academic texts.
2. Ran a perplexity filter using a KenLM model trained on the EN OSCAR corpus and removed documents with a perplexity of more than 325 and less than 7.
3. Removed samples which have a repeating <=4-gram proportion of 15%.
4. Removed samples which have lower than 99% confidence of being EN using the lingua language detector.
5. Performed an aggressive MinHash de-dupe using a shingle size of 8 and a low threshold of 0.5.

原始信息汇总

数据集概述

数据集配置

1. all

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 56,194,997 个样本，占用 360,187,653,412.6177 字节
• 下载大小: 199,030,076,349 字节
• 数据集大小: 360,187,653,412.6177 字节

2. c4_realnews

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 11,427,438 个样本，占用 31,597,106,256.723488 字节
• 下载大小: 19,889,880,484 字节
• 数据集大小: 31,597,106,256.723488 字节

3. openwebtext

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 6,474,479 个样本，占用 30,974,178,275.039234 字节
• 下载大小: 19,069,709,415 字节
• 数据集大小: 30,974,178,275.039234 字节

4. peS2o

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 32,612,199 个样本，占用 221,900,508,006.5479 字节
• 下载大小: 116,217,303,065 字节
• 数据集大小: 221,900,508,006.5479 字节

5. redpajama_books

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 107,443 个样本，占用 49,246,538,575.26426 字节
• 下载大小: 29,612,204,926 字节
• 数据集大小: 49,246,538,575.26426 字节

6. stackexchange

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 716,532 个样本，占用 2,034,535,930.2150385 字节
• 下载大小: 1,222,605,537 字节
• 数据集大小: 2,034,535,930.2150385 字节

7. uspto

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 3,247,716 个样本，占用 14,755,999,149.910166 字节
• 下载大小: 7,058,272,149 字节
• 数据集大小: 14,755,999,149.910166 字节

8. wiki

• 特征:
  • content: 类型为 string
• 分割:
  • train: 包含 1,609,190 个样本，占用 7,528,525,537.163156 字节
• 下载大小: 4,593,971,902 字节
• 数据集大小: 7,528,525,537.163156 字节

数据文件路径

• all: all/train-*
• c4_realnews: c4_realnews/train-*
• openwebtext: openwebtext/train-*
• peS2o: peS2o/train-*
• redpajama_books: redpajama_books/train-*
• stackexchange: stackexchange/train-*
• uspto: uspto/train-*
• wiki: wiki/train-*

任务类别

• 文本生成

语言

• 英语

大小类别

• 10M<n<100M