datajuicer/redpajama-cc-2023-06-refined-by-data-juicer

---
license: apache-2.0
task_categories:
- text-generation
language:
- en
tags:
- data-juicer
- pretraining
size_categories:
- 10M<n<100M
---

# RedPajama -- CommonCrawl-2023-06 (refined by Data-Juicer)

A refined version of CommonCrawl-2023-06 dataset in RedPajama by [Data-Juicer](https://github.com/alibaba/data-juicer). Removing some "bad" samples from the original dataset to make it higher-quality.

This dataset is usually used to pretrain a Large Language Model.

**Notice**: Here is a small subset for previewing. The whole dataset is available [here](https://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com/LLM_data/our_refined_datasets/pretraining/redpajama-cc-refine-results/redpajama-cc-2023-06-refine-result.jsonl) (About 310GB).

## Dataset Information

- Number of samples: 50,643,699 (Keep ~45.46% from the original dataset)

## Refining Recipe
```yaml
# global parameters
project_name: 'Data-Juicer-recipes-cc-2013-06'
dataset_path: '/path/to/your/dataset' # path to your dataset directory or file
export_path: '/path/to/your/dataset.jsonl'

np: 50 # number of subprocess to process your dataset
open_tracer: true

# process schedule
# a list of several process operators with their arguments
process:
- document_simhash_deduplicator:
tokenization: space
window_size: 6
lowercase: true
ignore_pattern: '\p{P}'
num_blocks: 6
hamming_distance: 4

- clean_email_mapper:
- clean_links_mapper:
- fix_unicode_mapper:
- punctuation_normalization_mapper:
- whitespace_normalization_mapper:

- alphanumeric_filter:
tokenization: false
min_ratio: 0.7508 # 3sigma
max_ratio: 0.8591 # 3sigma -- 1036821
- average_line_length_filter: # for code
max_len: 1500 # < 3sigma -- 395868
- character_repetition_filter:
rep_len: 10
max_ratio: 0.3 # > 3sigma -- 195026
- flagged_words_filter:
lang: en
tokenization: true
max_ratio: 0.0015 # 3sigma -- 287896
- language_id_score_filter: # remove language filter
min_score: 0.793 # 3sigma -- 2173246
- maximum_line_length_filter: # for code
max_len: 5000 # < 3sigma -- 797111
- perplexity_filter:
lang: en
max_ppl: 5000 # 3sigma -- 942162
- special_characters_filter:
min_ratio: 0.15 # > 3sigma
max_ratio: 0.35 # > 3sigma -- 1155090
- text_length_filter:
max_len: 58187 # 3sigma -- 1165902
- words_num_filter:
lang: en
tokenization: true
min_num: 20
max_num: 11529 # 3sigma -- 1185363
- word_repetition_filter:
lang: en
tokenization: true
rep_len: 10
max_ratio: 0.2962 # 3sigma -- 2407282
```

原始信息汇总

RedPajama -- CommonCrawl-2023-06 (refined by Data-Juicer)

数据集概述

• 数据集名称: RedPajama -- CommonCrawl-2023-06 (refined by Data-Juicer)
• 数据集类型: 预训练数据集
• 语言: 英语
• 标签: data-juicer, pretraining
• 数据量: 50,643,699 样本（保留了原始数据集的约45.46%）
• 大小: 10M<n<100M
• 许可证: Apache-2.0

数据集用途

该数据集通常用于预训练大型语言模型。

数据集处理

数据集通过Data-Juicer进行了精炼，去除了一些“不良”样本，以提高数据质量。

精炼配方

• 全局参数:

  • 项目名称: Data-Juicer-recipes-cc-2013-06
  • 数据集路径: /path/to/your/dataset
  • 导出路径: /path/to/your/dataset.jsonl
  • 子进程数量: 50
  • 开启追踪: true

• 处理流程:

  • 文档simhash去重
  • 清理电子邮件
  • 清理链接
  • 修复Unicode字符
  • 标点符号规范化
  • 空白规范化
  • 字母数字过滤
  • 平均行长度过滤
  • 字符重复过滤
  • 标记词过滤
  • 语言ID分数过滤
  • 最大行长度过滤
  • 困惑度过滤
  • 特殊字符过滤
  • 文本长度过滤
  • 单词数量过滤
  • 单词重复过滤