datajuicer/the-pile-pubmed-central-refined-by-data-juicer

--- license: apache-2.0 task_categories: - text-generation language: - en tags: - data-juicer - pretraining size_categories: - 1M<n<10M --- # The Pile -- PubMed Central (refined by Data-Juicer) A refined version of PubMed Central dataset in The Pile by [Data-Juicer](https://github.com/alibaba/data-juicer). Removing some "bad" samples from the original dataset to make it higher-quality. This dataset is usually used to pretrain a Large Language Model. **Notice**: Here is a small subset for previewing. The whole dataset is available [here](https://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com/LLM_data/our_refined_datasets/pretraining/the-pile-pubmed-central-refine-result.jsonl) (About 83G). ## Dataset Information - Number of samples: 2,694,860 (Keep ~86.96% from the original dataset) ## Refining Recipe ```yaml # global parameters project_name: 'Data-Juicer-recipes-pubmed-central' dataset_path: '/path/to/your/dataset' # path to your dataset directory or file export_path: '/path/to/your/dataset.jsonl' np: 50 # number of subprocess to process your dataset open_tracer: true # process schedule # a list of several process operators with their arguments process: - clean_email_mapper: - clean_links_mapper: - fix_unicode_mapper: - punctuation_normalization_mapper: - whitespace_normalization_mapper: - alphanumeric_filter: # 89217 tokenization: false min_ratio: 0.2787 # 3sigma - average_line_length_filter: # for code max_len: 1200 # < 3sigma (1478) -- 7410 - character_repetition_filter: rep_len: 10 max_ratio: 0.3741 # 3sigma -- 65849 - flagged_words_filter: lang: en tokenization: true max_ratio: 0.00195 # 3sigma -- 8305 - language_id_score_filter: # remove language filter min_score: 0.5 # 272359 - maximum_line_length_filter: # for code max_len: 7328 # remove 23808 samples - perplexity_filter: lang: en max_ppl: 8000 # remove 173883 samples - special_characters_filter: max_ratio: 0.842 # remove 87661 samples - text_length_filter: max_len: 136028 # 3sigma -- 15118 - words_num_filter: lang: en tokenization: true min_num: 20 # remove 176537 samples max_num: 23305 # remove 15016 samples - word_repetition_filter: lang: en tokenization: true rep_len: 10 max_ratio: 0.5981 # 3sigma -- 93843 - document_simhash_deduplicator: tokenization: space window_size: 6 lowercase: true ignore_pattern: '\p{P}' num_blocks: 6 hamming_distance: 4 ```This is a refined version of the PubMed Central dataset within The Pile, processed using the Data-Juicer tool. Low-quality samples from the original dataset were removed to improve the overall quality of the dataset. This dataset is typically used for pre-training Large Language Models (LLMs). It contains 2,694,860 samples, retaining approximately 86.96% of the original dataset. The refinement process involves the application of multiple filters and mappers, including cleaning emails and links, normalizing Unicode, punctuation and whitespace characters, as well as filtering based on criteria such as character repetition, language identification scores, text length and word count.

原始信息汇总

The Pile -- PubMed Central (refined by Data-Juicer)

概述

这是一个经过Data-Juicer精炼的PubMed Central数据集版本，去除了原始数据集中的一些“不良”样本，以提高数据质量。该数据集通常用于预训练大型语言模型。

数据集信息

• 样本数量: 2,694,860（保留了原始数据集的约86.96%）
• 语言: 英语
• 标签: data-juicer, pretraining
• 大小类别: 1M<n<10M

精炼配方

• 全局参数:

  • 项目名称: Data-Juicer-recipes-pubmed-central
  • 数据集路径: /path/to/your/dataset
  • 导出路径: /path/to/your/dataset.jsonl
  • 子进程数量: 50
  • 开启追踪: true

• 处理流程:

  • 清洗电子邮件映射器
  • 清洗链接映射器
  • 修复Unicode映射器
  • 标点符号规范化映射器
  • 空白规范化映射器
  • 字母数字过滤器
  • 平均行长度过滤器
  • 字符重复过滤器
  • 标记词过滤器
  • 语言ID分数过滤器
  • 最大行长度过滤器
  • 困惑度过滤器
  • 特殊字符过滤器
  • 文本长度过滤器
  • 词数过滤器
  • 词重复过滤器
  • 文档Simhash去重器

搜集汇总

背景与挑战

背景概述

该数据集是The Pile中PubMed Central数据的精炼版本，通过Data-Juicer工具移除了约13.04%的低质量样本，以提高数据质量，适用于大语言模型的预训练。数据集包含约269万个英文文本样本，以JSON格式存储，采用Apache-2.0许可证，主要用于文本生成任务。