ArmelR/the-pile-splitted

该数据集是一个由EleutherAI设计的800GB英文文本数据集，用于训练大规模语言模型。数据集分为22个高质量的子数据集，并通过`meta`列将数据集正确划分为子集。每个实例都有一个`domain`列，表示其所属的子集。数据集进一步被划分为训练集和测试集（97%/3%）。

原始信息汇总

数据集概述

数据集结构

配置名称：all

• 训练数据路径:

  • "data/ArXiv/train/*.arrow"
  • "data/BookCorpus2/train/*.arrow"
  • "data/Books3/train/*.arrow"
  • "data/DM Mathematics/train/*.arrow"
  • "data/Enron Emails/train/*.arrow"
  • "data/EuroParl/train/*.arrow"
  • "data/FreeLaw/train/*.arrow"
  • "data/Github/train/*.arrow"
  • "data/Gutenberg (PG-19)/train/*.arrow"
  • "data/HackerNews/train/*.arrow"
  • "data/NIH ExPorter/train/*.arrow"
  • "data/OpenSubtitles/train/*.arrow"
  • "data/OpenWebText2/train/*.arrow"
  • "data/PhilPapers/train/*.arrow"
  • "data/Pile-CC/train/*.arrow"
  • "data/PubMed Abstracts/train/*.arrow"
  • "data/PubMed Central/train/*.arrow"
  • "data/StackExchange/train/*.arrow"
  • "data/UPSTO Backgrounds/train/*.arrow"
  • "data/Ubuntu IRC/train/*.arrow"
  • "data/Wikipedia (en)/train/*.arrow"
  • "data/YoutubeSubtitles/train/*.arrow"

• 测试数据路径:

  • "data/ArXiv/test/*.arrow"
  • "data/BookCorpus2/test/*.arrow"
  • "data/Books3/test/*.arrow"
  • "data/DM Mathematics/test/*.arrow"
  • "data/Enron Emails/test/*.arrow"
  • "data/EuroParl/test/*.arrow"
  • "data/FreeLaw/test/*.arrow"
  • "data/Github/test/*.arrow"
  • "data/Gutenberg (PG-19)/test/*.arrow"
  • "data/HackerNews/test/*.arrow"
  • "data/NIH ExPorter/test/*.arrow"
  • "data/OpenSubtitles/test/*.arrow"
  • "data/OpenWebText2/test/*.arrow"
  • "data/PhilPapers/test/*.arrow"
  • "data/Pile-CC/test/*.arrow"
  • "data/PubMed Abstracts/test/*.arrow"
  • "data/PubMed Central/test/*.arrow"
  • "data/StackExchange/test/*.arrow"
  • "data/UPSTO Backgrounds/test/*.arrow"
  • "data/Ubuntu IRC/test/*.arrow"
  • "data/Wikipedia (en)/test/*.arrow"
  • "data/YoutubeSubtitles/test/*.arrow"

其他配置名称

• ArXiv, BookCorpus2, Books3, DM Mathematics, Enron Emails, EuroParl, FreeLaw, Github, Gutenberg (PG-19), HackerNews, NIH ExPorter, OpenSubtitles, OpenWebText2, PhilPapers, Pile-CC, PubMed Abstracts, PubMed Central, StackExchange, UPSTO Backgrounds, Ubuntu IRC, Wikipedia (en), YoutubeSubtitles
  • 训练数据路径: 对应数据集的训练目录下的*.arrow文件
  • 测试数据路径: 对应数据集的测试目录下的*.arrow文件

数据集使用

• 使用load_dataset函数加载数据集，通过指定subset_of_interest参数来选择加载的数据子集。默认加载整个数据集。