the_pile_books3

the_pile_books3 仓库提供英文书籍文本数据，规模约为19万条样本，总量超过100GB。每条数据包含书名和正文内容，主要用于语言建模等任务。该仓库基于MIT许可协议开放，数据来源于Shawn Presser的工作成果，并被整合到EleutherAI/The Pile数据集中。

The the_pile_books3 repository provides English book text datasets, which contains approximately 190,000 samples with a total volume exceeding 100 GB. Each sample includes the book's title and its full body text, and is primarily intended for tasks such as language modeling. This repository is released under the MIT License, and the dataset is derived from the work of Shawn Presser before being integrated into the EleutherAI/The Pile dataset.

搜集汇总

背景与挑战

背景概述

the_pile_books3数据集包含约19万本英文书籍的文本数据，总大小超过100GB，每条数据包含书名和正文内容，主要用于语言建模任务。该数据集基于MIT许可协议，但已因版权问题下架不再可访问。