pietrolesci/pile-deduped

这是一个用于文本生成任务的英文大型数据集，包含原始文档、标记化文档和序列跟踪文档。每个文件大约包含100万份文档，每个文档都有唯一的ID。数据集分为不同的配置，包括默认配置、标记化配置和统计配置。原始数据与EleutherAI/the_pile_deduplicated数据集类似，但经过了一些处理，如去标记化，并增加了每份文档的字符数和标记数的列。

This is a large English dataset for text generation tasks, containing raw documents, tokenized documents, and sequence-tracking documents. Each file includes approximately 1 million documents, each with a unique ID. The dataset is divided into different configurations, including default, tokenized, and stats configurations. The original data is similar to the EleutherAI/the_pile_deduplicated dataset but has undergone some processing, such as detokenization, and additional columns for the number of characters and tokens in each document have been added.