bookcorpus

BookCorpus主要用于文本生成和填充掩码任务。它包含从Smashwords网站抓取的超过7000本英文书籍的文本数据，总大小约为4.85GB。这些书籍最初由作者在Smashwords上自行发布，并按流派进行分类，但数据集本身没有提供明确的标注信息。BookCorpus主要用于非监督训练，但存在数据重复、版权问题和抽样偏差等局限性，使用时需要谨慎。数据集的使用需遵循Smashwords的服务条款。BookCorpus is primarily designed for text generation and masked filling tasks. It consists of textual data from over 7,000 English books scraped from the Smashwords website, with a total size of approximately 4.85 GB. These books were originally self-published by authors on Smashwords and categorized by genre, yet no explicit annotation information is provided within the dataset itself. BookCorpus is mainly utilized for unsupervised training, but it has limitations such as data duplication, copyright issues and sampling bias, so caution should be exercised when using it. The use of this dataset must comply with the terms of service of Smashwords.

搜集汇总

背景与挑战

背景概述

BookCorpus是一个用于文本生成和填充掩码任务的英文书籍数据集，包含从Smashwords网站抓取的超过7000本自出版书籍的文本，总大小约4.85GB，按流派分类但无明确标注。该数据集主要用于非监督训练，但存在数据重复、版权问题和抽样偏差等局限性，使用时需谨慎并遵守服务条款。