RichardErkhov/pile-uncopyrighted
收藏Hugging Face2024-08-26 更新2024-12-14 收录
下载链接：
https://hf-mirror.com/datasets/RichardErkhov/pile-uncopyrighted
下载链接
链接失效反馈官方服务：
资源简介：
Pile Uncopyrighted数据集是一个移除了所有受版权保护内容的数据集，旨在响应作者要求大型语言模型停止使用其作品的呼吁。该数据集鼓励在训练未来大型语言模型时尊重作者并遵守版权法。此外，有计划创建更大数据集（如RedPajama）的无版权版本，但尚未提供具体时间表。数据集的清理方法包括移除Books3、BookCorpus2、OpenSubtitles、YTSubtitles和OWT2子集中的所有内容，这些子集在原始论文的第7.1节中被指出不允许用于AI训练。
The Pile Uncopyrighted dataset is a variant of The Pile dataset, with all copyrighted content removed. This dataset was created in response to authors demanding that large language models stop using their works, aiming to respect authors copyrights and abide by copyright law. The cleaning process involved removing all content from the Books3, BookCorpus2, OpenSubtitles, YTSubtitles, and OWT2 subsets, as these subsets are not explicitly allowed to be used in AI training.
提供机构：
RichardErkhov
