nikolina-p/gutenberg_flat
收藏Hugging Face2025-10-20 更新2025-10-25 收录
下载链接：
https://hf-mirror.com/datasets/nikolina-p/gutenberg_flat
下载链接
链接失效反馈官方服务：
资源简介：
Project Gutenberg数据集的清洗和分词英文子集，包含38026本书。该数据集经过过滤，去除了非英文文本、重复项和版权声明等，以便更清晰和方便使用。数据集使用OpenAI的tiktoken分词器进行分词，并为高效流式传输和分布式训练进行了优化。每行包含65537个分词，适合自回归模型训练。
A cleaned and tokenized English-language subset of the Project Gutenberg dataset containing 38,026 books. Non-English texts, duplicates, and boilerplate license sections were removed for clarity and usability. The dataset was tokenized using OpenAIs tiktoken tokenizer and optimized for efficient streaming and distributed training. Each row includes 65,537 tokens, which is suitable for autoregressive model training.
提供机构：
nikolina-p
