CBooks

该数据集名为CBooks，是通过开源MD5书籍链接收集的大规模中文书籍语料库，为语言建模提供了宝贵的长距离上下文信息。此外，该数据集因提升了连贯叙事和长距离上下文建模的能力而受到认可。其规模超过10万本图书，旨在用于大型语言模型的预训练任务。

The dataset named CBooks is a large-scale Chinese book corpus collected via open-source MD5 book links, which provides valuable long-distance contextual information for language modeling. Moreover, this dataset is recognized for enhancing the capabilities of coherent narrative and long-distance contextual modeling. Boasting a scale of over 100,000 books, it is specifically designed for pre-training tasks of large language models.

搜集汇总

背景与挑战

背景概述

CBook-150K是一个中文图书语料数据集，包含超过15万本图书的MD5链接，支持PDF、EPUB和MOBI格式的解析。该数据集由复旦大学自然语言处理实验室开发，主要用于科研目的，提供了完整的图书获取和文本提取工具链。