m-a-p/PIN-100M

PIN-100M数据集是一个包含100M样本的知识密集型配对和交错多模态文档数据集。数据集包含多个子集，如PIN-PMC、DocLayNet、Linux-CN、chinese-markdown、OBELICS、MMC4、leetcode和PG19。每个样本包含文本内容和相关的图像文件，存储在内容图像文件夹和总体图像文件夹中。JSON Lines格式的文件提供了每个样本的详细信息，包括元数据和质量信号。数据集支持多种语言，如英语和中文。

The PIN-100M dataset is a knowledge-intensive paired and interleaved multimodal document dataset containing 100M samples. It includes multiple subsets such as PIN-PMC, DocLayNet, Linux-CN, chinese-markdown, OBELICS, MMC4, leetcode, and PG19. Each sample consists of text content and associated image files, stored in content image folders and overall image folders. JSON Lines formatted files provide detailed information for each sample, including metadata and quality signals. The dataset supports multiple languages, such as English and Chinese.