mlfoundations/MINT-1T-PDF-CC-2024-10

MINT-1T是一个开源的多模态数据集，包含1万亿文本标记和34亿张图像，是现有开源数据集规模的10倍。该数据集旨在促进多模态预训练研究，涵盖了HTML文档、PDF文档和ArXiv论文等多种来源。数据集由华盛顿大学与Salesforce Research等机构合作创建，经过严格的文档提取、过滤、图像处理和文本处理等步骤，并使用了多种开源工具。尽管数据集经过严格过滤，但仍可能存在一些敏感信息，用户在使用时需谨慎。

MINT-1T is an open-source multimodal interleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. The dataset also includes previously untapped sources such as PDFs and ArXiv papers. MINT-1T is designed to facilitate research in multimodal pretraining and was created by a team from the University of Washington in collaboration with Salesforce Research and other academic institutions including Stanford University, University of Texas at Austin, and University of California Berkeley. The dataset is a comprehensive collection of multimodal documents from various sources, including HTML, PDF, and ArXiv documents, and underwent extensive data collection, filtering, and processing to ensure content relevance and readability.