mlfoundations/MINT-1T-PDF-CC-2023-14

MINT-1T是一个开源的多模态交错数据集，包含1万亿文本标记和34亿张图像，是现有开源数据集的10倍规模。数据集旨在促进多模态预训练研究，包含来自HTML文档、PDF文档和ArXiv论文的多模态文档。数据集经过严格的过滤和处理，以确保内容的相关性和可读性，并尽可能减少个人和敏感信息的包含。数据集由华盛顿大学与Salesforce Research及其他学术机构合作创建。

MINT-1T is an open-source multimodal interleaved dataset containing 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, the dataset includes previously untapped sources such as PDFs and ArXiv papers. MINT-1T is designed to facilitate research in multimodal pretraining. The dataset was created by a team from the University of Washington in collaboration with Salesforce Research and other academic institutions including Stanford University, University of Texas at Austin, and University of California Berkeley.