mlfoundations/MINT-1T-PDF-CC-2023-06

MINT-1T是一个开源的多模态交错数据集，包含1万亿文本标记和34亿张图像，旨在促进多模态预训练研究。数据集由华盛顿大学与Salesforce Research等机构合作创建，涵盖了HTML、PDF和ArXiv文档等多种来源。数据集经过严格的过滤和处理，以确保内容的质量和相关性。尽管数据集主要用于研究，但用户在使用时应注意潜在的数据偏见、内容风险和伦理问题。MINT-1T is an open-source multimodal interleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. The dataset includes previously untapped sources such as PDFs and ArXiv papers. MINT-1T is designed to facilitate research in multimodal pretraining and is created by a team from the University of Washington in collaboration with Salesforce Research, Stanford University, University of Texas at Austin, and University of California Berkeley. The dataset comprises a comprehensive collection of multimodal documents from various sources, including HTML, PDF, and ArXiv documents, which undergo a multi-step data collection, filtering, and processing to ensure content relevance and readability.