mlfoundations/MINT-1T-PDF-CC-2023-23

MINT-1T是一个开源的多模态交错数据集，包含1万亿文本标记和34亿张图像，是现有开源数据集的10倍规模。数据集旨在促进多模态预训练研究，数据来源于HTML文档、PDF文档和ArXiv论文。数据处理包括文档提取、过滤、图像和文本处理等步骤，以提高数据集的质量和适用性。数据集由华盛顿大学与Salesforce Research及其他学术机构合作创建。

MINT-1T is an open-source multimodal interleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. It is designed to facilitate research in multimodal pretraining, with data sourced from HTML documents, PDF documents, and ArXiv papers. The dataset undergoes extensive processing including document extraction, filtering, image and text processing to enhance quality and applicability. It was created by a team from the University of Washington in collaboration with Salesforce Research and other academic institutions.