Spawning/PD12M

PD12M是一个包含1240万张图像-标题对的公共领域图像-文本数据集，是目前最大的此类数据集，旨在解决网络抓取训练数据中的质量问题，如版权问题、低质量图像和标题、暴力或不适宜内容、个人身份信息（PII）以及链接失效等。数据集由元数据和图像两部分组成，元数据包括图像URL、标题、尺寸等信息，图像存储在AWS S3桶中。数据集还提供了CLIP Vit-L/14嵌入。

PD12M is a large public domain image-text dataset containing 12.4 million image-caption pairs. The dataset was built and curated through the Source.Plus platform to address common data quality issues in web-scraped training data, such as copyrighted material, low-quality images and captions, violent or NSFW content, PII, and dataset quality decay due to broken links. PD12M consists entirely of public domain and CC0-licensed images, with automated image recaptioning and quality and safety filtering. The images are hosted on dedicated cloud storage to avoid placing undue burden on the original image hosts and to ensure the dataset remains intact over its lifetime. The dataset comprises two main components: metadata and images. The metadata includes image URLs, captions, dimensions, etc., while the image files are stored in an AWS S3 bucket.