LAION-400M Dataset

LAION-400M is a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search.

⚠️ Disclaimer & Content Warning (from the authors)
Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, non-curated set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with caution. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering.

There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated.

Using KNN clustering should make it easy to further deduplicate by image content.

LAION-400M 是一个包含经过 CLIP 过滤的 4 亿个图像-文本对的数据库，其中包含了相应的 CLIP 嵌入和 kNN 索引，这些索引使得高效的相似性搜索成为可能。

⚠️ 声明与内容警告（来自作者）：本过滤协议仅移除了被检测为非法的 NSFW 图像，但数据集中仍包含相应标记的 NSFW 内容。在自由浏览数据集时，请记住，这是一个大规模、未经编辑的、出于研究目的从互联网爬取的集合，因此收集的链接可能导向令人不适或困扰的内容。因此，请谨慎使用演示链接。您可以通过过滤掉标记为 NSFW 或通过更严格的 CLIP 过滤选取的样本来提取一个“安全”的子集。

由于我们使用了 URL+文本作为去重标准，因此存在一定程度的重复。同一图像与相同说明可能位于不同的 URL 上，从而导致重复。然而，同一图像与其他说明的情况则不被视为重复。

利用 KNN 聚类应能简化通过图像内容进一步去重的操作。