lingao123/coyo-700m

---
annotations_creators:
- no-annotation
language:
- en
language_creators:
- other
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: COYO-700M
size_categories:
- 100M<n<1B
source_datasets:
- original
tags:
- image-text pairs
task_categories:
- text-to-image
- image-to-text
- zero-shot-classification
task_ids:
- image-captioning
---

# Dataset Card for COYO-700M

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** [COYO homepage](https://kakaobrain.com/contents/?contentId=7eca73e3-3089-43cb-b701-332e8a1743fd)
- **Repository:** [COYO repository](https://github.com/kakaobrain/coyo-dataset)
- **Paper:**
- **Leaderboard:**
- **Point of Contact:** [COYO email]([e-mail])

### Dataset Summary

**COYO-700M** is a large-scale dataset that contains **747M image-text pairs** as well as many other **meta-attributes** to increase the usability to train various models. Our dataset follows a similar strategy to previous vision-and-language datasets, collecting many informative pairs of alt-text and its associated image in HTML documents. We expect COYO to be used to train popular large-scale foundation models
complementary to other similar datasets. For more details on the data acquisition process, please refer to the technical paper to be released later.

### Supported Tasks and Leaderboards

We empirically validated the quality of COYO dataset by re-implementing popular models such as [ALIGN](https://arxiv.org/abs/2102.05918), [unCLIP](https://arxiv.org/abs/2204.06125), and [ViT](https://arxiv.org/abs/2010.11929).
We trained these models on COYO-700M or its subsets from scratch, achieving competitive performance to the reported numbers or generated samples in the original papers.
Our pre-trained models and training codes will be released soon along with the technical paper.

### Languages

The texts in the COYO-700M dataset consist of English.

## Dataset Structure

### Data Instances

Each instance in COYO-700M represents single image-text pair information with meta-attributes:
```
{
'id': 841814333321,
'url': 'https://blog.dogsof.com/wp-content/uploads/2021/03/Image-from-iOS-5-e1614711641382.jpg',
'text': 'A Pomsky dog sitting and smiling in field of orange flowers',
'width': 1000,
'height': 988,
'image_phash': 'c9b6a7d8469c1959',
'text_length': 59,
'word_count': 11,
'num_tokens_bert': 13,
'num_tokens_gpt': 12,
'num_faces': 0,
'clip_similarity_vitb32': 0.4296875,
'clip_similarity_vitl14': 0.35205078125,
'nsfw_score_opennsfw2': 0.00031447410583496094,
'nsfw_score_gantman': 0.03298913687467575,
'watermark_score': 0.1014641746878624,
'aesthetic_score_laion_v2': 5.435476303100586
}
```

### Data Fields

| name | type | description |
|--------------------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| id | long | Unique 64-bit integer ID generated by [monotonically_increasing_id()](https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.monotonically_increasing_id.html) |
| url | string | The image URL extracted from the `src` attribute of the `<img>` tag |
| text | string | The text extracted from the `alt` attribute of the `<img>` tag |
| width | integer | The width of the image |
| height | integer | The height of the image |
| image_phash | string | The [perceptual hash(pHash)](http://www.phash.org/) of the image |
| text_length | integer | The length of the text |
| word_count | integer | The number of words separated by spaces. |
| num_tokens_bert | integer | The number of tokens using [BertTokenizer](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer) |
| num_tokens_gpt | integer | The number of tokens using [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast) |
| num_faces | integer | The number of faces in the image detected by [SCRFD](https://insightface.ai/scrfd) |
| clip_similarity_vitb32 | float | The cosine similarity between text and image(ViT-B/32) embeddings by [OpenAI CLIP](https://github.com/openai/CLIP) |
| clip_similarity_vitl14 | float | The cosine similarity between text and image(ViT-L/14) embeddings by [OpenAI CLIP](https://github.com/openai/CLIP) |
| nsfw_score_opennsfw2 | float | The NSFW score of the image by [OpenNSFW2](https://github.com/bhky/opennsfw2) |
| nsfw_score_gantman | float | The NSFW score of the image by [GantMan/NSFW](https://github.com/GantMan/nsfw_model) |
| watermark_score | float | The watermark probability of the image by our internal model |
| aesthetic_score_laion_v2 | float | The aesthetic score of the image by [LAION-Aesthetics-Predictor-V2](https://github.com/christophschuhmann/improved-aesthetic-predictor) |

### Data Splits

Data was not split, since the evaluation was expected to be performed on more widely used downstream task(s).

## Dataset Creation

### Curation Rationale

Similar to most vision-and-language datasets, our primary goal in the data creation process is to collect many pairs of alt-text and image sources in HTML documents crawled from the web. Therefore, We attempted to eliminate uninformative images or texts with minimal cost and improve our dataset's usability by adding various meta-attributes. Users can use these meta-attributes to sample a subset from COYO-700M and use it to train the desired model. For instance, the *num_faces* attribute could be used to make a subset like *COYO-Faces* and develop a privacy-preserving generative model.

### Source Data

#### Initial Data Collection and Normalization

We collected about 10 billion pairs of alt-text and image sources in HTML documents in [CommonCrawl](https://commoncrawl.org/) from Oct. 2020 to Aug. 2021. and eliminated uninformative pairs through the image and/or text level filtering process with minimal cost.

**Image Level**
* Included all image formats that [Pillow library](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html) can decode. (JPEG, WEBP, PNG, BMP, ...)
* Removed images less than 5KB image size.
* Removed images with an aspect ratio greater than 3.0.
* Removed images with min(width, height) < 200.
* Removed images with a score of [OpenNSFW2](https://github.com/bhky/opennsfw2) or [GantMan/NSFW](https://github.com/GantMan/nsfw_model) higher than 0.5.
* Removed all duplicate images based on the image [pHash](http://www.phash.org/) value from external public datasets.
* ImageNet-1K/21K, Flickr-30K, MS-COCO, CC-3M, CC-12M

**Text Level**
* Collected only English text using [cld3](https://github.com/google/cld3).
* Replaced consecutive whitespace characters with a single whitespace and removed the whitespace before and after the sentence.
(e.g. `"\n \n Load image into Gallery viewer, valentine&amp;#39;s day roses\n \n" → "Load image into Gallery viewer, valentine&amp;#39;s day roses"`)
* Removed texts with a length of 5 or less.
* Removed texts that do not have a noun form.
* Removed texts with less than 3 words or more than 256 words and texts over 1000 in length.
* Removed texts appearing more than 10 times.
(e.g. `“thumbnail for”, “image for”, “picture of”`)
* Removed texts containing NSFW words collected from [profanity_filter](https://github.com/rominf/profanity-filter/blob/master/profanity_filter/data/en_profane_words.txt), [better_profanity](https://github.com/snguyenthanh/better_profanity/blob/master/better_profanity/profanity_wordlist.txt), and [google_twunter_lol](https://gist.github.com/ryanlewis/a37739d710ccdb4b406d).

**Image-Text Level**
* Removed duplicated samples based on (image_phash, text).
(Different text may exist for the same image URL.)

#### Who are the source language producers?

[Common Crawl](https://commoncrawl.org/) is the data source for COYO-700M.

### Annotations

#### Annotation process

The dataset was built in a fully automated process that did not require human annotation.

#### Who are the annotators?

No human annotation

### Personal and Sensitive Information

#### Disclaimer & Content Warning

The COYO dataset is recommended to be used for research purposes.
Kakao Brain tried to construct a "Safe" dataset when building the COYO dataset. (See [Data Filtering](#source-data) Section) Kakao Brain is constantly making efforts to create more "Safe" datasets.
However, despite these efforts, this large-scale dataset was not hand-picked by humans to avoid the risk due to its very large size (over 700M).
Keep in mind that the unscreened nature of the dataset means that the collected images can lead to strongly discomforting and disturbing content for humans.
The COYO dataset may contain some inappropriate data, and any problems resulting from such data are the full responsibility of the user who used it.
Therefore, it is strongly recommended that this dataset be used only for research, keeping this in mind when using the dataset, and Kakao Brain does not recommend using this dataset as it is without special processing to clear inappropriate data to create commercial products.

## Considerations for Using the Data

### Social Impact of Dataset

It will be described in a paper to be released soon.

### Discussion of Biases

It will be described in a paper to be released soon.

### Other Known Limitations

It will be described in a paper to be released soon.

## Additional Information

### Dataset Curators

COYO dataset was released as an open source in the hope that it will be helpful to many research institutes and startups for research purposes. We look forward to contacting us from various places who wish to cooperate with us.

[[e-mail]](mailto:[e-mail])

### Licensing Information

#### License

The COYO dataset of Kakao Brain is licensed under [CC-BY-4.0 License](https://creativecommons.org/licenses/by/4.0/).
The full license can be found in the [LICENSE.cc-by-4.0 file](./coyo-700m/blob/main/LICENSE.cc-by-4.0).
The dataset includes “Image URL” and “Text” collected from various sites by analyzing Common Crawl data, an open data web crawling project.
The collected data (images and text) is subject to the license to which each content belongs.

#### Obligation to use

While Open Source may be free to use, that does not mean it is free of obligation.
To determine whether your intended use of the COYO dataset is suitable for the CC-BY-4.0 license, please consider the license guide.
If you violate the license, you may be subject to legal action such as the prohibition of use or claim for damages depending on the use.

### Citation Information

If you apply this dataset to any project and research, please cite our code:

```
@misc{kakaobrain2022coyo-700m,
title = {COYO-700M: Image-Text Pair Dataset},
author = {Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, Saehoon Kim},
year = {2022},
howpublished = {\url{https://github.com/kakaobrain/coyo-dataset}},
}
```

### Contributions

- Minwoo Byeon ([@mwbyeon](https://github.com/mwbyeon))
- Beomhee Park ([@beomheepark](https://github.com/beomheepark))
- Haecheon Kim ([@HaecheonKim](https://github.com/HaecheonKim))
- Sungjun Lee ([@justhungryman](https://github.com/justHungryMan))
- Woonhyuk Baek ([@wbaek](https://github.com/wbaek))
- Saehoon Kim ([@saehoonkim](https://github.com/saehoonkim))
- and Kakao Brain Large-Scale AI Studio

---
标注创建者:
- 无标注
语言:
- 英语
语言创建者:
- 其他
许可协议:
- cc-by-4.0
多语言类型:
- 单语言
友好名称: COYO-700M
规模类别:
- 1亿<样本数<10亿
源数据集:
- 原始数据集
标签:
- 图像-文本对（image-text pairs）
任务类别:
- 文本到图像
- 图像到文本
- 零样本分类（zero-shot-classification）
任务子项:
- 图像字幕生成（image-captioning）
---

# COYO-700M 数据集卡片

## 目录
- [目录](#table-of-contents)
- [数据集描述](#dataset-description)
- [数据集摘要](#dataset-summary)
- [支持任务与基准榜单](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [构建初衷](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集构建者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献者](#contributions)

## 数据集描述

- **主页:** [COYO 主页](https://kakaobrain.com/contents/?contentId=7eca73e3-3089-43cb-b701-332e8a1743fd)
- **代码仓库:** [COYO 代码仓库](https://github.com/kakaobrain/coyo-dataset)
- **论文:**
- **基准榜单:**
- **联系方式:** [COYO 官方邮箱]([e-mail])

### 数据集摘要

**COYO-700M** 是一款大规模数据集，包含 **7.47亿组图像-文本对（image-text pairs）**，同时附带诸多额外**元属性（meta-attributes）**，以提升其在各类模型训练中的易用性。本数据集遵循此前视觉-语言数据集的通用构建策略，从HTML文档中采集大量包含有效信息的替代文本（alt-text）与对应图像的配对数据。我们期望COYO数据集可用于训练主流大规模基础模型，作为其他同类数据集的补充。关于数据采集流程的更多细节，请参阅后续发布的技术论文。

### 支持任务与基准榜单

我们通过复现[ALIGN](https://arxiv.org/abs/2102.05918)、[unCLIP](https://arxiv.org/abs/2204.06125)与[ViT](https://arxiv.org/abs/2010.11929)等主流模型，从实验层面验证了COYO数据集的质量。我们基于COYO-700M及其子集从零开始训练上述模型，最终取得了与原论文报告数值或生成样本相当的性能。我们的预训练模型与训练代码将随技术论文同步发布。

### 语言

COYO-700M数据集中的文本均为英语。

## 数据集结构

### 数据实例

COYO-700M中的每个实例均代表单组图像-文本对及其元属性信息，示例格式如下：

{
'id': 841814333321,
'url': 'https://blog.dogsof.com/wp-content/uploads/2021/03/Image-from-iOS-5-e1614711641382.jpg',
'text': 'A Pomsky dog sitting and smiling in field of orange flowers',
'width': 1000,
'height': 988,
'image_phash': 'c9b6a7d8469c1959',
'text_length': 59,
'word_count': 11,
'num_tokens_bert': 13,
'num_tokens_gpt': 12,
'num_faces': 0,
'clip_similarity_vitb32': 0.4296875,
'clip_similarity_vitl14': 0.35205078125,
'nsfw_score_opennsfw2': 0.00031447410583496094,
'nsfw_score_gantman': 0.03298913687467575,
'watermark_score': 0.1014641746878624,
'aesthetic_score_laion_v2': 5.435476303100586
}

### 数据字段

| 字段名 | 类型 | 描述 |
|--------------------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| id | long | 唯一64位整数ID，由[monotonically_increasing_id()](https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.monotonically_increasing_id.html)生成 |
| url | string | 从`<img>`标签的`src`属性中提取的图像URL |
| text | string | 从`<img>`标签的`alt`属性中提取的文本 |
| width | integer | 图像宽度 |
| height | integer | 图像高度 |
| image_phash | string | 图像的[感知哈希（perceptual hash, pHash）](http://www.phash.org/) |
| text_length | integer | 文本长度 |
| word_count | integer | 以空格分隔的单词数。 |
| num_tokens_bert | integer | 使用[Bert分词器（BertTokenizer）](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer)得到的Token数 |
| num_tokens_gpt | integer | 使用[GPT2快速分词器（GPT2TokenizerFast）](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast)得到的Token数 |
| num_faces | integer | 通过[SCRFD](https://insightface.ai/scrfd)检测到的图像中人脸数量 |
| clip_similarity_vitb32 | float | 通过[OpenAI CLIP](https://github.com/openai/CLIP)计算的文本与图像(ViT-B/32)嵌入向量之间的余弦相似度 |
| clip_similarity_vitl14 | float | 通过[OpenAI CLIP](https://github.com/openai/CLIP)计算的文本与图像(ViT-L/14)嵌入向量之间的余弦相似度 |
| nsfw_score_opennsfw2 | float | 通过[OpenNSFW2](https://github.com/bhky/opennsfw2)计算的图像NSFW评分 |
| nsfw_score_gantman | float | 通过[GantMan/NSFW](https://github.com/GantMan/nsfw_model)计算的图像NSFW评分 |
| watermark_score | float | 通过我们的内部模型计算的图像水印概率 |
| aesthetic_score_laion_v2 | float | 通过[LAION美学预测器V2（LAION-Aesthetics-Predictor-V2）](https://github.com/christophschuhmann/improved-aesthetic-predictor)计算的图像美学评分 |

### 数据划分

由于评估通常基于更广泛使用的下游任务，本数据集未进行划分。

## 数据集构建

### 构建初衷

与大多数视觉-语言数据集类似，我们在数据构建过程中的核心目标是从网络爬取的HTML文档中采集大量替代文本与图像源的配对数据。因此，我们尝试以最低成本剔除无效的图像或文本，并通过添加各类元属性提升数据集的易用性。用户可利用这些元属性从COYO-700M中采样子集，用于训练目标模型。例如，*num_faces*属性可用于构建如*COYO-Faces*这类子集，进而开发隐私保护的生成模型。

### 源数据

#### 初始数据采集与标准化

我们于2020年10月至2021年8月期间，从[CommonCrawl](https://commoncrawl.org/)中采集了约100亿组HTML文档中的替代文本与图像源配对数据，并通过图像和/或文本层面的过滤流程，以最低成本剔除无效配对。

**图像层面**
* 保留所有[Pillow库](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html)可解码的图像格式（JPEG、WEBP、PNG、BMP等）。
* 剔除文件大小小于5KB的图像。
* 剔除宽高比大于3.0的图像。
* 剔除最小边（宽、高中较小者）小于200像素的图像。
* 剔除[OpenNSFW2](https://github.com/bhky/opennsfw2)或[GantMan/NSFW](https://github.com/GantMan/nsfw_model)评分高于0.5的图像。
* 基于图像[感知哈希（pHash）](http://www.phash.org/)值，剔除来自外部公开数据集的重复图像：
* ImageNet-1K/21K、Flickr-30K、MS-COCO、CC-3M、CC-12M

**文本层面**
* 通过[cld3](https://github.com/google/cld3)仅保留英语文本。
* 将连续空白字符替换为单个空格，并移除句子前后的空白字符（例如：`"

Load image into Gallery viewer, valentine&amp;#39;s day roses

" → "Load image into Gallery viewer, valentine&amp;#39;s day roses"`）。
* 剔除长度小于5的文本。
* 剔除不包含名词形式的文本。
* 剔除单词数少于3个或多于256个，且文本长度超过1000的文本。
* 剔除出现次数超过10次的文本（例如：`“thumbnail for”、“image for”、“picture of”`）。
* 剔除包含从[profanity_filter](https://github.com/rominf/profanity-filter/blob/master/profanity_filter/data/en_profane_words.txt)、[better_profanity](https://github.com/snguyenthanh/better_profanity/blob/master/better_profanity/profanity_wordlist.txt)以及[google_twunter_lol](https://gist.github.com/ryanlewis/a37739d710ccdb4b406d)中收集的不雅词汇的文本。

**图像-文本层面**
* 基于（image_phash, text）剔除重复样本（同一图像URL可能对应不同文本）。

#### 源语言生产者是谁？

[Common Crawl](https://commoncrawl.org/)是COYO-700M的数据源。

### 标注信息

#### 标注流程

本数据集通过完全自动化的流程构建，无需人工标注。

#### 标注者是谁？

无人工标注

### 个人与敏感信息

#### 免责声明与内容警告

COYO数据集仅推荐用于研究用途。Kakao Brain在构建COYO数据集时，致力于打造“安全”的数据集（详见[数据过滤](#source-data)章节），并持续投入更多精力以构建更安全的数据集。然而，尽管做出了这些努力，由于本数据集规模庞大（超过7亿样本），未经过人工逐一筛选。请注意，数据集的未筛选特性意味着其中的图像可能包含令使用者感到强烈不适或冒犯的内容。COYO数据集可能包含部分不当数据，因使用此类数据引发的任何问题均由使用者自行承担全部责任。因此，强烈建议本数据集仅用于研究用途，使用时请牢记这一点；Kakao Brain不建议在未进行特殊处理以清除不当数据的情况下，直接使用本数据集开发商业产品。

## 数据集使用注意事项

### 数据集的社会影响

将在后续发布的技术论文中详述。

### 偏差讨论

将在后续发布的技术论文中详述。

### 其他已知局限性

将在后续发布的技术论文中详述。

## 附加信息

### 数据集构建者

COYO数据集以开源形式发布，以期为众多研究机构与初创公司的研究工作提供帮助。我们期待与各类有意向合作的机构取得联系。
[[e-mail]](mailto:[e-mail])

### 许可信息

#### 许可协议

Kakao Brain发布的COYO数据集采用[CC-BY-4.0许可协议](https://creativecommons.org/licenses/by/4.0/)。完整许可协议可在[LICENSE.cc-by-4.0文件](./coyo-700m/blob/main/LICENSE.cc-by-4.0)中查看。本数据集包含通过分析Common Crawl（一项开源网络爬取项目）数据，从各类网站收集的“图像URL”与“文本”。所收集的数据（图像与文本）受各自所属内容的许可协议约束。

#### 使用义务

尽管开源软件可免费使用，但这并不意味着使用时无需承担义务。若要确定您对COYO数据集的预期使用方式是否符合CC-BY-4.0许可协议，请参阅许可指南。若违反许可协议，您可能会面临诸如禁止使用或损害赔偿索赔等法律诉讼。

### 引用信息

若您将本数据集应用于任何项目或研究中，请引用以下代码：

@misc{kakaobrain2022coyo-700m,
title = {COYO-700M: Image-Text Pair Dataset},
author = {Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, Saehoon Kim},
year = {2022},
howpublished = {url{https://github.com/kakaobrain/coyo-dataset}},
}

### 贡献者

- Minwoo Byeon ([@mwbyeon](https://github.com/mwbyeon))
- Beomhee Park ([@beomheepark](https://github.com/beomheepark))
- Haecheon Kim ([@HaecheonKim](https://github.com/HaecheonKim))
- Sungjun Lee ([@justhungryman](https://github.com/justHungryMan))
- Woonhyuk Baek ([@wbaek](https://github.com/wbaek))
- Saehoon Kim ([@saehoonkim](https://github.com/saehoonkim))
- 以及 Kakao Brain 大规模人工智能工作室