COCO-Counterfactuals

# Dataset Card for COCO-Counterfactuals

<!-- Provide a quick summary of the dataset. -->

COCO-Counterfactuals is a high quality synthetic dataset for multimodal vision-language model evaluation and for training data augmentation.

## Dataset Details

### Dataset Description

COCO-Counterfactuals is a high quality synthetic dataset for multimodal vision-language model evaluation and for training data augmentation. Each COCO-Counterfactuals example includes a pair of image-text pairs; one is a counterfactual variation of the other. The two captions are identical to each other except a noun subject. The two corresponding synthetic images differ only in terms of the altered subject in the two captions. In our accompanying paper, we showed that the COCO-Counterfactuals dataset is challenging for existing pre-trained multimodal models and significantly increase the difficulty of the zero-shot image-text retrieval and image-text matching tasks. Our experiments also demonstrate that augmenting training data with COCO-Counterfactuals improves OOD generalization on multiple downstream tasks.

- **License:** CC-BY-4.0

### Dataset Sources

<!-- Provide the basic links for the dataset. -->

- **Repository:** https://huggingface.co/datasets/Intel/COCO-Counterfactuals
- **Paper:** https://openreview.net/forum?id=7AjdHnjIHX

### Data

The captions are located in `data/examples.jsonl` and the images are located in `data/images.zip`. You can load the data as follows:
```python
from datasets import load_dataset
examples = load_dataset('Intel/COCO-Counterfactuals', use_auth_token=<YOUR USER ACCESS TOKEN>)
```
You can get `<YOUR USER ACCESS TOKEN>` by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens"
5) generate an access token

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->
Despite the impressive recent improvements in text-to-image generation capabilities, models such as Stable Diffusion have well-known limitations that should be considered when utilizing datasets which
are derived from them. We do not foresee significant risks of security threats or human rights violations in our work. However, the automated nature of our image generation process may introduce the possibility of our COCO-Counterfactuals dataset
containing images that some individuals may consider inappropriate or offensive.

## Citation

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->
https://openreview.net/forum?id=7AjdHnjIHX

Tiep Le and Phillip Howard contributed equally.

**BibTeX:**
```
@inproceedings{le2023cococounterfactuals,
author = {Tiep Le and Vasudev Lal and Phillip Howard},
title = {{COCO}-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs},
booktitle = {Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year = 2023,
url={https://openreview.net/forum?id=7AjdHnjIHX},
}
```
## Dataset Card Authors

Tiep Le and Vasudev Lal and Phillip Howard

## Dataset Card Contact

[e-mail]; [e-mail]; [e-mail]

# COCO-反事实数据集卡片
<!-- 提供数据集的快速摘要。 -->
COCO-Counterfactuals是一款高质量合成数据集，适用于多模态视觉语言模型评估以及训练数据增强任务。

## 数据集详情
### 数据集描述
COCO-Counterfactuals是一款高质量合成数据集，用于多模态视觉语言模型评估与训练数据增强。每个COCO-Counterfactuals样本均包含一对图文对，二者互为反事实变体。两段标题文本除名词主语外完全一致，对应的两张合成图像也仅因标题中修改的主语存在差异。在配套论文中，我们证实COCO-Counterfactuals数据集对现有预训练多模态模型极具挑战性，可显著提升零样本（zero-shot）图文检索与图文匹配任务的难度。实验结果同时表明，通过COCO-Counterfactuals增强训练数据，能够改善多个下游任务的分布外（Out-of-Distribution, OOD）泛化性能。

- **许可证**：CC-BY-4.0

### 数据集来源
<!-- 提供数据集的基础链接。 -->
- **代码仓库**：https://huggingface.co/datasets/Intel/COCO-Counterfactuals
- **论文**：https://openreview.net/forum?id=7AjdHnjIHX

### 数据说明
标题文本存储于`data/examples.jsonl`，图像文件存储于`data/images.zip`。你可通过以下方式加载数据：
python
from datasets import load_dataset
examples = load_dataset('Intel/COCO-Counterfactuals', use_auth_token=<YOUR USER ACCESS TOKEN>)

你可通过以下步骤获取`<YOUR USER ACCESS TOKEN>`：
1) 登录你的Hugging Face账号
2) 点击个人头像
3) 点击「设置」
4) 点击「访问令牌」
5) 生成访问令牌

## 数据集结构
<!-- 本节用于描述数据集字段，以及划分标准、样本间关联等额外结构信息。 -->
[需要更多信息]

## 偏倚、风险与局限性
<!-- 本节用于说明技术与社会技术层面的局限性。 -->
尽管近期文本到图像生成技术取得了显著进展，但Stable Diffusion等模型存在公认的局限性，在使用基于此类模型构建的数据集时需加以考量。我们未预见本研究存在重大安全威胁或侵犯人权的风险。然而，由于图像生成流程具备自动化特性，COCO-Counterfactuals数据集可能包含部分个体认为不当或冒犯性的图像。

## 引用信息
<!-- 若有介绍该数据集的论文或博客文章，需在此处提供APA与Bibtex格式的引用信息。 -->
https://openreview.net/forum?id=7AjdHnjIHX

Tiep Le与Phillip Howard为共同第一作者。

**Bibtex格式引用：**

@inproceedings{le2023cococounterfactuals,
author = {Tiep Le and Vasudev Lal and Phillip Howard},
title = {{COCO}-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs},
booktitle = {第三十七届神经信息处理系统大会数据集与基准跟踪赛道},
year = 2023,
url={https://openreview.net/forum?id=7AjdHnjIHX},
}

## 数据集卡片作者
Tiep Le、Vasudev Lal与Phillip Howard

## 数据集卡片联系方式
[e-mail]; [e-mail]; [e-mail]