adymaharana/cococon

---
annotations_creators:
- crowdsourced
language:
- en
language_creators:
- found
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: CoCoCON
size_categories:
- 1K<n<10K
tags:
- consistency
- visual-reasoning
task_ids: []
---

# Dataset Card for CoCoCON

- [Dataset Description](#dataset-description)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Description

CocoCON is a challenging dataset for evaluating cross-task consistency in vision-and-language models. We use contrast sets created by modifying COCO test instances for multiple tasks in small but semantically meaningful ways to change the gold label, and outline metrics for measuring if a model is consistent by ranking the original and perturbed instances across tasks. We find that state-of-the-art systems suffer from a surprisingly high degree of inconsistent behavior across tasks, especially for more heterogeneous tasks.

- **Homepage:**
https://adymaharana.github.io/cococon/
- **Repository:**
https://github.com/adymaharana/cococon
- **Paper:**
https://arxiv.org/abs/2303.16133
- **Point of Contact:**
[e-mail];

### Languages

English.

## Dataset Structure

Each sample in this dataset corresponds to a COCO image, a set of ground truth annotations for the image captioning, visual question-answering (VQA), and localization (optional) tasks, and their respective contrast sets.

### Data Fields

caption (string): ground truth caption.
query (string): VQA question.
answer (string): ground truth VQA answer.
question_id (int64): unordered unique identifier for sample.
image_id (int64): COCO image id.
detection (string): (optional) localization query.
boxes (list): (optional) list of ground truth bounding boxes for the localization query.
contrast_sets: Each sample in "contrast_sets" is a set of perturbed annotations corresponding to the ground truth annotations. Perturbed annotations are prefixed with "mutex_".
file_name (string): COCO filename for the image.
coco_url (string): url for downloading the image from the COCO server.
flickr_url (string): url for downloading the image from Flickr.
height (int64): height of image.
width (int64): width of image.
id (int64): ordered unique identifier for sample.

### Data Splits

The CocoCON benchmark is an evaluation-only dataset. The data accessible through this link should be considered as the test split.

## Dataset Creation

The CoCoCON dataset is created by a combination of machine + expert annotators who perturbed ground truth COCO annotations to create contrast sets.

## Considerations for Using the Data

### Licensing Information
CC-By 4.0
### Citation Information
@article{maharana2023cococon,
author = {Maharana, Adyasha and Kamath, Amita and Clark, Christopher and Bansal, Mohit and Kembhavi, Aniruddha},
title = {Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models.},
journal = {arxiv},
year = {2023},
}

原始信息汇总

数据集概述

名称： CoCoCON

语言： 英语

创建方式： 机器 + 专家注释者

许可证： CC-By 4.0

数据集大小： 1K<n<10K

标签：

• 一致性
• 视觉推理

数据集描述

CoCoCON是一个用于评估视觉与语言模型跨任务一致性的挑战性数据集。该数据集通过修改COCO测试实例创建对比集，以小但语义上有意义的方式改变黄金标签，并定义了衡量模型是否在跨任务中保持一致的排名指标。

数据集结构

数据字段：

• caption (字符串): 地面实况标题。
• query (字符串): VQA问题。
• answer (字符串): 地面实况VQA答案。
• question_id (int64): 样本的无序唯一标识符。
• image_id (int64): COCO图像ID。
• detection (字符串): (可选) 本地化查询。
• boxes (列表): (可选) 本地化查询的地面实况边界框列表。
• contrast_sets: 包含与地面实况注释对应的扰动注释集。
• file_name (字符串): COCO图像文件名。
• coco_url (字符串): 从COCO服务器下载图像的URL。
• flickr_url (字符串): 从Flickr下载图像的URL。
• height (int64): 图像高度。
• width (int64): 图像宽度。
• id (int64): 样本的有序唯一标识符。

数据分割：

• CocoCON基准是一个仅用于评估的数据集，通过此链接访问的数据应视为测试分割。

数据集创建

CoCoCON数据集由机器和专家注释者共同创建，通过扰动COCO地面实况注释来生成对比集。

使用数据注意事项

许可证信息： CC-By 4.0

引用信息：

@article{maharana2023cococon, author = {Maharana, Adyasha and Kamath, Amita and Clark, Christopher and Bansal, Mohit and Kembhavi, Aniruddha}, title = {Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models.}, journal = {arxiv}, year = {2023}, }