severo/winogavil

---
annotations_creators:
- crowdsourced
language:
- en
language_creators:
- found
license:
- cc-by-4.0
multilinguality:
- monolingual
paperswithcode_id: winogavil
pretty_name: WinoGAViL
size_categories:
- 10K<n<100K
source_datasets:
- original
tags:
- commonsense-reasoning
- visual-reasoning
task_ids: []
extra_gated_prompt: "By clicking on “Access repository” below, you also agree that you are using it solely for research purposes. The full license agreement is available in the dataset files."
---

# Dataset Card for WinoGAViL

- [Dataset Description](#dataset-description)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Colab notebook code for Winogavil evaluation with CLIP](#colab-notebook-code-for-winogavil-evaluation-with-clip)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Description

WinoGAViL is a challenging dataset for evaluating vision-and-language commonsense reasoning abilities. Given a set of images, a cue, and a number K, the task is to select the K images that best fits the association. This dataset was collected via the WinoGAViL online game to collect vision-and-language associations, (e.g., werewolves to a full moon). Inspired by the popular card game Codenames, a spymaster gives a textual cue related to several visual candidates, and another player has to identify them. Human players are rewarded for creating associations that are challenging for a rival AI model but still solvable by other human players. We evaluate several state-of-the-art vision-and-language models, finding that they are intuitive for humans (>90% Jaccard index) but challenging for state-of-the-art AI models, where the best model (ViLT) achieves a score of 52%, succeeding mostly where the cue is visually salient. Our analysis as well as the feedback we collect from players indicate that the collected associations require diverse reasoning skills, including general knowledge, common sense, abstraction, and more.

- **Homepage:**
https://winogavil.github.io/
- **Colab**
https://colab.research.google.com/drive/19qcPovniLj2PiLlP75oFgsK-uhTr6SSi
- **Repository:**
https://github.com/WinoGAViL/WinoGAViL-experiments/
- **Paper:**
https://arxiv.org/abs/2207.12576
- **Leaderboard:**
https://winogavil.github.io/leaderboard
- **Point of Contact:**
[e-mail]; [e-mail]

### Supported Tasks and Leaderboards

https://winogavil.github.io/leaderboard.
https://paperswithcode.com/dataset/winogavil.

## Colab notebook code for Winogavil evaluation with CLIP
https://colab.research.google.com/drive/19qcPovniLj2PiLlP75oFgsK-uhTr6SSi

### Languages

English.

## Dataset Structure

### Data Fields

candidates (list): ["bison", "shelter", "beard", "flea", "cattle", "shave"] - list of image candidates.
cue (string): pogonophile - the generated cue.
associations (string): ["bison", "beard", "shave"] - the images associated with the cue selected by the user.
score_fool_the_ai (int64): 80 - the spymaster score (100 - model score) for fooling the AI, with CLIP RN50 model.
num_associations (int64): 3 - The number of images selected as associative with the cue.
num_candidates (int64): 6 - the number of total candidates.
solvers_jaccard_mean (float64): 1.0 - three solvers scores average on the generated association instance.
solvers_jaccard_std (float64): 1.0 - three solvers scores standard deviation on the generated association instance
ID (int64): 367 - association ID.

### Data Splits
There is a single TEST split. In the accompanied paper and code we sample it to create different training sets, but the intended use is to use winogavil as a test set.
There are different number of candidates, which creates different difficulty levels:
-- With 5 candidates, random model expected score is 38%.
-- With 6 candidates, random model expected score is 34%.
-- With 10 candidates, random model expected score is 24%.
-- With 12 candidates, random model expected score is 19%.

<details>
<summary>Why random chance for success with 5 candidates is 38%?</summary>

It is a binomial distribution probability calculation.

Assuming N=5 candidates, and K=2 associations, there could be three events:
(1) The probability for a random guess is correct in 0 associations is 0.3 (elaborate below), and the Jaccard index is 0 (there is no intersection between the correct labels and the wrong guesses). Therefore the expected random score is 0.
(2) The probability for a random guess is correct in 1 associations is 0.6, and the Jaccard index is 0.33 (intersection=1, union=3, one of the correct guesses, and one of the wrong guesses). Therefore the expected random score is 0.6*0.33 = 0.198.
(3) The probability for a random guess is correct in 2 associations is 0.1, and the Jaccard index is 1 (intersection=2, union=2). Therefore the expected random score is 0.1*1 = 0.1.
* Together, when K=2, the expected score is 0+0.198+0.1 = 0.298.

To calculate (1), the first guess needs to be wrong. There are 3 "wrong" guesses and 5 candidates, so the probability for it is 3/5. The next guess should also be wrong. Now there are only 2 "wrong" guesses, and 4 candidates, so the probability for it is 2/4. Multiplying 3/5 * 2/4 = 0.3.
Same goes for (2) and (3).

Now we can perform the same calculation with K=3 associations.
Assuming N=5 candidates, and K=3 associations, there could be four events:
(4) The probability for a random guess is correct in 0 associations is 0, and the Jaccard index is 0. Therefore the expected random score is 0.
(5) The probability for a random guess is correct in 1 associations is 0.3, and the Jaccard index is 0.2 (intersection=1, union=4). Therefore the expected random score is 0.3*0.2 = 0.06.
(6) The probability for a random guess is correct in 2 associations is 0.6, and the Jaccard index is 0.5 (intersection=2, union=4). Therefore the expected random score is 0.6*5 = 0.3.
(7) The probability for a random guess is correct in 3 associations is 0.1, and the Jaccard index is 1 (intersection=3, union=3). Therefore the expected random score is 0.1*1 = 0.1.
* Together, when K=3, the expected score is 0+0.06+0.3+0.1 = 0.46.

Taking the average of 0.298 and 0.46 we reach 0.379.

Same process can be recalculated with 6 candidates (and K=2,3,4), 10 candidates (and K=2,3,4,5) and 123 candidates (and K=2,3,4,5,6).

</details>

## Dataset Creation

Inspired by the popular card game Codenames, a “spymaster” gives a textual cue related to several visual candidates, and another player has to identify them. Human players are rewarded for creating
associations that are challenging for a rival AI model but still solvable by other
human players.

### Annotations

#### Annotation process

We paid Amazon Mechanical Turk Workers to play our game.

## Considerations for Using the Data

All associations were obtained with human annotators.

### Licensing Information

CC-By 4.0

### Citation Information

@article{bitton2022winogavil,
title={WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models},
author={Bitton, Yonatan and Guetta, Nitzan Bitton and Yosef, Ron and Elovici, Yuval and Bansal, Mohit and Stanovsky, Gabriel and Schwartz, Roy},
journal={arXiv preprint arXiv:2207.12576},
year={2022}

原始信息汇总

数据集概述

名称: WinoGAViL
语言: 英语
许可证: CC-By 4.0
多语言性: 单语种
规模: 10K<n<100K
来源: 原创数据集
标签: 常识推理, 视觉推理
任务: 视觉与语言常识推理能力评估

数据集描述

WinoGAViL是一个用于评估视觉与语言常识推理能力的挑战性数据集。任务涉及根据一组图像、一个提示和数字K，选择最适合与提示关联的K张图像。此数据集通过WinoGAViL在线游戏收集，灵感来源于流行的卡牌游戏Codenames。

数据集结构

数据字段

• candidates (list): 图像候选列表。
• cue (string): 生成的提示。
• associations (string): 用户选择的与提示关联的图像。
• score_fool_the_ai (int64): 愚弄AI的得分。
• num_associations (int64): 选择的关联图像数量。
• num_candidates (int64): 总候选图像数量。
• solvers_jaccard_mean (float64): 解题者得分的平均值。
• solvers_jaccard_std (float64): 解题者得分的标准差。
• ID (int64): 关联ID。

数据分割

• 分割: 单个测试集。
• 难度级别: 根据候选图像数量不同，难度不同。

数据集创建

数据集灵感来源于Codenames游戏，通过支付Amazon Mechanical Turk Workers参与游戏来收集数据。

使用数据集的考虑

所有关联均由人工注释者获得。

搜集汇总

数据集介绍

构建方式

WinoGAViL数据集的构建灵感源自经典桌游《Codenames》，通过众包游戏化方式收集视觉与语言之间的关联。在游戏过程中，一名“间谍大师”需给出一个文本线索，该线索与若干视觉候选项相关联，另一名玩家则需从中识别出正确的图像。参与者被激励去创建既能挑战AI模型、又能被人类玩家破解的关联。所有数据均经由亚马逊Mechanical Turk平台上的众包工作者标注，最终形成了一个包含约数万条样本的测试集。

特点

该数据集的核心特点在于其难度设计：人类玩家在关联识别任务上的Jaccard指数超过90%，而当前最优的视觉语言模型ViLT仅能达到52%的准确率，且仅在线索具有显著视觉特征时表现较好。数据集包含不同数量的候选项（5至12个），对应不同的随机基线水平（从38%降至19%）。此外，每条样本记录了线索、关联图像、欺骗AI的得分、求解者Jaccard均值与标准差等丰富字段，便于多维度分析模型的推理能力。

使用方法

WinoGAViL主要作为测试集使用，官方提供了单一的TEST划分。研究者可基于该数据集评估视觉语言模型的常识推理能力，具体任务为：给定一组图像、一个线索及数量K，模型需选出最符合关联的K张图像。官方提供了基于CLIP模型的Colab评估代码，便于快速复现基线结果。数据集的官方排行榜和论文代码仓库提供了详细的评估流程与训练集采样方法，使用者应遵循CC-BY 4.0许可协议，并仅用于研究目的。

背景与挑战

背景概述

WinoGAViL数据集由Yonatan Bitton及其合作者于2022年提出，旨在评估视觉与语言模型在常识推理方面的能力。该数据集受经典桌游Codenames启发，通过在线游戏形式收集视觉与文本之间的关联，例如将“狼人”与“满月”联系起来。研究团队来自多家知名机构，包括以色列本·古里安大学和北卡罗来纳大学教堂山分校。其核心研究问题在于，当前的视觉-语言模型是否能够像人类一样理解复杂且抽象的常识关联。实验表明，人类在该任务上的表现超过90%的Jaccard指数，而最优模型ViLT仅达到52%的准确率，揭示了现有模型在处理需要背景知识、抽象思维和常识推理的任务时存在显著短板。该数据集已成为评估多模态模型常识推理能力的重要基准。

当前挑战

WinoGAViL数据集所解决的领域问题在于，传统视觉-语言任务（如图像分类或视觉问答）往往依赖浅层视觉特征，而忽略了需要常识和抽象推理的复杂关联。具体挑战包括：第一，模型需要从多个候选图像中选出与给定文本线索最匹配的K张图像，这要求模型具备理解隐喻、文化背景和常识知识的能力；第二，数据集中线索的多样性（如pogonophile与bison、beard的关联）使得模型难以通过简单视觉相似性完成任务。在构建过程中，研究人员面临确保关联既对AI模型具有挑战性又对人类可解的难题，同时需要平衡候选图像数量（5至12张）以控制任务难度。此外，通过众包平台收集数据时，还需设计奖励机制以激励参与者创造高质量且富有创意的关联，这增加了数据收集的复杂性和成本。

常用场景

经典使用场景

WinoGAViL数据集在视觉与语言常识推理领域扮演着标杆性角色，其核心使用场景在于评估多模态模型对抽象语义关联的捕捉能力。该数据集巧妙借鉴了流行桌游“代号”的机制，要求模型从一组图像中精准挑选出与给定文本线索最匹配的K张图片，例如将“狼人”线索与满月图像关联。这一任务设计不仅考验模型对视觉特征的理解，更强调对常识知识、抽象概念及隐喻关系的整合推理，从而成为衡量当前先进视觉-语言模型（如CLIP、ViLT）性能的严峻试金石。

衍生相关工作

WinoGAViL的发布催生了多项富有启发性的衍生研究。原始论文中，研究者系统评估了CLIP、ViLT等主流模型，发现最佳模型ViLT仅达52%的Jaccard指数，与人类超90%的表现存在鸿沟，这一发现直接激发了后续针对多模态模型常识推理能力的改进工作。此后，学界涌现出一系列基于该数据集的扩展研究，包括探索提示工程对模型关联预测的增强效果、设计针对抽象概念的对抗性训练策略，以及将其与视觉问答、图像描述等任务结合以构建复合推理基准。这些工作共同推动了视觉-语言领域从感知智能向认知智能的跨越。

数据集最近研究

最新研究方向

在视觉与语言常识推理的前沿领域，WinoGAViL数据集通过游戏化方式收集多模态关联，为评估AI模型的抽象和常识推理能力提供了全新基准。该数据集模拟“间谍游戏”机制，要求模型从多张图像中选出与文本线索最匹配的候选，其设计巧妙融合了人类协作与对抗性挑战——人类玩家需创作既能迷惑AI但又被其他人类理解的关联。当前研究聚焦于揭示现有视觉语言模型（如CLIP、ViLT）在此任务中的显著短板：人类表现（Jaccard指数>90%）与最优模型（ViLT得分仅52%）间的鸿沟凸显了AI在非视觉显著性线索上的推理缺陷。这一基准的发布推动了多模态常识推理、抽象概念理解及对抗性生成等热点方向的发展，其意义在于系统性地暴露了模型对隐含知识、文化背景和跨模态隐喻的认知局限，为构建更接近人类认知的通用人工智能提供了关键测试场。