nightingal3/fig-qa

---
annotations_creators:
- expert-generated
- crowdsourced
language_creators:
- crowdsourced
language:
- en
license:
- mit
multilinguality:
- monolingual
pretty_name: Fig-QA
size_categories:
- 10K<n<100K
source_datasets:
- original
task_categories:
- multiple-choice
task_ids:
- multiple-choice-qa
---

# Dataset Card for Fig-QA

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Splits](#data-splits)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Discussion of Biases](#discussion-of-biases)
- [Additional Information](#additional-information)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Description

- **Repository:** https://github.com/nightingal3/Fig-QA
- **Paper:** https://arxiv.org/abs/2204.12632
- **Leaderboard:** https://explainaboard.inspiredco.ai/leaderboards?dataset=fig_qa
- **Point of Contact:** [e-mail]

### Dataset Summary

This is the dataset for the paper [Testing the Ability of Language Models to Interpret Figurative Language](https://arxiv.org/abs/2204.12632). Fig-QA consists of 10256 examples of human-written creative metaphors that are paired as a Winograd schema. It can be used to evaluate the commonsense reasoning of models. The metaphors themselves can also be used as training data for other tasks, such as metaphor detection or generation.

### Supported Tasks and Leaderboards

You can evaluate your models on the test set by submitting to the [leaderboard](https://explainaboard.inspiredco.ai/leaderboards?dataset=fig_qa) on Explainaboard. Click on "New" and select `qa-multiple-choice` for the task field. Select `accuracy` for the metric. You should upload results in the form of a system output file in JSON or JSONL format.

### Languages

This is the English version. Multilingual version can be found [here](https://huggingface.co/datasets/cmu-lti/multi-figqa).

### Data Splits

Train-{S, M(no suffix), XL}: different training set sizes
Dev
Test (labels not provided for test set)

## Considerations for Using the Data

### Discussion of Biases

These metaphors are human-generated and may contain insults or other explicit content. Authors of the paper manually removed offensive content, but users should keep in mind that some potentially offensive content may remain in the dataset.

## Additional Information

### Licensing Information

MIT License

### Citation Information

If you found the dataset useful, please cite this paper:

@misc{https://doi.org/10.48550/arxiv.2204.12632,
doi = {10.48550/ARXIV.2204.12632},
url = {https://arxiv.org/abs/2204.12632},
author = {Liu, Emmy and Cui, Chen and Zheng, Kenneth and Neubig, Graham},
keywords = {Computation and Language (cs.CL), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Testing the Ability of Language Models to Interpret Figurative Language},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution Share Alike 4.0 International}
}

原始信息汇总

数据集概述

数据集名称

• 名称: Fig-QA

数据集基本信息

• 语言: 英语 (en)
• 许可证: MIT License
• 多语言性: 单语种
• 大小: 10K<n<100K
• 来源: 原始数据
• 任务类别: 多项选择
• 任务ID: 多项选择问答 (multiple-choice-qa)

数据集描述

• 概述: Fig-QA包含10256个人类编写的创意隐喻示例，这些示例作为Winograd模式配对，用于评估模型的常识推理能力。这些隐喻也可用于其他任务，如隐喻检测或生成。
• 支持的任务和排行榜: 可通过提交模型到Explainaboard的排行榜来评估模型在测试集上的表现。

数据集结构

• 数据分割: 训练集（不同大小）、开发集、测试集（测试集标签未提供）

使用数据的考虑

• 偏见讨论: 数据集中的隐喻由人类生成，可能包含侮辱或其他明确内容。虽然论文作者手动移除了攻击性内容，但用户应注意可能仍存在潜在的攻击性内容。

附加信息

• 许可证信息: MIT License
• 引用信息: 如果发现数据集有用，请引用相关论文。