lmms-lab/NLVR2

---
dataset_info:
features:
- name: question_id
dtype: string
- name: question
dtype: string
- name: left_image
dtype: image
- name: right_image
dtype: image
- name: answer
dtype: string
- name: writer
dtype: string
- name: synset
dtype: string
- name: query
dtype: string
- name: identifier
dtype: string
- name: extra_validations
dtype: string
- name: left_url
dtype: string
- name: right_url
dtype: string
splits:
- name: balanced_dev
num_bytes: 488008399.0
num_examples: 2300
- name: balanced_test_public
num_bytes: 507142965.656
num_examples: 2316
- name: balanced_test_unseen
num_bytes: 409992570.832
num_examples: 2124
- name: unbalanced_dev
num_bytes: 762577280.968
num_examples: 3562
- name: unbalanced_test_unseen
num_bytes: 804486738.08
num_examples: 3668
- name: unbalanced_test_public
num_bytes: 731416041.32
num_examples: 3536
download_size: 1879819700
dataset_size: 3703623995.856
configs:
- config_name: default
data_files:
- split: balanced_dev
path: data/balanced_dev-*
- split: balanced_test_public
path: data/balanced_test_public-*
- split: balanced_test_unseen
path: data/balanced_test_unseen-*
- split: unbalanced_dev
path: data/unbalanced_dev-*
- split: unbalanced_test_unseen
path: data/unbalanced_test_unseen-*
- split: unbalanced_test_public
path: data/unbalanced_test_public-*
---
# Dataset Card for "nlvr2"

<p align="center" width="100%">
<img src="https://i.postimg.cc/g0QRgMVv/WX20240228-113337-2x.png" width="100%" height="80%">
</p>

# Large-scale Multi-modality Models Evaluation Suite

> Accelerating the development of large-scale multi-modality models (LMMs) with `lmms-eval`

🏠 [Homepage](https://lmms-lab.github.io/) | 📚 [Documentation](docs/README.md) | 🤗 [Huggingface Datasets](https://huggingface.co/lmms-lab)

# This Dataset

This is a formatted version of [NLVR2](https://lil.nlp.cornell.edu/nlvr/). It is used in our `lmms-eval` pipeline to allow for one-click evaluations of large multi-modality models.

```
@inproceedings{suhr2017corpus,
title={A corpus of natural language for visual reasoning},
author={Suhr, Alane and Lewis, Mike and Yeh, James and Artzi, Yoav},
booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
pages={217--223},
year={2017}
}
```

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

原始信息汇总

数据集概述

数据集信息

特征

• question_id: 字符串类型
• question: 字符串类型
• left_image: 图像类型
• right_image: 图像类型
• answer: 字符串类型
• writer: 字符串类型
• synset: 字符串类型
• query: 字符串类型
• identifier: 字符串类型
• extra_validations: 字符串类型
• left_url: 字符串类型
• right_url: 字符串类型

数据分割

• balanced_dev:
  • 字节数: 488008399.0
  • 样本数: 2300
• balanced_test_public:
  • 字节数: 507142965.656
  • 样本数: 2316
• balanced_test_unseen:
  • 字节数: 409992570.832
  • 样本数: 2124
• unbalanced_dev:
  • 字节数: 762577280.968
  • 样本数: 3562
• unbalanced_test_unseen:
  • 字节数: 804486738.08
  • 样本数: 3668
• unbalanced_test_public:
  • 字节数: 731416041.32
  • 样本数: 3536

数据集大小

• 下载大小: 1879819700 字节
• 数据集大小: 3703623995.856 字节

配置

• config_name: default
  • 数据文件:
    • balanced_dev: data/balanced_dev-*
    • balanced_test_public: data/balanced_test_public-*
    • balanced_test_unseen: data/balanced_test_unseen-*
    • unbalanced_dev: data/unbalanced_dev-*
    • unbalanced_test_unseen: data/unbalanced_test_unseen-*
    • unbalanced_test_public: data/unbalanced_test_public-*