deepvk/GQA-ru

---
license: apache-2.0
dataset_info:
- config_name: testdev_balanced_images
features:
- name: id
dtype: string
- name: image
dtype: image
splits:
- name: testdev
num_examples: 398
- config_name: testdev_balanced_instructions
features:
- name: id
dtype: string
- name: imageId
dtype: string
- name: question
dtype: string
- name: answer
dtype: string
- name: fullAnswer
dtype: string
- name: isBalanced
dtype: bool
- name: groups
struct:
- name: global
dtype: string
- name: local
dtype: string
- name: entailed
dtype: string
- name: equivalent
dtype: string
- name: types
struct:
- name: structural
dtype: string
- name: semantic
dtype: string
- name: detailed
dtype: string
- name: annotations
sequence:
- name: question
struct:
- name: objectId
dtype: string
- name: value
dtype: string
- name: answer
struct:
- name: objectId
dtype: string
- name: value
dtype: string
- name: fullAnswer
struct:
- name: objectId
dtype: string
- name: value
dtype: string
- name: semantic
list:
- name: operation
dtype: string
- name: argument
dtype: string
- name: dependencies
sequence: int32
- name: semanticStr
dtype: string
splits:
- name: testdev
num_examples: 12216
- config_name: train_balanced_images
features:
- name: id
dtype: string
- name: image
dtype: image
splits:
- name: train
num_examples: 27519
- config_name: train_balanced_instructions
features:
- name: id
dtype: string
- name: imageId
dtype: string
- name: question
dtype: string
- name: answer
dtype: string
- name: fullAnswer
dtype: string
- name: isBalanced
dtype: bool
- name: groups
struct:
- name: global
dtype: string
- name: local
dtype: string
- name: entailed
dtype: string
- name: equivalent
dtype: string
- name: types
struct:
- name: structural
dtype: string
- name: semantic
dtype: string
- name: detailed
dtype: string
- name: annotations
sequence:
- name: question
struct:
- name: objectId
dtype: string
- name: value
dtype: string
- name: answer
struct:
- name: objectId
dtype: string
- name: value
dtype: string
- name: fullAnswer
struct:
- name: objectId
dtype: string
- name: value
dtype: string
- name: semantic
list:
- name: operation
dtype: string
- name: argument
dtype: string
- name: dependencies
sequence: int32
- name: semanticStr
dtype: string
splits:
- name: train
num_examples: 40000
configs:
- config_name: testdev_balanced_images
data_files:
- split: testdev
path: testdev_balanced_images/testdev-*
- config_name: testdev_balanced_instructions
data_files:
- split: testdev
path: testdev_balanced_instructions/testdev-*
- config_name: train_balanced_images
data_files:
- split: train
path: train_balanced_images/train-*
- config_name: train_balanced_instructions
data_files:
- split: train
path: train_balanced_instructions/train-*
task_categories:
- visual-question-answering
language:
- ru
size_categories:
- 10K<n<100K
---

# GQA-ru

This is a translated version of original [GQA](https://cs.stanford.edu/people/dorarad/gqa/about.html) dataset and
stored in format supported for [`lmms-eval`](https://github.com/EvolvingLMMs-Lab/lmms-eval) pipeline.

For this dataset, we:
1. Translate the original one with `gpt-4-turbo`
2. Filter out unsuccessful translations, i.e. where the model protection was triggered
3. Manually validate most common errors

## Dataset Structure

Dataset includes both train and test splits translated from original `train_balanced` and `testdev_balanced`.

Train split includes 27519 images with 40000 questions to them and test split contains 398 images with 12216 different question to them.

Storage format is similar to [`lmms-lab/GQA`](https://huggingface.co/datasets/lmms-lab/GQA), key fields:
* `id`: ID of a question
* `imageId`: ID of an image (images stored in a separate table)
* `question`: text of a question
* `answer`: one word answer
* `fullAnswer`: detailed answer

## Usage

The easiest way to evaluate model on `GQA-ru` is through [`lmms-eval`](https://github.com/EvolvingLMMs-Lab/lmms-eval)

For example, to evaluate [`deepvk/llava-saiga-8b`](https://huggingface.co/deepvk/llava-saiga-8b):
```bash
accelerate launch -m lmms_eval --model llava_hf \
--model_args pretrained="deepvk/llava-saiga-8b" \
--tasks gqa-ru --batch_size 1 \
--log_samples --log_samples_suffix llava-saiga-8b --output_path ./logs/
```

This would print a table with a result, the main metric for this task is `ExactMatch` for one word answer -- whether generated word is completely similar to ground truth.

## Citation

```
@inproceedings{hudson2019gqa,
title={Gqa: A new dataset for real-world visual reasoning and compositional question answering},
author={Hudson, Drew A and Manning, Christopher D},
booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
pages={6700--6709},
year={2019}
}
```
```
@misc{deepvk2024gqa_ru,
title={GQA-ru},
author={Belopolskih, Daniil and Spirin, Egor},
url={https://huggingface.co/datasets/deepvk/GQA-ru},
publisher={Hugging Face}
year={2024},
}
```

原始信息汇总

数据集概述

数据集信息

• 许可证: Apache-2.0

数据集配置

配置一: testdev_balanced_images

• 特征:
  • id: 数据类型为字符串
  • image: 数据类型为图像
• 分割:
  • testdev: 包含398个样本

配置二: testdev_balanced_instructions

• 特征:
  • id: 数据类型为字符串
  • imageId: 数据类型为字符串
  • question: 数据类型为字符串
  • answer: 数据类型为字符串
  • fullAnswer: 数据类型为字符串
  • isBalanced: 数据类型为布尔值
  • groups: 结构类型，包含global和local，均为字符串类型
  • entailed: 数据类型为字符串
  • equivalent: 数据类型为字符串
  • types: 结构类型，包含structural, semantic, detailed，均为字符串类型
  • annotations: 序列类型，包含question, answer, fullAnswer的结构，每个结构包含objectId和value，均为字符串类型
  • semantic: 列表类型，包含operation, argument, dependencies，其中dependencies为整数序列
  • semanticStr: 数据类型为字符串
• 分割:
  • testdev: 包含12216个样本

数据文件

• 配置一:
  • split: testdev
  • path: testdev_balanced_images/testdev-*
• 配置二:
  • split: testdev
  • path: testdev_balanced_instructions/testdev-*

任务类别

• visual-question-answering

语言

• ru

大小类别

• 10K<n<100K