mair-lab/CulturalVQA

--- dataset_info: features: - name: image dtype: image - name: question dtype: string - name: u_id dtype: string splits: - name: test num_bytes: 547562502.236 num_examples: 2374 download_size: 573979907 dataset_size: 547562502.236 configs: - config_name: default data_files: - split: test path: data/test-* --- ### CulturalVQA Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions - rather than cultural comprehension. We introduce CulturalVQA, a visual question-answering benchmark aimed at assessing VLM's geo-diverse cultural understanding. We curate a diverse collection of 2,378 image-question pairs with 1-5 answers per question representing cultures from 11 countries across 5 continents. The questions probe understanding of various facets of culture such as clothing, food, drinks, rituals, and traditions. > **Note:** The answers for CulturalVQA benchmark is not publicly available. We are working on creating a competition where participants can upload their predictions and evaluate their models. Stay tuned for more updates! If you need to urgently need to evaluate please contact [e-mail] ### Loading the dataset To load and use the CulturalVQA benchmark, use the following commands: ``` from datasets import load_dataset culturalvqa_dataset = load_dataset('mair-lab/CulturalVQA') ``` Once the dataset is loaded each instance contains the following fields: - `u_id`: A unique identifier for each image-question pair - `image`: The image data in binary format - `question`: The question pertaining to the image ### Usage and License CulturalVQA is a test-only benchmark and can be used to evaluate models. The images are scraped from the internet and are not owned by the authors. All annotations are released under the CC BY-SA 4.0 license. ### Citation Information If you are using this dataset, please cite ``` @inproceedings{nayak-etal-2024-benchmarking, title = "Benchmarking Vision Language Models for Cultural Understanding", author = "Nayak, Shravan and Jain, Kanishk and Awal, Rabiul and Reddy, Siva and Steenkiste, Sjoerd Van and Hendricks, Lisa Anne and Stanczak, Karolina and Agrawal, Aishwarya", editor = "Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung", booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing", month = nov, year = "2024", address = "Miami, Florida, USA", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.emnlp-main.329", pages = "5769--5790" } ```CulturalVQA is a visual question-answering benchmark designed to evaluate Vision Language Models in their ability to understand cross-cultural concepts. It includes 2,378 image-question pairs from 11 countries across 5 continents, covering various cultural elements such as clothing, food, drinks, rituals, and traditions.

搜集汇总

背景与挑战

背景概述

CulturalVQA是一个视觉问答基准测试数据集，包含2,378个图像-问题对，旨在评估视觉语言模型对跨文化知识的理解能力。该数据集覆盖了11个国家、5大洲的文化内容，涉及服装、食物、饮料、仪式和传统等多个方面，每个问题提供1-5个答案，专门用于测试模型的文化理解能力。