Chinese-SimpleVQA

# Overview

<p align="center">
🌐 <a href="https://chinesesimplevqa.github.io/ChieseSimpleVQA.github.io/#leaderboard" target="_blank">Website</a> • 🤗 <a href="https://huggingface.co/datasets/OpenStellarTeam/Chinese-SimpleVQA" target="_blank">Hugging Face</a> • ⏬ <a href="#data" target="_blank">Data</a> • 📃 <a href="https://arxiv.org/pdf/2502.11718" target="_blank">Paper</a><br> <a href="https://github.com/OpenStellarTeam/ChineseSimpleVQA/blob/master/README_zh.md"> 中文</a> | <a href="https://github.com/OpenStellarTeam/ChineseSimpleVQA/blob/master/README.md">English
</p>

## Dataset
1.chinese_simple_vqa.jsonl（the image is in url format）

2.chinese_simplevqa.parquet (the image is in base64 format and can be downloaded)

**Chinese SimpleVQA** is the first factuality-based visual question-answering benchmark in Chinese, aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the **Chinese** language, **diverse** knowledge types, a **multi-hop** question construction, **high-quality** data, **static** consistency, and **easy-to-evaluate** through short answers.

Please visit our [website](https://chinesesimplevqa.github.io/ChieseSimpleVQA.github.io/#leaderboard/) or check our [paper](https://arxiv.org/pdf/2502.11718) for more details.

## 💫 Introduction

* To comprehensively assess the factual knowledge of LVLMs, we present a **ChineseSimpleVQA** benchmark, which consists of a dataset containing 2,200 high-quality questions across 56 topics, spanning from the humanities to science and engineering. Specifically, the key distinguishing features of our proposed ChineseSimpleVQA are as follows:
* **Multi-hop:** Visual factuality inquiries are decomposed into two steps: object recognition and knowledge assessment. This multi-hop strategy allows us to analyze the capability boundaries and execution mechanisms of LVLMs.
* 🍀**Diverse:** ChineseSimpleVQA emphasizes the Chinese language and covers 8 major topics (i.e., ``Nature, Sciences, Engineering, Humanities & Society, modern Architecture, Ancient Architecture, Geography Meteorological ``and ``Life Culture & Art``). These topics encompass 56 fine-grained subtopics.
* ⚡**High-quality:** We implement a rigorous pipeline for the benchmark construction, including automatic verification, difficulty filtering, and human verification.
* 💡**Static:** To maintain the enduring quality of ChineseSimpleVQA, all reference answers will remain unchanged over time.
* 🗂️**Easy-to-evaluate:** All of the questions and answers are in a short format for quick evaluation.

- Based on Chinese SimpleVQA, we have conducted a comprehensive evaluation of the factual capabilities of existing 34 LVLMs. We also maintain a comprehensive leaderboard list.

## 📊 Leaderboard

Please visit our [website](https://chinesesimplevqa.github.io/ChieseSimpleVQA.github.io/#leaderboard/)

## ⚖️ Evals

Please visit our [github](https://github.com/OpenStellarTeam/ChineseSimpleVQA/tree/main)

## Citation

Please cite our paper if you use our dataset.

```
@article{gu2025see,
title={" See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models},
author={Gu, Jihao and Wang, Yingyao and Bu, Pi and Wang, Chen and Wang, Ziming and Song, Tengtao and Wei, Donglai and Yuan, Jiale and Zhao, Yingxiu and He, Yancheng and others},
journal={arXiv preprint arXiv:2502.11718},
year={2025}
}
```

# 概述

<p align="center">
🌐 <a href="https://chinesesimplevqa.github.io/ChieseSimpleVQA.github.io/#leaderboard" target="_blank">官方网站</a> • 🤗 <a href="https://huggingface.co/datasets/OpenStellarTeam/Chinese-SimpleVQA" target="_blank">Hugging Face</a> • ⏬ <a href="#data" target="_blank">数据集</a> • 📃 <a href="https://arxiv.org/pdf/2502.11718" target="_blank">论文</a><br> <a href="https://github.com/OpenStellarTeam/ChineseSimpleVQA/blob/master/README_zh.md">中文文档</a> | <a href="https://github.com/OpenStellarTeam/ChineseSimpleVQA/blob/master/README.md">English 文档</a>
</p>

## 数据集
1. chinese_simple_vqa.jsonl（图片采用URL格式）
2. chinese_simplevqa.parquet（图片采用Base64格式，支持下载）

**中文简单视觉问答（Chinese SimpleVQA）**是首个面向中文的事实性视觉问答基准数据集，旨在评估大视觉语言模型（Large Vision Language Model, LVLM）在8大主题与56个子主题下的视觉事实性理解能力。该基准数据集的核心特性包括：聚焦**中文**语言环境、覆盖**多样**的知识类型、采用**多跳**式问题构建方式、数据质量**上乘**、具备**静态**一致性，且可通过简短答案实现**易评估**性。

如需了解更多细节，请访问我们的[官方网站](https://chinesesimplevqa.github.io/ChieseSimpleVQA.github.io/#leaderboard/)或查阅[论文](https://arxiv.org/pdf/2502.11718)。

## 💫 数据集介绍

为全面评估大视觉语言模型的事实性知识掌握能力，我们构建了**中文简单视觉问答（Chinese SimpleVQA）**基准数据集，该数据集包含覆盖56个主题的2200道高质量问题，主题范畴涵盖人文社科至工程科学领域。具体而言，本数据集的核心差异化特性如下：
* **多跳式设计**：将视觉事实性查询拆解为目标识别与知识评估两个步骤。这种多跳策略便于我们分析大视觉语言模型的能力边界与运行机制。
* 🍀**覆盖广泛**：本数据集聚焦中文语言环境，涵盖8大主题（即自然、科学、工程、人文社科、现代建筑、古代建筑、地理气象以及生命文化与艺术），并细分为56个精细子主题。
* ⚡**数据高质量**：我们构建了严格的基准数据集构建流程，涵盖自动校验、难度筛选与人工审核三个环节。
* 💡**静态一致性**：为保障数据集的长期可用性，所有参考答案将永久保持不变。
* 🗂️**易于评估**：所有问题与答案均采用简短格式，可实现快速评估。

基于中文简单视觉问答数据集，我们对当前已有的34个大视觉语言模型的事实性理解能力开展了全面评估，并维护了完整的排行榜列表。

## 📊 排行榜

请访问我们的[官方网站](https://chinesesimplevqa.github.io/ChieseSimpleVQA.github.io/#leaderboard/)查看完整排行榜。

## ⚖️ 评估方法

请访问我们的[GitHub仓库](https://github.com/OpenStellarTeam/ChineseSimpleVQA/tree/main)了解评估相关细节。

## 引用声明

若您使用本数据集，请引用我们的论文：

@article{gu2025see,
title={" See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models},
author={Gu, Jihao and Wang, Yingyao and Bu, Pi and Wang, Chen and Wang, Ziming and Song, Tengtao and Wei, Donglai and Yuan, Jiale and Zhao, Yingxiu and He, Yancheng and others},
journal={arXiv preprint arXiv:2502.11718},
year={2025}
}

搜集汇总

背景与挑战

背景概述

Chinese-SimpleVQA是一个基于事实的中文视觉问答基准数据集，包含2,200个高质量问题，涵盖8个主要主题和56个子主题。其特点包括多跳问题构建、多样性、高质量数据和易于评估，适用于评估大型视觉语言模型的事实性能力。