imdatta0/MathVista

---
annotations_creators:
- expert-generated
- found
language_creators:
- expert-generated
- found
language:
- en
- zh
- fa
license: cc-by-sa-4.0
multilinguality:
- monolingual
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- multiple-choice
- question-answering
- visual-question-answering
- text-classification
task_ids:
- multiple-choice-qa
- closed-domain-qa
- open-domain-qa
- visual-question-answering
- multi-class-classification
paperswithcode_id: mathvista
pretty_name: MathVista
tags:
- multi-modal-qa
- math-qa
- figure-qa
- geometry-qa
- math-word-problem
- textbook-qa
- vqa
- arithmetic-reasoning
- statistical-reasoning
- algebraic-reasoning
- geometry-reasoning
- numeric-common-sense
- scientific-reasoning
- logical-reasoning
- geometry-diagram
- synthetic-scene
- chart
- plot
- scientific-figure
- table
- function-plot
- abstract-scene
- puzzle-test
- document-image
- medical-image
- mathematics
- science
- chemistry
- biology
- physics
- engineering
- natural-science
configs:
- config_name: default
data_files:
- split: testmini
path: data/testmini-*
- split: test
path: data/test-*
dataset_info:
features:
- name: pid
dtype: string
- name: question
dtype: string
- name: image
dtype: string
- name: decoded_image
dtype: image
- name: choices
sequence: string
- name: unit
dtype: string
- name: precision
dtype: float64
- name: answer
dtype: string
- name: question_type
dtype: string
- name: answer_type
dtype: string
- name: metadata
struct:
- name: category
dtype: string
- name: context
dtype: string
- name: grade
dtype: string
- name: img_height
dtype: int64
- name: img_width
dtype: int64
- name: language
dtype: string
- name: skills
sequence: string
- name: source
dtype: string
- name: split
dtype: string
- name: task
dtype: string
- name: query
dtype: string
splits:
- name: testmini
num_bytes: 142635198.0
num_examples: 1000
- name: test
num_bytes: 648291350.22
num_examples: 5141
download_size: 885819490
dataset_size: 790926548.22
---
# Dataset Card for MathVista

- [Dataset Description](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#dataset-description)
- [Paper Information](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#paper-information)
- [Dataset Examples](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#dataset-examples)
- [Leaderboard](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#leaderboard)
- [Dataset Usage](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#dataset-usage)
- [Data Downloading](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-downloading)
- [Data Format](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-format)
- [Data Visualization](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-visualization)
- [Data Source](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-source)
- [Automatic Evaluation](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#automatic-evaluation)
- [License](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#license)
- [Citation](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#citation)

## Dataset Description

**MathVista** is a consolidated Mathematical reasoning benchmark within Visual contexts. It consists of **three newly created datasets, IQTest, FunctionQA, and PaperQA**, which address the missing visual domains and are tailored to evaluate logical reasoning on puzzle test figures, algebraic reasoning over functional plots, and scientific reasoning with academic paper figures, respectively. It also incorporates **9 MathQA datasets** and **19 VQA datasets** from the literature, which significantly enrich the diversity and complexity of visual perception and mathematical reasoning challenges within our benchmark. In total, **MathVista** includes **6,141 examples** collected from **31 different datasets**.

## Paper Information

- Paper: https://arxiv.org/abs/2310.02255
- Code: https://github.com/lupantech/MathVista
- Project: https://mathvista.github.io/
- Visualization: https://mathvista.github.io/#visualization
- Leaderboard: https://mathvista.github.io/#leaderboard

## Dataset Examples

Examples of our newly annotated datasets: IQTest, FunctionQA, and PaperQA:

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/our_new_3_datasets.png" style="zoom:40%;" />

<details>
<summary>🔍 Click to expand/collapse more examples</summary>

Examples of seven mathematical reasoning skills:

1. Arithmetic Reasoning

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/ari.png" style="zoom:40%;" />

2. Statistical Reasoning

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/sta.png" style="zoom:40%;" />

3. Algebraic Reasoning

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/alg.png" style="zoom:40%;" />

4. Geometry Reasoning

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/geo.png" style="zoom:40%;" />

5. Numeric common sense

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/num.png" style="zoom:40%;" />

6. Scientific Reasoning

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/sci.png" style="zoom:40%;" />

7. Logical Reasoning

<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/log.png" style="zoom:40%;" />

</details>

## Leaderboard

🏆 The leaderboard for the *testmini* set (1,000 examples) is available [here](https://mathvista.github.io/#leaderboard).

🏆 The leaderboard for the *test* set (5,141 examples) and the automatic evaluation on [CodaLab](https://codalab.org/) are under construction.

## Dataset Usage

### Data Downloading

All the data examples were divided into two subsets: *testmini* and *test*.

- **testmini**: 1,000 examples used for model development, validation, or for those with limited computing resources.
- **test**: 5,141 examples for standard evaluation. Notably, the answer labels for test will NOT be publicly released.

You can download this dataset by the following command (make sure that you have installed [Huggingface Datasets](https://huggingface.co/docs/datasets/quickstart)):

```python
from datasets import load_dataset

dataset = load_dataset("AI4Math/MathVista")
```

Here are some examples of how to access the downloaded dataset:

```python
# print the first example on the testmini set
print(dataset["testmini"][0])
print(dataset["testmini"][0]['pid']) # print the problem id
print(dataset["testmini"][0]['question']) # print the question text
print(dataset["testmini"][0]['query']) # print the query text
print(dataset["testmini"][0]['image']) # print the image path
print(dataset["testmini"][0]['answer']) # print the answer
dataset["testmini"][0]['decoded_image'] # display the image

# print the first example on the test set
print(dataset["test"][0])
```

### Data Format

The dataset is provided in json format and contains the following attributes:

```json
{
"question": [string] The question text,
"image": [string] A file path pointing to the associated image,
"choices": [list] Choice options for multiple-choice problems. For free-form problems, this could be a 'none' value,
"unit": [string] The unit associated with the answer, e.g., "m^2", "years". If no unit is relevant, it can be a 'none' value,
"precision": [integer] The number of decimal places the answer should be rounded to,
"answer": [string] The correct answer for the problem,
"question_type": [string] The type of question: "multi_choice" or "free_form",
"answer_type": [string] The format of the answer: "text", "integer", "float", or "list",
"pid": [string] Problem ID, e.g., "1",
"metadata": {
"split": [string] Data split: "testmini" or "test",
"language": [string] Question language: "English", "Chinese", or "Persian",
"img_width": [integer] The width of the associated image in pixels,
"img_height": [integer] The height of the associated image in pixels,
"source": [string] The source dataset from which the problem was taken,
"category": [string] The category of the problem: "math-targeted-vqa" or "general-vqa",
"task": [string] The task of the problem, e.g., "geometry problem solving",
"context": [string] The visual context type of the associated image,
"grade": [string] The grade level of the problem, e.g., "high school",
"skills": [list] A list of mathematical reasoning skills that the problem tests
},
"query": [string] the query text used as input (prompt) for the evaluation model
}
```

### Data Visualization

🎰 You can explore the dataset in an interactive way [here](https://mathvista.github.io/#visualization).

<details>
<summary>Click to expand/collapse the visualization page screeshot.</summary>
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/data_visualizer.png" style="zoom:40%;" />
</details>

### Data Source

The **MathVista** dataset is derived from three newly collected datasets: IQTest, FunctionQA, and Paper, as well as 28 other source datasets. Details can be found in the [source.json](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/source.json) file. All these source datasets have been preprocessed and labeled for evaluation purposes.

### Automatic Evaluation

🔔 To automatically evaluate a model on the dataset, please refer to our GitHub repository [here](https://github.com/lupantech/MathVista/tree/main).

## License

The new contributions to our dataset are distributed under the [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) license, including

- The creation of three datasets: IQTest, FunctionQA, and Paper;
- The filtering and cleaning of source datasets;
- The standard formalization of instances for evaluation purposes;
- The annotations of metadata.

The copyright of the images and the questions belongs to the original authors, and the source of every image and original question can be found in the `metadata` field and in the [source.json](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/source.json) file. Alongside this license, the following conditions apply:

- **Purpose:** The dataset was primarily designed for use as a test set.
- **Commercial Use:** The dataset can be used commercially as a test set, but using it as a training set is prohibited. By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) license.

## Citation

If you use the **MathVista** dataset in your work, please kindly cite the paper using this BibTeX:

```
@inproceedings{lu2024mathvista,
author = {Lu, Pan and Bansal, Hritik and Xia, Tony and Liu, Jiacheng and Li, Chunyuan and Hajishirzi, Hannaneh and Cheng, Hao and Chang, Kai-Wei and Galley, Michel and Gao, Jianfeng},
title = {MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2024}
}
```

annotations_creators:
- 专家生成
- 公开发掘
language_creators:
- 专家生成
- 公开发掘
language:
- 英语（en）
- 中文（zh）
- 波斯语（fa）
license: 知识共享署名-相同方式共享4.0（CC BY-SA 4.0）
multilinguality:
- 单语言（每个样本单一语言）
size_categories:
- 1000 < n < 10000
source_datasets:
- 原始数据集
task_categories:
- 多项选择
- 问答
- 视觉问答（VQA）
- 文本分类
task_ids:
- 多项选择问答
- 封闭域问答
- 开放域问答
- 视觉问答（VQA）
- 多分类
paperswithcode_id: mathvista
pretty_name: MathVista
tags:
- 多模态问答（multi-modal-qa）
- 数学问答（math-qa）
- 图文问答（figure-qa）
- 几何问答（geometry-qa）
- 数学应用题（math-word-problem）
- 教材问答（textbook-qa）
- 视觉问答（VQA）
- 算术推理（arithmetic-reasoning）
- 统计推理（statistical-reasoning）
- 代数推理（algebraic-reasoning）
- 几何推理（geometry-reasoning）
- 数值常识（numeric-common-sense）
- 科学推理（scientific-reasoning）
- 逻辑推理（logical-reasoning）
- 几何图表（geometry-diagram）
- 合成场景（synthetic-scene）
- 图表（chart）
- 函数绘图（plot）
- 科学图表（scientific-figure）
- 表格（table）
- 函数绘图（function-plot）
- 抽象场景（abstract-scene）
- 智力测试（puzzle-test）
- 文档图像（document-image）
- 医学图像（medical-image）
- 数学（mathematics）
- 科学（science）
- 化学（chemistry）
- 生物学（biology）
- 物理学（physics）
- 工程学（engineering）
- 自然科学（natural-science）
configs:
- config_name: default
data_files:
- split: testmini
path: data/testmini-*
- split: test
path: data/test-*
dataset_info:
features:
- name: pid
dtype: 字符串（string）
- name: question
dtype: 字符串（string）
- name: image
dtype: 字符串（string）
- name: decoded_image
dtype: 图像（image）
- name: choices
sequence: 字符串（string）
- name: unit
dtype: 字符串（string）
- name: precision
dtype: 浮点数64位（float64）
- name: answer
dtype: 字符串（string）
- name: question_type
dtype: 字符串（string）
- name: answer_type
dtype: 字符串（string）
- name: metadata
struct:
- name: category
dtype: 字符串（string）
- name: context
dtype: 字符串（string）
- name: grade
dtype: 字符串（string）
- name: img_height
dtype: 整数64位（int64）
- name: img_width
dtype: 整数64位（int64）
- name: language
dtype: 字符串（string）
- name: skills
sequence: 字符串（string）
- name: source
dtype: 字符串（string）
- name: split
dtype: 字符串（string）
- name: task
dtype: 字符串（string）
- name: query
dtype: 字符串（string）
splits:
- name: testmini
num_bytes: 142635198.0
num_examples: 1000
- name: test
num_bytes: 648291350.22
num_examples: 5141
download_size: 885819490
dataset_size: 790926548.22
---
# MathVista 数据集卡片
- [数据集描述](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#dataset-description)
- [论文信息](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#paper-information)
- [数据集示例](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#dataset-examples)
- [排行榜](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#leaderboard)
- [数据集使用](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#dataset-usage)
- [数据下载](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-downloading)
- [数据格式](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-format)
- [数据可视化](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-visualization)
- [数据来源](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#data-source)
- [自动评估](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#automatic-evaluation)
- [许可证](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#license)
- [引用方式](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/README.md#citation)

## 数据集描述
**MathVista** 是一个整合了视觉语境下数学推理任务的基准测试集。它包含三个新建数据集：IQTest、FunctionQA与PaperQA，分别用于评估拼图测试图表的逻辑推理能力、函数绘图相关的代数推理能力，以及学术论文图表对应的科学推理能力，填补了现有视觉数学推理数据集的领域空白。此外，本基准还整合了来自现有文献的9个MathQA类数据集与19个视觉问答（VQA）类数据集，极大丰富了视觉感知与数学推理挑战的多样性与复杂度。总体而言，**MathVista** 共包含来自31个不同数据集的6141个测试样本。

## 论文信息
- 论文：https://arxiv.org/abs/2310.02255
- 代码：https://github.com/lupantech/MathVista
- 项目主页：https://mathvista.github.io/
- 可视化页面：https://mathvista.github.io/#visualization
- 排行榜：https://mathvista.github.io/#leaderboard

## 数据集示例
我们新建标注的三个数据集示例：IQTest、FunctionQA与PaperQA：
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/our_new_3_datasets.png" style="zoom:40%;" />

<details>
<summary>🔍 点击展开/收起更多示例</summary>

七种数学推理技能的示例：
1. 算术推理
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/ari.png" style="zoom:40%;" />

2. 统计推理
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/sta.png" style="zoom:40%;" />

3. 代数推理
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/alg.png" style="zoom:40%;" />

4. 几何推理
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/geo.png" style="zoom:40%;" />

5. 数值常识
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/num.png" style="zoom:40%;" />

6. 科学推理
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/sci.png" style="zoom:40%;" />

7. 逻辑推理
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/log.png" style="zoom:40%;" />

</details>

## 排行榜
🏆 针对*testmini*子集（1000个样本）的排行榜已上线[此处](https://mathvista.github.io/#leaderboard)。
🏆 针对*test*子集（5141个样本）的排行榜以及在[CodaLab](https://codalab.org/)上的自动评估功能正在开发中。

## 数据集使用方法
### 数据下载
所有数据样本分为两个子集：*testmini*与*test*。
- **testmini**：包含1000个样本，可用于模型开发、验证，或供计算资源有限的使用者测试。
- **test**：包含5141个样本，用于标准评估。请注意：*test*子集的答案标签不会公开。

你可以通过以下命令下载该数据集（请确保已安装[Huggingface Datasets](https://huggingface.co/docs/datasets/quickstart)）：
python
from datasets import load_dataset

dataset = load_dataset("AI4Math/MathVista")

以下为访问已下载数据集的示例代码：
python
# 打印 testmini 子集的第一个样本
print(dataset["testmini"][0])
print(dataset["testmini"][0]['pid']) # 打印问题ID
print(dataset["testmini"][0]['question']) # 打印问题文本
print(dataset["testmini"][0]['query']) # 打印查询文本
print(dataset["testmini"][0]['image']) # 打印图像路径
print(dataset["testmini"][0]['answer']) # 打印答案
dataset["testmini"][0]['decoded_image'] # 显示图像

# 打印 test 子集的第一个样本
print(dataset["test"][0])

### 数据格式
本数据集以JSON格式提供，包含以下属性：
json
{
"question": [字符串] 问题文本,
"image": [字符串] 指向关联图像的文件路径,
"choices": [列表] 多项选择题的选项。对于自由作答题型，该字段可设为`none`,
"unit": [字符串] 答案对应的单位，例如`"m^2"`、`"年"`。若无相关单位，可设为`none`,
"precision": [整数] 答案需要保留的小数位数,
"answer": [字符串] 问题的正确答案,
"question_type": [字符串] 问题类型：`"multi_choice"`（多项选择）或`"free_form"`（自由作答）,
"answer_type": [字符串] 答案格式：`"text"`（文本）、`"integer"`（整数）、`"float"`（浮点数）或`"list"`（列表）,
"pid": [字符串] 问题ID，例如`"1"`,
"metadata": {
"split": [字符串] 数据拆分方式：`"testmini"`或`"test"`,
"language": [字符串] 问题语言：`"English"`（英语）、`"Chinese"`（中文）或`"Persian"`（波斯语）,
"img_width": [整数] 关联图像的像素宽度,
"img_height": [整数] 关联图像的像素高度,
"source": [字符串] 该问题所属的源数据集,
"category": [字符串] 问题类别：`"math-targeted-vqa"`（面向数学的视觉问答）或`"general-vqa"`（通用视觉问答）,
"task": [字符串] 问题对应的任务，例如`"geometry problem solving"`（几何解题）,
"context": [字符串] 关联图像的视觉上下文类型,
"grade": [字符串] 问题适用的学段，例如`"high school"`（高中）,
"skills": [列表] 该问题考察的数学推理技能列表
},
"query": [字符串] 用于评估模型的输入查询文本（提示词）
}

### 数据可视化
🎰 你可以通过[此处](https://mathvista.github.io/#visualization)以交互方式探索本数据集。

<details>
<summary>点击展开/收起可视化页面截图</summary>
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/data_visualizer.png" style="zoom:40%;" />
</details>

### 数据来源
**MathVista** 数据集来源于三个新建采集的数据集：IQTest、FunctionQA与PaperQA，以及另外28个源数据集。详细信息可参阅[source.json](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/source.json)文件。所有源数据集均已针对评估任务完成预处理与标注。

### 自动评估
🔔 若需在本数据集上对模型进行自动评估，请参阅我们的GitHub仓库[此处](https://github.com/lupantech/MathVista/tree/main)。

## 许可证
本数据集的新增贡献内容遵循[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)许可协议发布，包括：
- 三个新建数据集（IQTest、FunctionQA与PaperQA）的创建;
- 源数据集的筛选与清洗;
- 针对评估任务的样本标准化规整;
- 元数据的标注。

图像与问题的版权归原作者所有，每张图像与原始问题的来源可在`metadata`字段以及[source.json](https://huggingface.co/datasets/AI4Math/MathVista/blob/main/source.json)文件中查询。除本许可协议外，还需遵守以下条款:
- **使用目的**: 本数据集主要设计用作测试集。
- **商业使用**: 本数据集可作为测试集进行商业使用，但严禁将其用作训练集。访问或使用本数据集即表示您已知晓并同意遵守本条款与[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)许可协议。

## 引用方式
若您在研究中使用**MathVista**数据集，请通过以下BibTeX条目引用该论文:

@inproceedings{lu2024mathvista,
author = {Lu, Pan and Bansal, Hritik and Xia, Tony and Liu, Jiacheng and Li, Chunyuan and Hajishirzi, Hannaneh and Cheng, Hao and Chang, Kai-Wei and Galley, Michel and Gao, Jianfeng},
title = {MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2024}
}