MTVQA
收藏魔搭社区2026-01-06 更新2024-08-31 收录
下载链接：
https://modelscope.cn/datasets/ByteDance/MTVQA
下载链接
链接失效反馈官方服务：
资源简介：
# Dataset Card
The dataset is oriented toward visual question answering of multilingual text scenes in nine languages, including Korean, Japanese, Italian, Russian, Deutsch, French, Thai, Arabic, and Vietnamese. The question-answer pairs are labeled by native annotators following a series of rules. A comprehensive description of the dataset can be found in the paper [MTVQA](https://arxiv.org/pdf/2405.11985).
## - Image Distribution
<table style="width:60%;">
<tr>
<td></td>
<td><b>KO</b></td>
<td><b>JA</b></td>
<td><b>IT</b></td>
<td><b>RU</b></td>
<td><b>DE</b></td>
<td><b>FR</b></td>
<td><b>TH</b></td>
<td><b>AR</b></td>
<td><b>VI</b></td>
<td><b>Total</b> </td>
</tr>
<tr>
<td><b>Train Images</b></td>
<td>580</td>
<td>1039</td>
<td>622</td>
<td>635</td>
<td>984</td>
<td>792</td>
<td>319</td>
<td>568</td>
<td>1139</td>
<td>6678 </td>
</tr>
<tr>
<td><b>Test Images</b></td>
<td>250</td>
<td>250</td>
<td>250</td>
<td>250</td>
<td>250</td>
<td>250</td>
<td>116</td>
<td>250</td>
<td>250</td>
<td>2116 </td>
</tr>
<tr>
<td><b>Train QA</b></td>
<td>1280</td>
<td>3332</td>
<td>2168</td>
<td>1835</td>
<td>4238</td>
<td>2743</td>
<td>625</td>
<td>1597</td>
<td>4011</td>
<td>21829 </td>
</tr>
<tr>
<td><b>Test QA</b></td>
<td>558</td>
<td>828</td>
<td>884</td>
<td>756</td>
<td>1048</td>
<td>886</td>
<td>231</td>
<td>703</td>
<td>884</td>
<td>6778</td>
</tr>
</table>
## - LeaderBoard
<table style="width:75%;">
<tr>
<th>Models</th>
<td><b>AR</b></td>
<td><b><b>DE</b></td>
<td><b>FR</b></td>
<td><b>IT</b></td>
<td><b>JA</b></td>
<td><b>KO</b></td>
<td><b>RU</b></td>
<td><b>TH</b></td>
<td><b>VI</b></td>
<td><b>Average</b> </td>
</tr>
<tr>
<th align="left">GPT-4O</th>
<td>20.2 </td>
<td>34.2 </td>
<td>41.2 </td>
<td>32.7 </td>
<td>20.0 </td>
<td>33.9 </td>
<td>11.5 </td>
<td>22.5 </td>
<td>34.2 </td>
<td>27.8 </td>
</tr>
<tr>
<th align="left">Claude3 Opus</th>
<td>15.1 </td>
<td>33.4 </td>
<td>40.6 </td>
<td>34.4 </td>
<td>19.4 </td>
<td>27.2 </td>
<td>13.0 </td>
<td>19.5 </td>
<td>29.1 </td>
<td>25.7 </td>
</tr>
<tr>
<th align="left">Gemini Ultra</th>
<td>14.7 </td>
<td>32.3 </td>
<td>40.0 </td>
<td>31.8 </td>
<td>12.3 </td>
<td>17.2 </td>
<td>11.8 </td>
<td>20.3 </td>
<td>28.6 </td>
<td>23.2 </td>
</tr>
<tr>
<th align="left">GPT-4V</th>
<td>11.5 </td>
<td>31.5 </td>
<td>40.4 </td>
<td>32.3 </td>
<td>11.5 </td>
<td>16.7 </td>
<td>10.3 </td>
<td>15.0 </td>
<td>28.9 </td>
<td>22.0 </td>
</tr>
<tr>
<th align="left">QwenVL Max</th>
<td>7.7 </td>
<td>31.4 </td>
<td>37.6 </td>
<td>30.2 </td>
<td>18.6 </td>
<td>25.4 </td>
<td>10.4 </td>
<td>4.8 </td>
<td>23.5 </td>
<td>21.1 </td>
</tr>
<tr>
<th align="left">Claude3 Sonnet</th>
<td>10.5 </td>
<td>28.9 </td>
<td>35.6 </td>
<td>31.8 </td>
<td>13.9 </td>
<td>22.2 </td>
<td>11.0 </td>
<td>15.2 </td>
<td>20.8 </td>
<td>21.1 </td>
</tr>
<tr>
<th align="left">QwenVL Plus</th>
<td>4.8 </td>
<td>28.8 </td>
<td>33.7 </td>
<td>27.1 </td>
<td>12.8 </td>
<td>19.9 </td>
<td>9.4 </td>
<td>5.6 </td>
<td>18.1 </td>
<td>17.8 </td>
</tr>
<tr>
<th align="left">MiniCPM-Llama3-V-2_5</th>
<td>6.1 </td>
<td>29.6 </td>
<td>35.7 </td>
<td>26.0 </td>
<td>12.1 </td>
<td>13.1 </td>
<td>5.7 </td>
<td>12.6 </td>
<td>15.3 </td>
<td>17.3 </td>
</tr>
<tr>
<th align="left">InternVL-V1.5</th>
<td>3.4 </td>
<td>27.1 </td>
<td>31.4 </td>
<td>27.1 </td>
<td>9.9 </td>
<td>9.0 </td>
<td>4.9 </td>
<td>8.7 </td>
<td>12.4 </td>
<td>14.9 </td>
</tr>
<tr>
<th align="left">GLM4V</th>
<td>0.3 </td>
<td>30.0 </td>
<td>34.1 </td>
<td>30.1 </td>
<td>3.4 </td>
<td>5.7 </td>
<td>3.0 </td>
<td>3.5 </td>
<td>12.3 </td>
<td>13.6 </td>
</tr>
<tr>
<th align="left">TextSquare</th>
<td>3.7 </td>
<td>27.0 </td>
<td>30.8 </td>
<td>26.7 </td>
<td>3.2 </td>
<td>7.2 </td>
<td>6.7 </td>
<td>5.2 </td>
<td>12.4 </td>
<td>13.6 </td>
</tr>
<tr>
<th align="left">Mini-Gemini-HD-34B</th>
<td>2.2 </td>
<td>25.0 </td>
<td>29.2 </td>
<td>25.5 </td>
<td>6.1 </td>
<td>8.6 </td>
<td>4.1 </td>
<td>4.3 </td>
<td>11.8 </td>
<td>13.0 </td>
</tr>
<tr>
<th align="left">InternLM-Xcomposer2-4KHD</th>
<td>2.0 </td>
<td>20.6 </td>
<td>23.2 </td>
<td>21.6 </td>
<td>5.6 </td>
<td>7.7 </td>
<td>4.1 </td>
<td>6.1 </td>
<td>10.1 </td>
<td>11.2 </td>
</tr>
<tr>
<th align="left">Llava-Next-34B</th>
<td>3.3 </td>
<td>24.0 </td>
<td>28.0 </td>
<td>22.3 </td>
<td>3.6 </td>
<td>6.1 </td>
<td>2.6 </td>
<td>0.4 </td>
<td>9.8 </td>
<td>11.1 </td>
</tr>
<tr>
<th align="left">TextMonkey</th>
<td>2.0 </td>
<td>18.1 </td>
<td>19.9 </td>
<td>22.1 </td>
<td>4.6 </td>
<td>7.2 </td>
<td>3.2 </td>
<td>0.9 </td>
<td>11.1 </td>
<td>9.9 </td>
</tr>
<tr>
<th align="left">MiniCPM-V-2</th>
<td>1.3 </td>
<td>12.7 </td>
<td>14.9 </td>
<td>17.0 </td>
<td>3.7 </td>
<td>5.6 </td>
<td>2.2 </td>
<td>2.2 </td>
<td>6.8 </td>
<td>7.4 </td>
</tr>
<tr>
<th align="left">mPLUG-DocOwl 1.5</th>
<td>1.0 </td>
<td>13.9 </td>
<td>14.9 </td>
<td>18.2 </td>
<td>2.9 </td>
<td>5.0 </td>
<td>2.0 </td>
<td>0.9 </td>
<td>6.4 </td>
<td>7.2 </td>
</tr>
<tr>
<th align="left">YI-VL-34B</th>
<td>1.7 </td>
<td>13.5 </td>
<td>15.7 </td>
<td>12.1 </td>
<td>4.8 </td>
<td>5.2 </td>
<td>0.8 </td>
<td>3.5 </td>
<td>4.1 </td>
<td>6.8 </td>
</tr>
<tr>
<th align="left">DeepSeek-VL</th>
<td>0.6 </td>
<td>14.2 </td>
<td>15.3 </td>
<td>15.2 </td>
<td>2.9 </td>
<td>3.8 </td>
<td>1.6 </td>
<td>0.9 </td>
<td>5.2 </td>
<td>6.6 </td>
</tr>
</table>
## - Direct usage
The data is designed to evaluate and enhance the multilingual textual vqa capabilities of multimodal models in the hope of facilitating the understanding of multilingual images, enabling AI to reach more people in the world.
### -- Huggingface dataloader
```
from datasets import load_dataset
dataset = load_dataset("ByteDance/MTVQA")
```
## - Out-of-Scope usage
Academic use only, not supported for commercial usage.
## - Ethics Assessment
Both GPT4V and manual assessment are employed to filter out unethical question and answer pairs.
## - Bias, Risks, and Limitations
Your access to and use of this dataset are at your own risk. We do not guarantee the accuracy of this dataset. The dataset is provided “as is” and we make no warranty or representation to you with respect to it and we expressly disclaim, and hereby expressly waive, all warranties, express, implied, statutory or otherwise. This includes, without limitation, warranties of quality, performance, merchantability or fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. In no event will we be liable to you on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this public license or use of the licensed material. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.
## - Citation
```
@misc{tang2024mtvqa,
title={MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering},
author={Jingqun Tang and Qi Liu and Yongjie Ye and Jinghui Lu and Shu Wei and Chunhui Lin and Wanqing Li and Mohamad Fitri Faiz Bin Mahmood and Hao Feng and Zhen Zhao and Yanjie Wang and Yuliang Liu and Hao Liu and Xiang Bai and Can Huang},
year={2024},
eprint={2405.11985},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
# 数据集卡片
本数据集面向**多语言文本场景视觉问答（Visual Question Answering, VQA）**任务，涵盖9种语言，分别为韩语（Korean）、日语（Japanese）、意大利语（Italian）、俄语（Russian）、德语（Deutsch）、法语（French）、泰语（Thai）、阿拉伯语（Arabic）与越南语（Vietnamese）。所有问答对均由本土标注人员遵循一系列标注规范进行标注。本数据集的详细描述可参阅论文《MTVQA》（链接：https://arxiv.org/pdf/2405.11985）。
## - 图像分布
| 数据集类型 | 韩语（KO） | 日语（JA） | 意大利语（IT） | 俄语（RU） | 德语（DE） | 法语（FR） | 泰语（TH） | 阿拉伯语（AR） | 越南语（VI） | 总计 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 训练集图像 | 580 | 1039 | 622 | 635 | 984 | 792 | 319 | 568 | 1139 | 6678 |
| 测试集图像 | 250 | 250 | 250 | 250 | 250 | 250 | 116 | 250 | 250 | 2116 |
| 训练集问答对 | 1280 | 3332 | 2168 | 1835 | 4238 | 2743 | 625 | 1597 | 4011 | 21829 |
| 测试集问答对 | 558 | 828 | 884 | 756 | 1048 | 886 | 231 | 703 | 884 | 6778 |
## - 排行榜
| 模型 | 阿拉伯语（AR） | 德语（DE） | 法语（FR） | 意大利语（IT） | 日语（JA） | 韩语（KO） | 俄语（RU） | 泰语（TH） | 越南语（VI） | 平均分 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4O | 20.2 | 34.2 | 41.2 | 32.7 | 20.0 | 33.9 | 11.5 | 22.5 | 34.2 | 27.8 |
| Claude 3 Opus | 15.1 | 33.4 | 40.6 | 34.4 | 19.4 | 27.2 | 13.0 | 19.5 | 29.1 | 25.7 |
| Gemini Ultra | 14.7 | 32.3 | 40.0 | 31.8 | 12.3 | 17.2 | 11.8 | 20.3 | 28.6 | 23.2 |
| GPT-4V | 11.5 | 31.5 | 40.4 | 32.3 | 11.5 | 16.7 | 10.3 | 15.0 | 28.9 | 22.0 |
| QwenVL Max | 7.7 | 31.4 | 37.6 | 30.2 | 18.6 | 25.4 | 10.4 | 4.8 | 23.5 | 21.1 |
| Claude 3 Sonnet | 10.5 | 28.9 | 35.6 | 31.8 | 13.9 | 22.2 | 11.0 | 15.2 | 20.8 | 21.1 |
| QwenVL Plus | 4.8 | 28.8 | 33.7 | 27.1 | 12.8 | 19.9 | 9.4 | 5.6 | 18.1 | 17.8 |
| MiniCPM-Llama3-V-2_5 | 6.1 | 29.6 | 35.7 | 26.0 | 12.1 | 13.1 | 5.7 | 12.6 | 15.3 | 17.3 |
| InternVL-V1.5 | 3.4 | 27.1 | 31.4 | 27.1 | 9.9 | 9.0 | 4.9 | 8.7 | 12.4 | 14.9 |
| GLM4V | 0.3 | 30.0 | 34.1 | 30.1 | 3.4 | 5.7 | 3.0 | 3.5 | 12.3 | 13.6 |
| TextSquare | 3.7 | 27.0 | 30.8 | 26.7 | 3.2 | 7.2 | 6.7 | 5.2 | 12.4 | 13.6 |
| Mini-Gemini-HD-34B | 2.2 | 25.0 | 29.2 | 25.5 | 6.1 | 8.6 | 4.1 | 4.3 | 11.8 | 13.0 |
| InternLM-Xcomposer2-4KHD | 2.0 | 20.6 | 23.2 | 21.6 | 5.6 | 7.7 | 4.1 | 6.1 | 10.1 | 11.2 |
| Llava-Next-34B | 3.3 | 24.0 | 28.0 | 22.3 | 3.6 | 6.1 | 2.6 | 0.4 | 9.8 | 11.1 |
| TextMonkey | 2.0 | 18.1 | 19.9 | 22.1 | 4.6 | 7.2 | 3.2 | 0.9 | 11.1 | 9.9 |
| MiniCPM-V-2 | 1.3 | 12.7 | 14.9 | 17.0 | 3.7 | 5.6 | 2.2 | 2.2 | 6.8 | 7.4 |
| mPLUG-DocOwl 1.5 | 1.0 | 13.9 | 14.9 | 18.2 | 2.9 | 5.0 | 2.0 | 0.9 | 6.4 | 7.2 |
| YI-VL-34B | 1.7 | 13.5 | 15.7 | 12.1 | 4.8 | 5.2 | 0.8 | 3.5 | 4.1 | 6.8 |
| DeepSeek-VL | 0.6 | 14.2 | 15.3 | 15.2 | 2.9 | 3.8 | 1.6 | 0.9 | 5.2 | 6.6 |
## - 直接使用方式
本数据集旨在评估并提升多模态模型的多语言文本视觉问答能力，以期推动多语言图像理解研究，助力人工智能惠及全球更多人群。
### -- Hugging Face 数据加载器
from datasets import load_dataset
dataset = load_dataset("ByteDance/MTVQA")
## - 非允许使用范围
仅可用于学术研究，不支持商业用途。
## - 伦理评估
本数据集通过GPT4V与人工评估两种方式，过滤不符合伦理规范的问答对。
## - 偏倚、风险与局限性
您对本数据集的访问与使用均由您自行承担风险。我们不对本数据集的准确性作出任何保证。本数据集按“现状”提供，我们未就其向您作出任何明示或默示的担保、法定担保或其他任何形式的担保，且特此明确放弃并免除所有此类担保，包括但不限于关于质量、性能、适销性、特定用途适用性、不侵权、无潜在或其他缺陷、准确性，以及是否存在已知或可发现的错误的担保。在任何情况下，无论基于何种法律理论（包括但不限于过失）或其他理由，我们均不对因本公开许可或使用许可材料而产生的任何直接、特殊、间接、继发性、惩罚性、惩戒性或其他损失、成本、费用或损害承担责任。上文所述的担保免责与责任限制条款，应在最大可能的范围内被解释为近乎绝对的免责与责任免除。
## - 引用格式
@misc{tang2024mtvqa,
title={MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering},
author={Jingqun Tang and Qi Liu and Yongjie Ye and Jinghui Lu and Shu Wei and Chunhui Lin and Wanqing Li and Mohamad Fitri Faiz Bin Mahmood and Hao Feng and Zhen Zhao and Yanjie Wang and Yuliang Liu and Hao Liu and Xiang Bai and Can Huang},
year={2024},
eprint={2405.11985},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
提供机构：
maas创建时间：
2024-08-01
搜集汇总
数据集介绍
以上内容由遇见数据集搜集并总结生成
