llm-jp/JAMMEval

--- dataset_info: - config_name: CC-OCR-JA-Refined features: - name: original_id dtype: int64 - name: image dtype: image - name: question dtype: string - name: answer dtype: string - name: ocr dtype: string - name: image_name dtype: string splits: - name: test num_bytes: 71840923 num_examples: 145 download_size: 71822359 dataset_size: 71840923 - config_name: CVQA-JA-Refined features: - name: image dtype: image - name: original_id dtype: string - name: question dtype: string - name: options list: string - name: answer dtype: int64 - name: Category dtype: string - name: Image Type dtype: string - name: Image Source dtype: string - name: License dtype: string splits: - name: test num_bytes: 62737078 num_examples: 200 download_size: 62713215 dataset_size: 62737078 - config_name: Heron-Bench-Refined features: - name: original_id dtype: int64 - name: image dtype: image - name: image_category dtype: string - name: context dtype: string - name: question dtype: string - name: answer dtype: string splits: - name: test num_bytes: 38067838 num_examples: 88 download_size: 38046258 dataset_size: 38067838 - config_name: JA-Multi-Image-VQA-Refined features: - name: original_id dtype: int64 - name: images list: image - name: question dtype: string - name: answer dtype: string - name: image_urls list: string - name: page_urls list: string splits: - name: test num_bytes: 99461072 num_examples: 53 download_size: 99466318 dataset_size: 99461072 - config_name: JA-VLM-Bench-Refined features: - name: original_id dtype: int64 - name: image dtype: image - name: question dtype: string - name: answer dtype: string - name: image_url dtype: string - name: page_url dtype: string splits: - name: test num_bytes: 6925318 num_examples: 49 download_size: 6917035 dataset_size: 6925318 - config_name: JDocQA-Refined features: - name: original_id dtype: int64 - name: question dtype: string - name: answer dtype: string splits: - name: test num_bytes: 103278 num_examples: 861 download_size: 64671 dataset_size: 103278 - config_name: JGraphQA-Refined features: - name: original_id dtype: int64 - name: type dtype: string - name: question dtype: string - name: answer dtype: string - name: image dtype: image splits: - name: test num_bytes: 33427252 num_examples: 196 download_size: 33423226 dataset_size: 33427252 configs: - config_name: CC-OCR-JA-Refined data_files: - split: test path: CC-OCR-JA-Refined/test-* - config_name: CVQA-JA-Refined data_files: - split: test path: CVQA-JA-Refined/test-* - config_name: Heron-Bench-Refined data_files: - split: test path: Heron-Bench-Refined/test-* - config_name: JA-Multi-Image-VQA-Refined data_files: - split: test path: JA-Multi-Image-VQA-Refined/test-* - config_name: JA-VLM-Bench-Refined data_files: - split: test path: JA-VLM-Bench-Refined/test-* - config_name: JDocQA-Refined data_files: - split: test path: JDocQA-Refined/test-* - config_name: JGraphQA-Refined data_files: - split: test path: JGraphQA-Refined/test-* language: - ja --- <div align="center" style="line-height: 1;"> <h1>JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation</h1> | <a href="https://huggingface.co/datasets/llm-jp/JAMMEval/" target="_blank">🤗 HuggingFace</a> &nbsp;| <a href="https://arxiv.org/abs/2604.00909" target="_blank">📄 Paper</a> &nbsp;| <a href="https://github.com/llm-jp/simple-evals-mm" target="_blank">🧑‍💻 Code</a> &nbsp;| <br/> <img src="JAMMEval-pipeline.svg" width="80%"/> </div> ## Overview JAMMEval is a curated benchmark collection for evaluating Vision-Language Models (VLMs) on Japanese Visual Question Answering (VQA) tasks. It is constructed by refining seven existing Japanese VQA evaluation datasets through two rounds of human annotation, with the goal of improving evaluation reliability and quality. **⚠️ This repository is a mirror of the original dataset hosted at https://gitlab.llm-jp.nii.ac.jp/datasets/jammeval** Due to copyright restrictions, images for JDocQA-Refined are hosted only on a domestic server and are not included in this mirror. If you would like to use the complete version of JAMMEval, please access it from the original repository above. ## Included Datasets JAMMEval consists of the following seven refined datasets: - CC-OCR-JA-Refined - CVQA-JA-Refined - Heron-Bench-Refined - JA-Multi-Image-VQA-Refined - JA-VLM-Bench-Refined - JDocQA-Refined - JGraphQA-Refined Each dataset is derived from its original version (without the `-Refined` suffix) through a systematic refinement process. ## Refinement Process The refinement focuses on addressing key issues found in the original datasets: - Ambiguity in questions or answers - Incorrect annotated answers - Questions solvable without visual input (i.e., not requiring the image) Through manual inspection and correction, JAMMEval improves the reliability of VLM evaluation, ensuring that tasks genuinely require multimodal understanding. ## Dataset Usage To use JAMMEval, first download the dataset from: 👉 https://gitlab.llm-jp.nii.ac.jp/datasets/jammeval ```bash git clone https://gitlab.llm-jp.nii.ac.jp/datasets/jammeval.git mv jammeval/data . ``` After placing the dataset directory locally (e.g., in a folder named `data/JAMMEval`), you can load each dataset using the 🤗 `datasets` library: ```python from datasets import load_dataset ds = load_dataset("data/JAMMEval/CC-OCR-JA-Refined", split="test") # or ds = load_dataset("data/JAMMEval/Heron-Bench-Refined", split="test") print(ds) ``` Example output: ``` Dataset({ features: ['original_id', 'image', 'question', 'answer', 'ocr', 'image_name'], num_rows: 145 }) ``` ## Evaluation All datasets in JAMMEval can be evaluated using our multimodal evaluation framework: 👉 https://github.com/llm-jp/simple-evals-mm This framework enables consistent evaluation of VLMs across both Japanese and English tasks, facilitating cross-lingual and multimodal benchmarking. ## License Each dataset is derived from its original source dataset and is subject to the license terms of the original dataset. - [CC-OCR](https://arxiv.org/abs/2412.02210) - MIT - [CVQA](https://arxiv.org/abs/2406.05967): - > Note that each question has its own license. All data here is free to use for research purposes, but not every entry is permissible for commercial use. - [Heron-Bench](https://arxiv.org/abs/2404.07824) - > We have collected images that are either in the public domain or licensed under Creative Commons Attribution 1.0 (CC BY 1.0) or Creative Commons Attribution 2.0 (CC BY 2.0). Please refer to the LICENSE.md file for details on the licenses. - [JA-Multi-Image-VQA](https://huggingface.co/datasets/SakanaAI/JA-Multi-Image-VQA) - The images in this dataset are sourced from Unsplash and are free to use under the Unsplash License. They cannot be sold without significant modification and cannot be used to replicate similar or competing services. All other parts of this dataset, excluding the images, are licensed under the Apache 2.0 License. - [JA-VLM-Bench](https://huggingface.co/datasets/SakanaAI/JA-VLM-Bench-In-the-Wild) - > The images in this dataset are sourced from Unsplash and are free to use under the Unsplash License. They cannot be sold without significant modification and cannot be used to replicate similar or competing services. - [JDocQA](https://arxiv.org/abs/2403.19454) - > JDocQA dataset annotations are distributed under CC BY-SA 4.0. We are delighted to see many derivations from JDocQA! When you create any derivations, e.g., datasets, papers, etc, from JDocQA, please cite our paper accordingly. If your derivations are web-based projects, please cite our paper and include the link to this github page. - [JGraphQA](https://huggingface.co/datasets/r-g2-2024/JGraphQA) - License information is not clearly specified. Users should verify the original source before use. ⚠️ Since JAMMEval is a collection of datasets with different licenses, users must check the license of each individual dataset and each data entry (if applicable) before use. In particular, some datasets (e.g., CVQA) may include data that is restricted to non-commercial use. **Note on JDocQA-Refined Images** The images included in JDocQA-Refined must be used in compliance with Japanese copyright law: > "Use is permitted only within the scope defined by Article 30-4 of the Japanese Copyright Act." Users are responsible for ensuring that their use of these images complies with applicable regulations. ## Acknowledgements We sincerely thank the authors and contributors of the original datasets: [CC-OCR](https://arxiv.org/abs/2412.02210), [CVQA](https://arxiv.org/abs/2406.05967), [Heron-Bench](https://arxiv.org/abs/2404.07824), [JA-Multi-Image-VQA](https://huggingface.co/datasets/SakanaAI/JA-Multi-Image-VQA), [JA-VLM-Bench](https://huggingface.co/datasets/SakanaAI/JA-VLM-Bench-In-the-Wild), [JDocQA](https://arxiv.org/abs/2403.19454), [JGraphQA](https://huggingface.co/datasets/r-g2-2024/JGraphQA) Their efforts made this refinement and benchmark construction possible. ## Citation If you find JAMMEval, please consider citing our work and giving the repository a ⭐️ :) ```bibtex @misc{sugiura2026jammevalrefinedcollectionjapanese, title={JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation}, author={Issa Sugiura and Koki Maeda and Shuhei Kurita and Yusuke Oda and Daisuke Kawahara and Naoaki Okazaki}, year={2026}, eprint={2604.00909}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2604.00909}, } ```### 数据集元信息 本数据集包含7个配置项，所有配置仅包含测试集划分： 1. **CC-OCR-JA-Refined** - 特征字段： - `original_id`：64位整数类型的原始ID - `image`：图像类型 - `question`：字符串类型的问题 - `answer`：字符串类型的答案 - `ocr`：光学字符识别（Optical Character Recognition，OCR）文本 - `image_name`：字符串类型的图像名称 - 测试集信息：样本数145，总字节数71840923 - 下载大小：71822359字节，数据集存储大小：71840923字节 2. **CVQA-JA-Refined** - 特征字段： - `image`：图像类型 - `original_id`：字符串类型的原始ID - `question`：字符串类型的问题 - `options`：字符串列表类型的候选选项 - `answer`：64位整数类型的答案索引 - `Category`：字符串类型的任务类别 - `Image Type`：字符串类型的图像类型 - `Image Source`：字符串类型的图像来源 - `License`：字符串类型的许可声明 - 测试集信息：样本数200，总字节数62737078 - 下载大小：62713215字节，数据集存储大小：62737078字节 3. **Heron-Bench-Refined** - 特征字段： - `original_id`：64位整数类型的原始ID - `image`：图像类型 - `image_category`：字符串类型的图像类别 - `context`：字符串类型的上下文文本 - `question`：字符串类型的问题 - `answer`：字符串类型的答案 - 测试集信息：样本数88，总字节数38067838 - 下载大小：38046258字节，数据集存储大小：38067838字节 4. **JA-Multi-Image-VQA-Refined** - 特征字段： - `original_id`：64位整数类型的原始ID - `images`：图像列表类型的多幅输入图像 - `question`：字符串类型的问题 - `answer`：字符串类型的答案 - `image_urls`：字符串列表类型的图像URL - `page_urls`：字符串列表类型的来源页面URL - 测试集信息：样本数53，总字节数99461072 - 下载大小：99466318字节，数据集存储大小：99461072字节 5. **JA-VLM-Bench-Refined** - 特征字段： - `original_id`：64位整数类型的原始ID - `image`：图像类型 - `question`：字符串类型的问题 - `answer`：字符串类型的答案 - `image_url`：字符串类型的图像URL - `page_url`：字符串类型的来源页面URL - 测试集信息：样本数49，总字节数6925318 - 下载大小：6917035字节，数据集存储大小：6925318字节 6. **JDocQA-Refined** - 特征字段： - `original_id`：64位整数类型的原始ID - `question`：字符串类型的问题 - `answer`：字符串类型的答案 - 测试集信息：样本数861，总字节数103278 - 下载大小：64671字节，数据集存储大小：103278字节 7. **JGraphQA-Refined** - 特征字段： - `original_id`：64位整数类型的原始ID - `type`：字符串类型的任务类型 - `question`：字符串类型的问题 - `answer`：字符串类型的答案 - `image`：图像类型 - 测试集信息：样本数196，总字节数33427252 - 下载大小：33423226字节，数据集存储大小：33427252字节 所有配置的数据文件路径均遵循`[配置名称]/test-*`格式，对应测试集划分。本数据集的语言为日语（ja）。 --- <div align="center" style="line-height: 1;"> <h1>JAMMEval：面向可靠视觉语言模型评估的精修日语基准集</h1> | <a href="https://huggingface.co/datasets/llm-jp/JAMMEval/" target="_blank">🤗 HuggingFace</a> &nbsp;| <a href="https://arxiv.org/abs/2604.00909" target="_blank">📄 论文</a> &nbsp;| <a href="https://github.com/llm-jp/simple-evals-mm" target="_blank">🧑‍💻 代码</a> &nbsp;| <br/> <img src="JAMMEval-pipeline.svg" width="80%"/> </div> ## 概述 JAMMEval是一套经过精心整理的基准集合，用于在日语视觉问答（Visual Question Answering，VQA）任务上评估视觉语言模型（Vision-Language Models，VLMs）。本数据集通过两轮人工标注流程，对7个现有日语VQA评估数据集进行系统性精修，旨在提升模型评估的可靠性与结果质量。 **⚠️ 本仓库为原始数据集的镜像，原始数据集托管于 https://gitlab.llm-jp.nii.ac.jp/datasets/jammeval** 受版权法规限制，JDocQA-Refined的图像仅托管于日本国内服务器，未包含在此镜像中。若需使用完整版本的JAMMEval，请通过上述原始仓库访问。 ## 包含数据集 JAMMEval由以下7个精修后的数据集组成： - CC-OCR-JA-Refined - CVQA-JA-Refined - Heron-Bench-Refined - JA-Multi-Image-VQA-Refined - JA-VLM-Bench-Refined - JDocQA-Refined - JGraphQA-Refined 每个数据集均源自其原始版本（即去掉`-Refined`后缀的同名数据集），并经过标准化的精修流程处理。 ## 精修流程 本次精修工作聚焦于解决原始数据集存在的三类核心问题： 1. 问题或答案表述存在歧义 2. 标注答案存在错误 3. 无需依赖视觉输入即可解答的问题（即任务无需多模态理解） 通过人工校验与修正，JAMMEval提升了VLM评估的可靠性，确保所有任务均需真正的多模态理解能力才能完成。 ## 数据集使用方法 若需使用JAMMEval，请首先从以下地址下载完整数据集： 👉 https://gitlab.llm-jp.nii.ac.jp/datasets/jammeval bash git clone https://gitlab.llm-jp.nii.ac.jp/datasets/jammeval.git mv jammeval/data . 将数据集目录部署至本地（例如放置在名为`data/JAMMEval`的文件夹中）后，可通过🤗 `datasets`库加载指定数据集： python from datasets import load_dataset ds = load_dataset("data/JAMMEval/CC-OCR-JA-Refined", split="test") # 或 ds = load_dataset("data/JAMMEval/Heron-Bench-Refined", split="test") print(ds) 示例输出： Dataset({ features: ['original_id', 'image', 'question', 'answer', 'ocr', 'image_name'], num_rows: 145 }) ## 评估方法 JAMMEval中的所有数据集均可通过我们开源的多模态评估框架进行标准化评估： 👉 https://github.com/llm-jp/simple-evals-mm 该框架支持在日语与英语任务上对VLMs进行一致性评估，助力跨语言多模态基准测试研究。 ## 许可声明 各数据集均源自其原始来源数据集，并受对应原始数据集的许可条款约束： - [CC-OCR](https://arxiv.org/abs/2412.02210) - MIT许可证 - [CVQA](https://arxiv.org/abs/2406.05967): - > 请注意，每个问题均拥有独立的许可条款。此处的所有数据均可免费用于研究用途，但并非所有条目均可用于商业用途。 - [Heron-Bench](https://arxiv.org/abs/2404.07824) - > 我们收集的图像均属于公有领域，或采用知识共享署名1.0（CC BY 1.0）、知识共享署名2.0（CC BY 2.0）许可。有关许可的详细信息，请参阅项目根目录下的`LICENSE.md`文件。 - [JA-Multi-Image-VQA](https://huggingface.co/datasets/SakanaAI/JA-Multi-Image-VQA) - 本数据集内的图像源自Unsplash平台，可根据Unsplash许可条款免费使用。未经大幅修改不得销售，亦不得用于复制类似或竞争性服务。数据集除图像外的其余部分采用Apache 2.0许可证。 - [JA-VLM-Bench](https://huggingface.co/datasets/SakanaAI/JA-VLM-Bench-In-the-Wild) - > 本数据集内的图像源自Unsplash平台，可根据Unsplash许可条款免费使用。未经大幅修改不得销售，亦不得用于复制类似或竞争性服务。 - [JDocQA](https://arxiv.org/abs/2403.19454) - > JDocQA数据集的标注内容采用CC BY-SA 4.0许可证分发。我们乐见基于JDocQA的各类衍生成果！若您基于JDocQA创建任何衍生内容（如数据集、论文等），请务必引用我们的论文。若您的衍生项目为网页类项目，请同时引用本论文并附上本GitHub页面的链接。 - [JGraphQA](https://huggingface.co/datasets/r-g2-2024/JGraphQA) - 未明确指定许可信息。使用者在使用前应核实原始来源的许可条款。 ⚠️ 由于JAMMEval是包含多许可数据集的集合，使用者在使用前必须检查每个单独数据集及各数据条目（如适用）的许可条款。特别地，部分数据集（如CVQA）可能包含仅可用于非商业用途的数据。 **关于JDocQA-Refined图像的特别说明** JDocQA-Refined中包含的图像必须符合日本版权法的规定： > "仅可在日本《著作权法》第30条之4定义的范围内使用。" 使用者需自行确保其对这些图像的使用符合相关法规要求。 ## 致谢 我们衷心感谢原始数据集的作者与贡献者： [CC-OCR](https://arxiv.org/abs/2412.02210)、[CVQA](https://arxiv.org/abs/2406.05967)、[Heron-Bench](https://arxiv.org/abs/2404.07824)、[JA-Multi-Image-VQA](https://huggingface.co/datasets/SakanaAI/JA-Multi-Image-VQA)、[JA-VLM-Bench](https://huggingface.co/datasets/SakanaAI/JA-VLM-Bench-In-the-Wild)、[JDocQA](https://arxiv.org/abs/2403.19454)、[JGraphQA](https://huggingface.co/datasets/r-g2-2024/JGraphQA) 他们的工作为本次精修与基准集构建提供了坚实基础。 ## 引用 若您使用JAMMEval，请考虑引用我们的工作并为该仓库点亮⭐️ :) bibtex @misc{sugiura2026jammevalrefinedcollectionjapanese, title={JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation}, author={Issa Sugiura and Koki Maeda and Shuhei Kurita and Yusuke Oda and Daisuke Kawahara and Naoaki Okazaki}, year={2026}, eprint={2604.00909}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2604.00909}, }