lenagibee/GenDocVQA

--- dataset_info: features: - name: unique_id dtype: int64 - name: image_path dtype: string - name: ocr sequence: - name: text dtype: string - name: bbox sequence: int64 - name: block_id dtype: int64 - name: text_id dtype: int64 - name: par_id dtype: int64 - name: line_id dtype: int64 - name: word_id dtype: int64 - name: question dtype: string - name: answer sequence: string splits: - name: train num_bytes: 11642104684 num_examples: 260814 - name: validation num_bytes: 1324439173 num_examples: 28473 download_size: 13295093966 dataset_size: 12966543857 license: other task_categories: - visual-question-answering language: - en tags: - documents - vqa - generative - document understanding size_categories: - 100K<n<1M --- # GenDocVQA This dataset provides a broad set of documents with questions related to their contents. These questions are non-extractive, meaning that the model, which solves our task should be generative and compute the answers by itself. ## Dataset Details ## Uses ### Direct Use In order to load dataset using following code: ```python ds = datasets.load_dataset('lenagibee/GenDocVQA') ``` ds is a dict consisting from two splits `train` and `validation`. To open the image use following example: ```python from PIL import Image im = Image.open(ds['train'][0]['image_path']) ``` Dataset generator: https://huggingface.co/datasets/lenagibee/GenDocVQA/resolve/main/GenDocVQA.py?download=true ## Dataset Structure All the necessary data is stored in the following archives: * Images: https://huggingface.co/datasets/lenagibee/GenDocVQA/resolve/main/archives/gendocvqa2024_imgs.tar.gz?download=true * OCR: https://huggingface.co/datasets/lenagibee/GenDocVQA/resolve/main/archives/gendocvqa2024_ocr.tar.gz?download=true * Annotations: https://huggingface.co/datasets/lenagibee/GenDocVQA/resolve/main/archives/gendocvqa2024_annotations.tar.gz?download=true Data parsing is already implemented in the attached dataset generator. Images should be processed by the user himself. The train split contains 260814 questions and dev (validation) contains 28473. ### Features of dataset The features of the dataset are the following: ```python features = datasets.Features( { "unique_id": datasets.Value("int64"), "image_path": datasets.Value("string"), "ocr": datasets.Sequence( feature={ 'text': datasets.Value("string"), 'bbox': datasets.Sequence(datasets.Value("int64")), 'block_id': datasets.Value("int64"), 'text_id': datasets.Value("int64"), 'par_id': datasets.Value("int64"), 'line_id': datasets.Value("int64"), 'word_id': datasets.Value("int64") } ), "question": datasets.Value("string"), "answer": datasets.Sequence(datasets.Value("string")), } ``` #### Features description * `unique_id` - integer, an id of a question * `image_path` - string, path to the image for a question (includes downloaded path) * `ocr` - dictionary, containing lists, where each element is an information related to a single word * `text` - string, a word itself * `bbox` - list of 4 integers, a bounding box of the word * `block_id` - integer, an index of the block, where the word is located * `text_id` - integer, an index of the set of paragraphs, where the word is located * `par_id` - integer, an index of the paragraph, where the word is located * `line_id` - integer, an index of the line, where the word is located * `word_id` - integer, an index of the word * `question` - string, containing the question * `answer` - list of strings, containing the answers to the question, can be empty (non-answerable) ### Images Are divided inside the archive into dev and train folders. Just regular images in PNG, JPG formats. You can use any image library to process them. ### OCR Same as the Images are divided into dev and train folders. Represented as JSON files. #### OCR JSON Description It is a list of elements, where each represents an information about the single word extracted by the ABBYY FineReader OCR, and contains fields in following order: 1. `block_id` - integer, an index of the block, where the word is located 2. `text_id` - integer, an index of the set of paragraphs, where the word is located 3. `par_id` - integer, an index of the paragraph, where the word is located 4. `line_id` - integer, an index of the line, where the word is located 5. `word_id` - integer, an index of the word 6. `bbox` - list of 4 integers, a bounding box of the word 7. `text` - string, a word itself ### Annotations dev (validation) and train splits are located in the archive. Question lists are represtened by csv files with following columns: 1. `unique_id` - an id of the question 2. `split` 3. `question` 4. `answer` 5. `image_filename` - a filename of the related image 6. `ocr_filename` - a filename of the json file, containing the related OCR data ## Dataset Creation ### Source Data The data for this dataset was collected from the following datasets: 1. SlideVQA - Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. "A Dataset for Document Visual Question Answering on Multiple Images". In Proc. of AAAI. 2023. 2. PDFVQA - Yihao Ding and Siwen Luo and Hyunsuk Chung and Soyeon Caren Han, PDFVQA: A New Dataset for Real-World VQA on PDF Documents, 2023 3. InfographicsVQA - InfographicVQA, Minesh Mathew and Viraj Bagal and Rubèn Pérez Tito and Dimosthenis Karatzas and Ernest Valveny and C. V Jawahar, 2021 4. TAT-DQA - Towards complex document understanding by discrete reasoning, Zhu, Fengbin and Lei, Wenqiang and Feng, Fuli and Wang, Chao and Zhang, Haozhou and Chua, Tat-Seng, 2022 5. DUDE - Document Understanding Dataset and Evaluation (DUDE), Jordy Van Landeghem and Rubén Tito and Łukasz Borchmann and Michał Pietruszka and Paweł Józiak and Rafał Powalski and Dawid Jurkiewicz and Mickaël Coustaty and Bertrand Ackaert and Ernest Valveny and Matthew Blaschko and Sien Moens and Tomasz Stanisławek, 2023 ### Data Processing The questions from each dataset were filtered by the types of the questions, leaving only non-extractive questions, related to one page. After that the questions were paraphrased. ### Source Data Licenses The dataset adheres to the licenses of its constituents. 1. SlideVQA: https://github.com/nttmdlab-nlp/SlideVQA/blob/main/LICENSE 2. PDFVQA: https://github.com/adlnlp/pdfvqa (Unknown) 3. InfographicsVQA: https://www.docvqa.org/datasets/infographicvqa (Unknown) 4. TAT-DQA: https://nextplusplus.github.io/TAT-DQA/ (CC BY 4.0) 5. DUDE: https://github.com/duchallenge-team/dude/blob/main/LICENSE (GPL 3.0) ## Dataset Card Contact Please feel free to contact in the community page of this dataset or via the Telegram chat of the challenge: https://t.me/gendocvqa2024

原始信息汇总

GenDocVQA 数据集概述

数据集详情

特征

• unique_id: 整数类型，问题的唯一标识符。
• image_path: 字符串类型，问题相关图像的路径。
• ocr: 字典类型，包含与单个单词相关的信息列表。
  • text: 字符串类型，单词本身。
  • bbox: 包含4个整数的列表，单词的边界框。
  • block_id: 整数类型，单词所在块的索引。
  • text_id: 整数类型，单词所在段落集的索引。
  • par_id: 整数类型，单词所在段落的索引。
  • line_id: 整数类型，单词所在行的索引。
  • word_id: 整数类型，单词的索引。
• question: 字符串类型，包含问题。
• answer: 字符串列表，包含问题的答案，可能为空（不可回答）。

数据分割

• train: 包含260814个样本，总字节数为11642104684。
• validation: 包含28473个样本，总字节数为1324439173。

数据集大小

• 下载大小: 13295093966字节
• 数据集大小: 12966543857字节

许可

• 数据集遵循其他许可。

任务类别

• 视觉问答（Visual Question Answering, VQA）

语言

• 英语（en）

标签

• 文档
• VQA
• 生成式
• 文档理解

大小类别

• 100K<n<1M

搜集汇总

数据集介绍

构建方式

GenDocVQA数据集旨在推动文档视觉问答领域的发展，专注于生成式而非抽取式的问答任务。该数据集整合了来自SlideVQA、PDFVQA、InfographicsVQA、TAT-DQA和DUDE五个公开数据集的文档图像与问题，经过严格筛选，仅保留与单页内容相关且答案需模型自行生成而非直接提取的问题。随后，所有保留的问题均经过改写处理，以增强多样性并避免原始数据集的风格偏差。最终，数据集划分为训练集（260,814个样本）和验证集（28,473个样本），每个样本包含唯一标识符、图像路径、OCR信息（包括文本、边界框及层级索引）、问题及答案列表。

特点

GenDocVQA的核心特点在于其非抽取性问题的设计，要求模型具备生成式理解能力，而非简单的文本匹配。每个样本均提供丰富的OCR结构信息，包括块、段落、行和词的层级索引，便于模型细粒度分析文档布局。数据集涵盖多种文档类型，如幻灯片、PDF和图表，增强了泛化性。此外，部分问题的答案列表可能为空，模拟了现实场景中不可回答的情况，提升了任务的挑战性。数据规模超过28万样本，属于中等规模，适合训练和评估端到端的视觉语言模型。

使用方法

使用GenDocVQA时，可通过HuggingFace的`datasets.load_dataset('lenagibee/GenDocVQA')`直接加载，返回包含'train'和'validation'两个分区的字典。图像需用户自行从提供的归档文件中下载，并通过PIL等库打开，路径存储在'image_path'字段。OCR数据以JSON格式提供，需解析为层级结构。数据加载器已集成于数据集生成脚本中，简化了预处理流程。用户可基于'question'字段构建输入，以'answer'作为监督信号，适用于训练生成式VQA模型。建议使用并行加载以加速大规模数据读取。

背景与挑战

背景概述

GenDocVQA数据集由Lena Gibee等人于2024年创建，旨在推动文档视觉问答领域的生成式理解研究。该数据集汇集了来自SlideVQA、PDFVQA、InfographicsVQA、TAT-DQA和DUDE等多个知名数据集中的文档图像，经过精心筛选与改写，构建了包含超过26万条训练样本和2.8万条验证样本的大规模资源。其核心研究问题聚焦于非抽取式文档问答，即模型需基于文档内容自主推理生成答案，而非简单定位文本片段。这一设计挑战了传统视觉问答模型对复杂文档布局、多模态信息融合及深层语义理解的极限，为文档智能领域提供了全新的基准测试平台，对推动生成式文档理解技术的发展具有深远影响。

当前挑战

GenDocVQA面临的核心挑战在于解决非抽取式文档问答的领域难题，即模型需超越简单的文本匹配，具备对文档图像中文字、表格、图表等多元素进行综合推理的能力，这要求模型同时掌握视觉布局分析、光学字符识别（OCR）噪声处理以及生成式语言模型的复杂协同。在数据集构建过程中，挑战尤为突出：首先，从多个源数据中筛选非抽取式问题需要精准的语义分类和重写策略，以确保问题的多样性与逻辑严密性；其次，对原始OCR结果（如ABBYY FineReader提取的块、段落、行等层级信息）的标准化与对齐工作繁琐，需保证与图像坐标系统的精确映射；此外，跨数据集许可协议的兼容性处理及大规模图像存储与加载的效率优化，也构成了工程实现上的显著障碍。

常用场景

经典使用场景

GenDocVQA数据集专为生成式文档视觉问答任务而设计，其核心场景在于评估模型对文档图像中蕴含的复杂语义进行推理与生成式回答的能力。不同于传统的抽取式问答，该数据集中的问题无法直接从文本中定位答案，而需要模型综合理解文档的布局、文本内容以及视觉信息，自主生成合理的回答。这一设定推动了视觉语言模型从简单的信息检索向深层语义推理的演进。

解决学术问题

该数据集有效解决了现有文档VQA任务中过度依赖文本匹配、缺乏生成式推理能力的学术瓶颈。通过整合SlideVQA、PDFVQA、InfographicsVQA等多个来源的文档数据，并精心筛选与改写为非抽取式问题，GenDocVQA为研究者提供了评估模型在抽象推理、多模态融合与零样本生成方面能力的标准化基准。其意义在于促进文档理解领域从判别式范式向生成式范式的转型，推动了更接近人类认知的文档智能研究。

衍生相关工作

GenDocVQA的发布催生了一系列围绕生成式文档理解的前沿工作，包括基于多模态大语言模型的文档问答架构、融合布局感知的视觉Transformer改进方法，以及针对非抽取式问题的对抗训练与数据增强策略。此外，该数据集被广泛用于2024年生成式文档VQA挑战赛，成为评估模型在复杂文档推理任务上鲁棒性与泛化能力的重要基准，进而激发了面向文档智能的预训练与微调范式创新。