jonathanli/Docmatix

---
language:
- en
license: mit
size_categories:
- 1M<n<10M
task_categories:
- visual-question-answering
pretty_name: Docmatix
tags:
- docvqa
configs:
- config_name: images
data_files:
- split: train
path: data/train-*
- config_name: pdf
data_files:
- split: train
path: pdf/train-*
- config_name: zero-shot-exp
data_files:
- split: train
path: zero-shot-exp/train-*
- split: test
path: zero-shot-exp/test-*
dataset_info:
- config_name: images
features:
- name: images
sequence: image
- name: texts
list:
- name: user
dtype: string
- name: assistant
dtype: string
- name: source
dtype: string
splits:
- name: train
num_bytes: 552957537722.77
num_examples: 1273215
download_size: 159404414330
dataset_size: 552957537722.77
- config_name: pdf
features:
- name: pdf
dtype: binary
- name: texts
list:
- name: user
dtype: string
- name: assistant
dtype: string
- name: source
dtype: string
splits:
- name: train
num_bytes: 458612867150
num_examples: 1273245
download_size: 431829972210
dataset_size: 458612867150
- config_name: zero-shot-exp
features:
- name: images
sequence: image
- name: texts
list:
- name: user
dtype: string
- name: assistant
dtype: string
- name: source
dtype: string
splits:
- name: test
num_bytes: 68900253.0
num_examples: 200
- name: train
num_bytes: 578335690.5
num_examples: 1700
download_size: 642963847
dataset_size: 647235943.5
---

# Dataset Card for Docmatix


## Dataset description

Docmatix is part of the Idefics3 release (stay tuned).

It is a massive dataset for Document Visual Question Answering that was used for the fine-tuning of the vision-language model Idefics3.

## Load the dataset

To load the dataset, install the library `datasets` with `pip install datasets`. Then,
```
from datasets import load_dataset
ds = load_dataset("HuggingFaceM4/Docmatix")
```

If you want the dataset to link to the pdf files as binaries instead of the images, do:
```
from datasets import load_dataset
ds = load_dataset("HuggingFaceM4/Docmatix", "pdf")
```

## Data fields

An example of a sample looks as follows:
```
{
"images" = [PIL.Image]
"texts" = [
{ "user": "What is the purpose of the Confirmation Statement mentioned in the document?",
"assistant": "The purpose of the Confirmation Statement is to confirm that all information required to be delivered by the company to the registrar in relation to the confirmation period concerned has been delivered or is being delivered at the same time as the confirmation statement.",
"source": "PDFA key: 244" },
{ "user": "When was the filing received as per the document?",
"assistant": "The filing was received for filing in Electronic Format on the 23/03/2021.",
"source": "PDFA key: 244" },
]
}
```

In `images`, there is a list of up to 4 images, to be placed before the text.
In `texts`, there is a conversation between a user and an assistant about the images that is represented by a list of turns.

## Comparison to other DocVQA datasets

| Dataset | # images | # Q/A pairs | # tokens |
|----------------------|----------|-------------|------------|
| *Document visual question answering* |
| **Docmatix** | **2,444,750**| **9,500,000** | **390,000,000**|
| DocVQA | 10,189 | 39,463 | 337,829 |
| TextCaps | 21,953 | 21,953 | 389,658 |
| TextVQA | 21,953 | 34,602 | 181,918 |
| ST-VQA | 17,247 | 23,121 | 127,846 |
| OCR-VQA | 165,746 | 801,579 | 6,073,824 |
| VisualMRC | 3,027 | 11,988 | 168,828 |
| IAM | 5,663 | 5,663 | 144,216 |
| InfoVQA | 2,118 | 10,074 | 61,048 |
| Diagram image-to-text| 300 | 300 | 22,196 |

# Citation

**BibTeX:**

```bibtex
@misc{laurençon2024building,
title={Building and better understanding vision-language models: insights and future directions.},
author={Hugo Laurençon and Andrés Marafioti and Victor Sanh and Léo Tronchon},
year={2024},
eprint={2408.12637},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```