Molmo2-SynMultiImageQA

# Molmo2-SynMultiImageQA
Molmo2-SynMultiImageQA is a collection of synthetic multi-image question-answer pairs about various kinds of text-rich images, including charts, tables, documents, diagrams, etc.

The synthetic data is generated by extending the [CoSyn framework](https://yueyang1996.github.io/cosyn/) into multi-image settings,
with [Claude-sonnet-4-5](https://www.anthropic.com/news/claude-sonnet-4-5) as the coding LLM to generate code that can be executed to render an image.
Then, we use [GPT-5](https://openai.com/gpt-5/) to generate question-answer pairs with code (without using the rendered image).

Molmo2-SynMultiImageQA is part of the [Molmo2 dataset collection](https://huggingface.co/collections/allenai/molmo2-data) and was used to train the [Molmo2 family of models](https://huggingface.co/collections/allenai/molmo2).

Quick links:
- 📃 [Paper](https://allenai.org/papers/molmo2)
- 🎥 [Blog with Videos](https://allenai.org/blog/molmo2)

## Loading
The dataset has eight subsets:
- `chart`: charts and plots
- `chemical`: chemical structures
- `circuit`: diagrams of electrical circuits
- `diagram`: diagram and graphs
- `document`: various types of documents
- `graphic`: vector graphics
- `music`: music sheets
- `table`: tables and sheets

Use `config_name` to specify which one to load. By default, `chart` will be loaded. For example:

```python
table_dataset = datasets.load_dataset("allenai/Molmo2-SynMultiImageQA", "table", split="train")
```

## Data Format
Each row of the example has the following information:
- `id`: the unique ID of each example
- `images`: a list of rendered images from the code
- `code`: a list of the source code for each image
- `qa_pairs`: a list of questions, answers, and chain-of-thought explanations
- `qa_pairs_raw`: the raw format of QA pairs without replacing the image reference (\<IMAGE-N\>)to natural format.
- `metadata`: metadata of each example, including the content type, persona, overall descriptions, and the number of images.

## Splits
The data is divided into validation and train splits. These splits are "unofficial" because we do not generally use this data for evaluation anyway.
However, they reflect what was used when training the Molmo2 models, which were only trained on the train splits.

## License
This dataset is licensed under ODC-BY. It is intended for research and educational use in accordance with Ai2’s [Responsible Use Guidelines](https://allenai.org/responsible-use).
This dataset includes synthetic images from model outputs using code generated from Claude-Sonnet-4.5, which is subject to Anthropic's [Terms of Service](https://www.anthropic.com/legal/consumer-terms).
The questions are generated from GPT-5, which is subject to OpenAI’s [Terms of Use](https://openai.com/policies/row-terms-of-use).

# Molmo2-SynMultiImageQA
Molmo2-SynMultiImageQA 是一个面向各类富文本图像（涵盖图表、表格、文档、示意图等）的合成式多图像问答配对数据集。

该合成数据集通过将[CoSyn框架](https://yueyang1996.github.io/cosyn/)拓展至多图像场景构建而成，以[Claude-sonnet-4-5](https://www.anthropic.com/news/claude-sonnet-4-5)作为编码大语言模型，生成可执行代码以渲染目标图像。随后，我们使用[GPT-5](https://openai.com/gpt-5/)基于代码生成问答配对（无需借助已渲染的图像）。

Molmo2-SynMultiImageQA 隶属于[Molmo2数据集合集](https://huggingface.co/collections/allenai/molmo2-data)，并被用于训练[Molmo2系列模型](https://huggingface.co/collections/allenai/molmo2)。

快速链接：
- 📃 [论文](https://allenai.org/papers/molmo2)
- 🎥 [带视频的官方博客](https://allenai.org/blog/molmo2)

## 加载方式
该数据集包含8个子集：
- `chart`：各类图表与绘图
- `chemical`：化学结构式
- `circuit`：电气电路示意图
- `diagram`：各类示意图与统计图
- `document`：各类文档
- `graphic`：矢量图形
- `music`：乐谱
- `table`：各类表格与工作表

可通过`config_name`参数指定待加载的子集，默认加载`chart`子集。示例代码如下：
python
table_dataset = datasets.load_dataset("allenai/Molmo2-SynMultiImageQA", "table", split="train")

## 数据格式
每条示例数据包含以下字段：
- `id`：每条示例的唯一标识符
- `images`：由代码渲染得到的图像列表
- `code`：对应各图像的源代码列表
- `qa_pairs`：包含问题、答案与思维链解释的问答配对列表
- `qa_pairs_raw`：问答配对的原始格式，未将图像引用（<IMAGE-N>）替换为自然语言表述
- `metadata`：每条示例的元数据，涵盖内容类型、人物设定、整体描述与图像数量

## 数据集划分
该数据集分为验证集与训练集两个划分，此类划分属于“非官方”划分——原因在于本数据集通常不用于模型评估。不过，其划分方式与训练Molmo2模型时所使用的数据划分完全一致，后者仅基于训练集完成模型训练。

## 许可协议
本数据集采用ODC-BY许可协议发布，仅可按照艾伦人工智能研究所（Ai2）的[负责任使用指南](https://allenai.org/responsible-use)用于研究与教育用途。本数据集包含由Claude-Sonnet-4.5生成的代码所渲染的合成图像，该部分内容受Anthropic的[服务条款](https://www.anthropic.com/legal/consumer-terms)约束。问答内容由GPT-5生成，受OpenAI的[使用条款](https://openai.com/policies/row-terms-of-use)约束。