ChartSketcher-Data

# ChartSketcher-Data

This is the dataset for the ChartSketcher. Due to its large size, the original file has been split into multiple parts for easier uploading and downloading.

## How to Use

Download all `chartsketcher_part_*` files from this repository into the same directory, then use the following command to merge them back into the original archive, `chartsketcher_release.tar.gz`.

**In a Linux or macOS terminal, run the following command:**

```bash
cat chartsketcher_part_* > chartsketcher_release.tar.gz
```

Once the merge is complete, you can then extract and use the `chartsketcher_release.tar.gz` file.

## Dataset Details

The composition of the dataset is as follows:

| Training Phase | Method | Data Source | Data Type | Quantity |
| :--- | :--- | :--- | :--- | :--- |
| **Cold Start** | SFT | EvoChart Synthetic Chart Data | Correct Reasoning Path | 155,203 (87.3%) |
| | | VisualCoT and its Annotations | Correct Reasoning Path | 22,510 (12.7%) |
| | | **Total** | | **177,713** |
| | DPO | EvoChart Synthetic Chart Data | Reflection Reasoning Path| **147,955** |
| **RL** | KTO | ChartQA and ChartBench | MCTS Sampled Paths | 41,196 (81.6%) |
| | | General QA-Pairs * | MCTS Sampled Paths | 9,259 (18.4%) |
| | | **Total** | | **50,455** |
| **Annealing** | - | Sampled from RL Data | MCTS Sampled Paths | 4,000 |

\* 18.4% of the KTO training data was derived from general vision-language QA-pairs. These were sourced from datasets aggregated by VisualCoT (TextVQA, TextCaps, DocVQA, DUDE, SROIE, CUB-200-2011, Flickr30k, Visual7W, InfographicsVQA, VSR, GQA, and OpenImages). For these samples, we only used their image and QA-pair without adopting the original annotations from VisualCoT, which is effectively equivalent to using the datasets listed above. In the main text, this collection was abbreviated as 'VisualCoT' to save space, and we provide individual citations for each of these datasets in the appendix.
* **Empirical Tip**: It is recommended to use the annealing dataset for a final fine-tuning step with a small learning rate after KTO training is complete. This practice has a negligible impact on performance but improves the model's robustness during OOD inference.

## Links

- 📝 [Paper](https://arxiv.org/abs/2505.19076)
- 💻 [GitHub](https://github.com/MuyeHuang/ChartSketcher)

## Citation
```bibtex
@misc{huang2025chartsketcherreasoningmultimodalfeedback,
title={ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding},
author={Muye Huang and Lingling Zhang and Jie Ma and Han Lai and Fangzhi Xu and Yifei Li and Wenjun Wu and Yaqiang Wu and Jun Liu},
year={2025},
eprint={2505.19076},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```

# ChartSketcher-Data 数据集

本数据集为ChartSketcher配套数据集。由于原始文件体量较大，为便于上传与下载，已将其拆分为多个分卷文件。

## 使用方法

请将本仓库中所有名为`chartsketcher_part_*`的文件下载至同一目录下，随后执行以下命令将其合并为原始归档文件`chartsketcher_release.tar.gz`。

**在Linux或macOS终端中执行如下命令：**
bash
cat chartsketcher_part_* > chartsketcher_release.tar.gz

合并完成后，即可解压并使用`chartsketcher_release.tar.gz`文件。

## 数据集详情

本数据集的构成如下：

| 训练阶段 | 训练方法 | 数据来源 | 数据类型 | 数据量 |
| :--- | :--- | :--- | :--- | :--- |
| **冷启动（Cold Start）** | SFT（监督微调，Supervised Fine-Tuning） | EvoChart 合成图表数据 | 正确推理路径 | 155,203（占比87.3%） |
| | | VisualCoT 及其标注数据 | 正确推理路径 | 22,510（占比12.7%） |
| | | **总计** | | **177,713** |
| | DPO（直接偏好优化，Direct Preference Optimization） | EvoChart 合成图表数据 | 反思推理路径 | **147,955** |
| **强化学习（Reinforcement Learning，RL）** | KTO（知识导向优化，Knowledge-Targeted Optimization） | ChartQA 与 ChartBench 数据集 | MCTS（蒙特卡洛树搜索，Monte Carlo Tree Search）采样路径 | 41,196（占比81.6%） |
| | | 通用视觉语言问答对（General QA-Pairs）* | MCTS采样路径 | 9,259（占比18.4%） |
| | | **总计** | | **50,455** |
| **退火微调（Annealing）** | - | 从强化学习数据中采样 | MCTS采样路径 | 4,000 |

* 注：KTO训练数据中有18.4%来自通用视觉语言问答对，其来源为VisualCoT整合的数据集（包括TextVQA、TextCaps、DocVQA、DUDE、SROIE、CUB-200-2011、Flickr30k、Visual7W、InfographicsVQA、VSR、GQA 与 OpenImages）。针对此类样本，我们仅使用其图像与问答对内容，未采用VisualCoT的原始标注，这等价于直接使用上述所列数据集。为精简篇幅，正文中将该数据集集合简称为“VisualCoT”，各数据集的单独引用信息详见附录。

* **经验建议**：建议在完成KTO训练后，使用退火数据集以较小的学习率执行最终微调步骤。该操作对模型性能影响极小，但可提升模型在分布外（Out-of-Distribution，OOD）推理场景下的鲁棒性。

## 相关链接

- 📝 [论文](https://arxiv.org/abs/2505.19076)
- 💻 [GitHub仓库](https://github.com/MuyeHuang/ChartSketcher)

## 引用格式

bibtex
@misc{huang2025chartsketcherreasoningmultimodalfeedback,
title={ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding},
author={Muye Huang and Lingling Zhang and Jie Ma and Han Lai and Fangzhi Xu and Yifei Li and Wenjun Wu and Yaqiang Wu and Jun Liu},
year={2025},
eprint={2505.19076},
archivePrefix={arXiv},
primaryClass={cs.CV}
}