ICON-QA

<p align="center" width="100%">
<img src="https://i.postimg.cc/g0QRgMVv/WX20240228-113337-2x.png" width="100%" height="80%">
</p>

# Large-scale Multi-modality Models Evaluation Suite

> Accelerating the development of large-scale multi-modality models (LMMs) with `lmms-eval`

🏠 [Homepage](https://lmms-lab.github.io/) | 📚 [Documentation](docs/README.md) | 🤗 [Huggingface Datasets](https://huggingface.co/lmms-lab)

# This Dataset

This is a formatted version of [ICONQA](https://iconqa.github.io/). It is used in our `lmms-eval` pipeline to allow for one-click evaluations of large multi-modality models.

```
@inproceedings{lu2021iconqa,
title = {IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning},
author = {Lu, Pan and Qiu, Liang and Chen, Jiaqi and Xia, Tony and Zhao, Yizhou and Zhang, Wei and Yu, Zhou and Liang, Xiaodan and Zhu, Song-Chun},
booktitle = {The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks},
year = {2021}
}
```

<p align="center" width="100%">
<img src="https://i.postimg.cc/g0QRgMVv/WX20240228-113337-2x.png" width="100%" height="80%">
</p>

# 大规模多模态模型评测套件（Large-scale Multi-modality Models Evaluation Suite）

> 借助`lmms-eval`加速大规模多模态模型（Large-scale Multi-modality Models, LMMs）的研发

🏠 [主页](https://lmms-lab.github.io/) | 📚 [文档](docs/README.md) | 🤗 [Huggingface 数据集](https://huggingface.co/lmms-lab)

# 本数据集

本数据集是[ICONQA](https://iconqa.github.io/)的格式化版本，已集成至我们的`lmms-eval`评测管线中，可实现大规模多模态模型的一键式自动化评测。

@inproceedings{lu2021iconqa,
title = {IconQA: 面向抽象图表理解与视觉语言推理的全新基准数据集},
author = {陆攀、邱亮、陈家琪、夏托尼、赵一洲、张伟、周宇、梁晓丹、朱松纯},
booktitle = {第35届神经信息处理系统大会（NeurIPS）数据集与基准赛道},
year = {2021}
}