MMVet

<p align="center" width="100%"> <img src="https://i.postimg.cc/g0QRgMVv/WX20240228-113337-2x.png" width="100%" height="80%"> </p> # Large-scale Multi-modality Models Evaluation Suite > Accelerating the development of large-scale multi-modality models (LMMs) with `lmms-eval` 🏠 [Homepage](https://lmms-lab.github.io/) | 📚 [Documentation](docs/README.md) | 🤗 [Huggingface Datasets](https://huggingface.co/lmms-lab) # This Dataset This is a formatted version of [MM-Vet](https://github.com/yuweihao/MM-Vet). It is used in our `lmms-eval` pipeline to allow for one-click evaluations of large multi-modality models. ``` @misc{yu2023mmvet, title={MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities}, author={Weihao Yu and Zhengyuan Yang and Linjie Li and Jianfeng Wang and Kevin Lin and Zicheng Liu and Xinchao Wang and Lijuan Wang}, year={2023}, eprint={2308.02490}, archivePrefix={arXiv}, primaryClass={cs.AI} } ```<p align="center" width="100%"> <img src="https://i.postimg.cc/g0QRgMVv/WX20240228-113337-2x.png" width="100%" height="80%"> </p> # 大规模多模态模型评测套件 > 借助`lmms-eval`加速大规模多模态模型（Large-scale Multi-modality Models, LMMs）的研发 🏠 [主页](https://lmms-lab.github.io/) | 📚 [文档](docs/README.md) | 🤗 [HuggingFace数据集](https://huggingface.co/lmms-lab) # 本数据集 本数据集是[MM-Vet](https://github.com/yuweihao/MM-Vet)的格式化版本，可应用于我们的`lmms-eval`评测流水线，实现大规模多模态模型的一键式评测。 @misc{yu2023mmvet, title={MM-Vet：面向集成能力的大规模多模态模型评测}, author={Weihao Yu and Zhengyuan Yang and Linjie Li and Jianfeng Wang and Kevin Lin and Zicheng Liu and Xinchao Wang and Lijuan Wang}, year={2023}, eprint={2308.02490}, archivePrefix={arXiv}, primaryClass={cs.AI} }

搜集汇总

背景与挑战

背景概述

MMVet是一个用于评估大型多模态模型（LMMs）综合能力的数据集，基于2023年的研究论文开发，采用Apache License 2.0许可。该数据集作为lmms-eval管道的一部分，支持一键评估，旨在加速多模态模型的开发和性能测试。