PengxiangLi/FIRE

FIRE数据集是一个用于反馈集成和视觉语言模型（VLMs）改进评估的多模态数据集。它包含1.1M多轮对话，这些对话来源于27个源数据集，旨在通过用户反馈来改进VLMs的响应。数据集分为FIRE-100K和FIRE-1M两部分，其中FIRE-100K由GPT-4V生成，FIRE-1M则是在FIRE-100K上训练的模型自由生成的。此外，还构建了FIRE-Bench基准，用于全面评估VLMs的反馈改进能力，包含11K测试对话、两种评估设置和一个提供反馈的模型。通过微调LLaVA模型在FIRE-100K和FIRE-1M上，开发了FIRE-LLaVA模型，该模型在FIRE-Bench上表现出显著的反馈改进能力，比未训练的VLMs提高了50%。

The FIRE dataset is a multimodal dataset designed for feedback integration and refinement evaluation of Vision Language Models (VLMs). It consists of 1.1M multi-turn conversations derived from 27 source datasets, aimed at improving VLMs responses based on user feedback. The dataset is divided into two components: FIRE-100K, generated by GPT-4V, and FIRE-1M, freely generated by models trained on FIRE-100K. Additionally, FIRE-Bench was constructed to comprehensively evaluate the feedback-refining capability of VLMs, containing 11K test conversations, two evaluation settings, and a model to provide feedback for VLMs. By fine-tuning the LLaVA model on FIRE-100K and FIRE-1M, the FIRE-LLaVA model was developed, which demonstrates remarkable feedback-refining capability on FIRE-Bench, outperforming untrained VLMs by 50%.

原始信息汇总

FIRE 数据集概述

数据集描述

• 名称: FIRE
• 类型: 反馈-细化数据集
• 规模: 包含1.1M多轮对话
• 来源: 由27个源数据集衍生
• 应用: 用于视觉语言模型（VLMs）的反馈细化能力训练

数据集组成

• FIRE-100K: 由GPT-4V生成
• FIRE-1M: 基于FIRE-100K训练的模型自由生成

评估基准

• FIRE-Bench: 包含11K反馈-细化对话的测试数据
• 评估设置: 两种评估设置
• 反馈模型: 提供反馈的模型

模型开发

• FIRE-LLaVA: 通过在FIRE-100K和FIRE-1M上微调LLaVA开发
• 性能: 在FIRE-Bench上表现出显著的反馈细化能力，优于未训练的VLMs 50%

引用

bibtex @misc{li2024firedatasetfeedbackintegration, title={FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models}, author={Pengxiang Li and Zhi Gao and Bofei Zhang and Tao Yuan and Yuwei Wu and Mehrtash Harandi and Yunde Jia and Song-Chun Zhu and Qing Li}, year={2024}, eprint={2407.11522}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2407.11522}, }