WaltonFuture/InstructionGPT-4

---
task_categories:
- visual-question-answering
size_categories:
- n<1K
---

# InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4
[Lai Wei](https://waltonfuture.github.io/), Zihao Jiang, [Weiran Huang](https://www.weiranhuang.com/), [Lichao Sun](https://lichao-sun.github.io/).

**Shanghai Jiao Tong University, Lehigh University**

[Paper](https://arxiv.org/abs/2308.12067), [Link](https://mp.weixin.qq.com/s/s4Acec71v5oMlFkyhlCL_g), [Code](https://github.com/waltonfuture/InstructionGPT-4)

## Introduction
Multimodal large language models acquire their instruction-following capabilities through a two-stage training process: pre-training on image-text pairs and fine-tuning on supervised vision-language instruction data. Recent studies have shown that large language models can achieve satisfactory results even with a limited amount of high-quality instruction-following data. In this paper, we introduce InstructionGPT-4, which is fine-tuned on a small dataset comprising only 200 examples, amounting to approximately 6% of the instruction-following data used in the alignment dataset for MiniGPT-4. We first propose several metrics to access the quality of multimodal instruction data. Based on these metrics, we present a simple and effective data selector to automatically identify and filter low-quality vision-language data. By employing this method, InstructionGPT-4 outperforms the original MiniGPT-4 on various evaluations (e.g., visual question answering, GPT-4 preference). Overall, our findings demonstrate that less but high-quality instruction tuning data is efficient to enable multimodal large language models to generate better output.

## Usage
You can download our vision-language dataset containing only 200 high-quality examples and replace the original cc_sbu_align dataset used in the fine-tuning stage of MiniGPT-4. The training settings are the same as [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4).

If you're using InstructionGPT-4 in your research or applications, please cite using this BibTeX:
```bibtex
@article{wei2023instructiongpt,
title={InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4},
author={Wei, Lai and Jiang, Zihao and Huang, Weiran and Sun, Lichao},
journal={arXiv preprint arXiv:2308.12067},
year={2023}
}
```

原始信息汇总

InstructionGPT-4 数据集概述

基本信息

• 任务类别: 视觉问答 (visual-question-answering)
• 数据集规模: 小于1000条 (n<1K)

数据集介绍

• 名称: InstructionGPT-4
• 来源: 上海交通大学, 理海大学
• 作者: Lai Wei, Zihao Jiang, Weiran Huang, Lichao Sun
• 相关链接: 论文, 代码

数据集详情

• 数据集大小: 仅包含200个高质量示例
• 用途: 用于替换MiniGPT-4微调阶段的原始cc_sbu_align数据集
• 训练设置: 与MiniGPT-4相同

引用信息

bibtex @article{wei2023instructiongpt, title={InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4}, author={Wei, Lai and Jiang, Zihao and Huang, Weiran and Sun, Lichao}, journal={arXiv preprint arXiv:2308.12067}, year={2023} }