PVIT/pvit_data_stage1

---
license: cc-by-nc-4.0
---

# PVIT dataset

This is the stage 1 pretraining dataset of paper: [Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models](https://arxiv.org/abs/2308.13437).

## Model description

Position-enhanced Visual Instruction Tuning (PVIT) extends the MLLM by incorporating an additional region-level vision encoder to facilitate support for region-based inputs. Specifically, we adopt the vision encoder from RegionCLIP and utilize it to extract region-level features by taking images and regions as inputs. As an additional source of information, the incorporation of region-level features in this way has a minimal impact on the original MLLM. Furthermore, since the features provided by RegionCLIP are themselves already aligned to the language at a fine-grained level, the overhead of aligning it to the MLLM will be relatively small. Following [LLaVA](https://github.com/haotian-liu/LLaVA), we design a two-stage training strategy for PVIT that first pre-training a linear projection to align the region features to the LLM word embedding, followed by end-to-end fine-tuning to follow complex fine-grained instructions.

For more details, please refer to our [paper](https://arxiv.org/abs/2308.13437) and [github repo](https://github.com/THUNLP-MT/PVIT).

## How to use

See [here](https://github.com/THUNLP-MT/PVIT#Train) for instructions of pretraining.

## Intended use

Primary intended uses: The primary use of PVIT is research on large multimodal models and chatbots.

Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.

## BibTeX entry and citation info

```bibtex
@misc{chen2023positionenhanced,
title={Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models},
author={Chi Chen and Ruoyu Qin and Fuwen Luo and Xiaoyue Mi and Peng Li and Maosong Sun and Yang Liu},
year={2023},
eprint={2308.13437},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```

原始信息汇总

PVIT 数据集

数据集描述

PVIT 数据集是论文《Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models》的阶段1预训练数据集。该数据集通过引入区域级视觉编码器扩展了多模态大语言模型（MLLM），以支持基于区域的输入。具体来说，采用了RegionCLIP的视觉编码器，通过输入图像和区域来提取区域级特征。这种区域级特征的引入对原始MLLM的影响最小，并且由于RegionCLIP提供的特征本身已经在细粒度级别上与语言对齐，因此将其对齐到MLLM的开销相对较小。

使用方法

预训练的详细说明请参见这里。

预期用途

主要用途：PVIT主要用于大型多模态模型和聊天机器人的研究。 主要用户：该模型的主要用户是计算机视觉、自然语言处理、机器学习和人工智能领域的研究人员和爱好者。

引用信息

bibtex @misc{chen2023positionenhanced, title={Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models}, author={Chi Chen and Ruoyu Qin and Fuwen Luo and Xiaoyue Mi and Peng Li and Maosong Sun and Yang Liu}, year={2023}, eprint={2308.13437}, archivePrefix={arXiv}, primaryClass={cs.CV} }