Octavius Dataset

Octavius: Mitigating Task Interference in MLLMs via MoE Zeren Chen1,2* Ziqin Wang1,3* Zhen Wang2* Huayang Liu2 Zhenfei Yin1,4 Si Liu3 Lu Sheng2† Wanli Ouyang1,4 Yu Qiao1 Jing Shao1† 1Shanghai AI Laboratory 2School of Software, Beihang University 3Institute of Artifical Intelligence, Beihang University 4University of Sydney * Equal Contribution † Corresponding Author ArXiv Code Model Zoo Introduction We propose Octavius, a unified, multimodal large language with a novel capability to comprehend various tasks across different modalities, including but not limited to 2D captioning, 2D detection, 3D VQA, and 3D dense captioning. Through combining well-known Mixture-of-Experts (MoE) and one of the representative PEFT techniques, i.e., LoRA, Octavius can efficiently be involved in more downstream tasks and more modalities by learning more LoRA modules, alleviating the potential task interference issuse arise from multimodal learning. Usage Environment installation. Prepare the instruction / benchmark dataset and required pretrained weights for LLMs and visual encoder. Training scripts: Image modality only cd src sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION> <NUM_GPU> \ config/Octavius/octavius_2d_e4_bs64.yaml octavius_2d_e4_bs64 Point cloud modality only cd src sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION> <NUM_GPU> \ config/Octavius/octavius_3d_e3_bs64.yaml octavius_3d_e3_bs64 Image & point cloud modality joint cd src sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION> <NUM_GPU> \ config/Octavius/octavius_2d+3d_e6_bs64.yaml octavius_2d +3d_e6_bs64 We provide pretrained Octavius model here. Evaluation We use ChEF to evaluate Octavius on both image and point cloud modalities, see here for details. Citation @misc{chen2023octavius, title={Octavius: Mitigating Task Interference in MLLMs via MoE}, author={Zeren Chen and Ziqin Wang and Zhen Wang and Huayang Liu and Zhenfei Yin and Si Liu and Lu Sheng and Wanli Ouyang and Yu Qiao and Jing Shao}, year={2023}, eprint={2311.02684}, archivePrefix={arXiv}, primaryClass={cs.CV} } License The project is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.Octavius：基于专家混合模型（Mixture-of-Experts，MoE）缓解多模态大语言模型的任务干扰问题 Zeren Chen1,2* Ziqin Wang1,3* Zhen Wang2* Huayang Liu2 Zhenfei Yin1,4 Si Liu3 Lu Sheng2† Wanli Ouyang1,4 Yu Qiao1 Jing Shao1† 1上海人工智能实验室 2北京航空航天大学软件学院 3北京航空航天大学人工智能研究院 4悉尼大学 * 共同第一作者 † 通讯作者 ArXiv Code Model Zoo 引言 本文提出Octavius，一款统一型多模态大语言模型，具备理解跨模态多样任务的全新能力，涵盖（但不限于）二维图像字幕生成、二维目标检测、三维视觉问答（3D VQA）以及三维密集字幕生成任务。通过结合经典的专家混合模型（Mixture-of-Experts，MoE）与代表性参数高效微调（Parameter-Efficient Fine-Tuning，PEFT）技术之一的LoRA，Octavius可通过学习更多LoRA模块高效适配更多下游任务与模态，有效缓解多模态学习中潜在的任务干扰问题。 使用方法 环境配置。 准备指令数据集/基准数据集以及大语言模型（Large Language Model，LLM）与视觉编码器所需的预训练权重。 训练脚本： 仅图像模态训练 cd src sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION> <NUM_GPU> config/Octavius/octavius_2d_e4_bs64.yaml octavius_2d_e4_bs64 仅点云模态训练 cd src sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION> <NUM_GPU> config/Octavius/octavius_3d_e3_bs64.yaml octavius_3d_e3_bs64 图像与点云模态联合训练 cd src sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION> <NUM_GPU> config/Octavius/octavius_2d+3d_e6_bs64.yaml octavius_2d+3d_e6_bs64 我们在此处提供预训练的Octavius模型。 评估 我们使用ChEF在图像与点云模态上对Octavius进行评估，详情请参见此处。 引用 @misc{chen2023octavius, title={Octavius: Mitigating Task Interference in MLLMs via MoE}, author={Zeren Chen and Ziqin Wang and Zhen Wang and Huayang Liu and Zhenfei Yin and Si Liu and Lu Sheng and Wanli Ouyang and Yu Qiao and Jing Shao}, year={2023}, eprint={2311.02684}, archivePrefix={arXiv}, primaryClass={cs.CV} } 许可协议 本项目采用CC BY NC 4.0许可协议（仅允许非商业性使用），且使用本数据集训练得到的模型不得用于科研以外的用途。

搜集汇总

背景与挑战

背景概述

Octavius Dataset是一个用于多模态大语言模型微调的开源数据集，基于CC BY-NC 4.0许可证，旨在通过结合MoE和LoRA技术处理2D和3D等多种模态任务，以减轻任务干扰。该数据集由上海人工智能实验室于2023年11月发布。