LAMM

LAMM-Dataset 包括一个包含 186,098 个图像语言指令-响应对的图像指令调整数据集和一个包含 10,262 个点云语言指令-响应对的点云指令调整数据集。 我们从公开可用的数据集中收集图像和点云，并使用 GPT API 和自我指导方法根据这些数据集中的原始标签生成指令和响应。 生成的 LAMM 数据集具有三个吸引人的属性：
1. 现有的多模态指令调优数据集主要关注整体和粗略的信息。 为了强调细粒度和密集信息，我们添加了更多视觉信息，例如视觉关系和细粒度类别作为 GPT API 的输入。
2. 我们观察到现有的 MLLM 可能难以理解视觉任务指令。 为了解决这个问题，我们设计了一种将视觉任务注释转换为指令-响应对的方法，从而增强了 MLLM 对视觉任务指令的理解和泛化。
3. LAMM-Dataset 还包括用于常识性知识问答的数据对，方法是结合来自 Bamboo 数据集的分层知识图标签系统和相应的维基百科描述。

The LAMM-Dataset consists of an image instruction tuning dataset containing 186,098 image-language instruction-response pairs and a point cloud instruction tuning dataset containing 10,262 point cloud-language instruction-response pairs. We collect images and point clouds from publicly available datasets, and generate instructions and responses based on the original labels within these datasets using the GPT API and self-instruction methods. The resulting LAMM-Dataset features three desirable properties:
1. Existing multimodal instruction tuning datasets primarily focus on holistic and coarse-grained information. To emphasize fine-grained and dense information, we add more visual details such as visual relationships and fine-grained categories as inputs to the GPT API.
2. We observe that existing MLLMs may struggle to comprehend visual task instructions. To address this issue, we design a method that converts visual task annotations into instruction-response pairs, thereby enhancing the MLLMs' understanding and generalization of visual task instructions.
3. The LAMM-Dataset also includes data pairs for commonsense knowledge question answering, by incorporating the hierarchical knowledge graph labeling system from the Bamboo dataset and corresponding Wikipedia descriptions.