M4-Instruct-Data

# M4-Instruct Dataset Card ## Dataset details **Dataset type:** M4-Instruct is a set of multi-image datasets that are collected from public datasets or generated by the GPT-4V API. It is constructed for training LMMs for their interleaved multi-image capbilities, e.g., LLaVA-NeXT-Interleave. **Dataset date:** M4-Instruct was collected in April 2024, and released in June 2024. **Paper or resources for more information:** Blog: https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/ **Data Statistics:** <img src="data-statistics.png" alt="isolated" width="400"/> Note that we only released the multi-image, multi-frame (video), and multi-view (3D) data of M4-Instruct in this repo. Please also download videos from [LLaVA-Hound](https://huggingface.co/datasets/ShareGPTVideo/train_video_and_instruction/tree/main/train_300k), and organize them according to their instructions. To reproduce LLaVA-NeXT-Interleave, you can complement the multi-patch (single-image) data by randomly sampling 307K of the stage-2 SFT data of [LLaVA-1.5](https://arxiv.org/pdf/2310.03744). **Data Content:** - json file: m4_instruct_annotations.json and m4_instruct_video.json - images: *.zip - For dreamsim_split.z01 and dreamsim_split.zip, please run "zip -s 0 dreamsim_split.zip --out dreamsim.zip" **License:** Creative Commons Attribution 4.0 International; and it should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use **Where to send questions or comments about the model:** [e-mail] [e-mail] ## Intended use **Primary intended uses:** The primary use of M4-Instruct Data is research on large multimodal models and chatbots. **Primary intended users:** The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.# M4-Instruct 数据集卡片 ## 数据集详情 **数据集类型：** M4-Instruct 是一套多图像数据集，其数据来源涵盖公开数据集，或通过 GPT-4V API 生成。本数据集专为训练大多模态模型（Large Multimodal Model, LMM）的交错式多图像处理能力而构建，例如 LLaVA-NeXT-Interleave。 **数据集采集与发布时间：** M4-Instruct 于2024年4月完成采集，并于2024年6月正式发布。 **更多信息参考资源：** 博客：https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/ **数据统计：** <img src="data-statistics.png" alt="isolated" width="400"/> 注：本仓库仅发布了 M4-Instruct 中的多图像、多帧（视频）及多视角（3D）数据。 请从 [LLaVA-Hound](https://huggingface.co/datasets/ShareGPTVideo/train_video_and_instruction/tree/main/train_300k) 下载视频资源，并按照配套指令进行整理。 若需复现 LLaVA-NeXT-Interleave，可通过随机采样 [LLaVA-1.5](https://arxiv.org/pdf/2310.03744) 的第二阶段监督微调（Supervised Fine-Tuning, SFT）数据中的307K条数据，以此补全多补丁（单图像）数据。 **数据内容：** - 标注文件：`m4_instruct_annotations.json` 与 `m4_instruct_video.json` - 图像文件：`*.zip` - 对于 `dreamsim_split.z01` 和 `dreamsim_split.zip`，请执行命令 `zip -s 0 dreamsim_split.zip --out dreamsim.zip` 进行分卷合并。 **许可协议：** 采用知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International），同时需遵守 OpenAI 的使用条款：https://openai.com/policies/terms-of-use **数据集反馈与咨询方式：** [e-mail] [e-mail] ## 预期用途 **核心预期用途：** 本数据集的核心用途为开展大多模态模型与聊天机器人相关研究。 **核心目标用户：** 本数据集的目标用户为计算机视觉、自然语言处理、机器学习及人工智能领域的研究人员与爱好者。

搜集汇总

背景与挑战

背景概述

M4-Instruct-Data是一个多图像数据集，通过收集公开数据或使用GPT-4V API生成，旨在训练大型多模态模型的交错多图像处理能力。该数据集于2024年6月发布，采用CC-BY-4.0许可证，并需遵循OpenAI政策，主要用于大型多模态模型和聊天机器人的研究，面向计算机视觉和人工智能领域的研究人员及爱好者。