DenseFusion-1M

- [Paper] https://arxiv.org/abs/2407.08303
- [GitHub] https://github.com/baaivision/DenseFusion

## Introduction
- An image is worth a thousand words". Comprehensive image descriptions are essential for multi-modal perception, while images contains various visual elements of different granularities that are challenging to harness.
- We propose Perceptural Fusion to integrate the diverse visual perception experts for capturing visual elements and adopt a MLLM as a centric pivot for comprehensive perception.
- We thereby provide DenseFusion-1M dataset for highly informative image descriptions with various visual details, including rich OCR information, accurate object and position recognition, and external knowledge, etc.

Detaset details: Comprehensive image descriptions obtained through perceptual fusion of different visual experts.

Usage: It is constructed for comprehensive perception ability for multi-modal large language model and show potentials for fine-grained text conditioned image generation.

DenseFusion Dataset Card

- DenseFusion-1M: DenseFusion-1M/DenseFusion-1M.jsonl is generated by our caption engine through perceptual fusion.
- DenseFusion-4V-100K: DenseFusion-4V-100k/DenseFusion-4V-100k.jsonl is generated by GPT-4V through perceptual fusion.

This project is under the policy of MIT License.

- [论文] https://arxiv.org/abs/2407.08303
- [GitHub] https://github.com/baaivision/DenseFusion

## 引言

- “一图胜千言”。详尽的图像描述对多模态感知至关重要，但图像包含不同粒度的多样视觉元素，难以被充分利用。
- 我们提出感知融合（Perceptural Fusion）框架，以整合各类视觉感知专家模块来捕捉视觉元素，并采用多模态大语言模型（MLLM）作为核心枢纽，实现全面感知。
- 为此，我们构建了DenseFusion-1M数据集，用于生成涵盖各类视觉细节的高信息量图像描述，包括丰富的光学字符识别（OCR）信息、精准的物体与位置识别结果以及外部知识等。

数据集详情：通过不同视觉专家模块的感知融合得到的详尽图像描述。

应用场景：本数据集旨在赋能多模态大语言模型的全面感知能力，同时在细粒度文本条件图像生成任务中展现出应用潜力。

DenseFusion数据集卡片

- DenseFusion-1M：该子集的DenseFusion-1M/DenseFusion-1M.jsonl文件由我们的图像字幕生成引擎通过感知融合流程生成。
- DenseFusion-4V-100K：该子集的DenseFusion-4V-100k/DenseFusion-4V-100k.jsonl文件由GPT-4V通过感知融合流程生成。

本项目遵循MIT开源许可协议。

搜集汇总

背景与挑战

背景概述

DenseFusion-1M是一个通过感知融合技术生成全面图像描述的数据集，包含丰富的视觉细节如OCR信息、对象识别和外部知识，旨在增强多模态大型语言模型的感知能力并支持细粒度图像生成。