大规模、多模态、多任务的天基遥感大模型指令微调数据

该数据知识产权数据的总数大于7000万条，能够直接用于天基多模态遥感大模型训练，使其具备图像、区域和像素3个层级、14种细分遥感视觉任务的统一处理能力。打造基于视觉-语言架构的高鲁棒、高性能的星上遥感大模型，通过简单自然语言指令即可实现复杂的遥感任务即时处理。依托“三体计算星座”，在轨部署该天基遥感大模型，大大简化星上任务处理流程，提升在轨处理精度。该模型可应用于全球对地观测，例如地表异常灾害监测，重点海域船舶流量统计等。1. 从互联网数据开放管理平台广泛收集遥感领域的原始数据集。
2. 对影像中带有的数据源标识、采样时间等进行过滤脱敏处理。
3. 人工+算法审查数据，对不符合要求、质量过低的原始数据进行剔除处理。
4. 设计新的标注体系与标准化转换算法，对原始数据集中不同任务、不同格式的标注文件转化为统一格式的标准化标注。
5. 对图像尺寸进行标准化，并设计标注修复算法，对图像裁剪后导致的标注截断问题进行修复。
6. 提出基于大模型的对话生成算法并制定对话模板，对全量数据进行处理，构建指令微调数据集。其中，每条数据包含图像地址、问题指令以及问题答案，总量大于7000万条。
7. 使用产出的指令微调数据集进行模型训练。模型基于视觉-语言模型架构，实现14种多模态遥感任务的统一处理。其中，图像级任务包含分类(IMG_CLS)、简短描述(IMG_CAP)、详细描述([IMG_CAP_DETAILED])、计数([IMG_CT])、视觉问答([IMG_VQA])；区域级任务包含水平框分类([REG_CLS_HBB])、旋转框分类([REG_CLS_OBB])、区域级描述([REG_CAP])、水平框检测([REG_DET_HBB])、旋转框检测([REG_DET_OBB])、视觉定位([REG_VG])；像素级任务包含像素级分类([PIX_CLS])、分割([PIX_SEG])、变化检测([PIX_CHG])

This dataset contains over 70 million intellectual property-related data entries, which can be directly used for training space-based multimodal remote sensing large models, endowing the models with the capability to uniformly handle 14 subdivided remote sensing visual tasks across three hierarchies: image level, regional level, and pixel level.

This work targets building a high-robustness, high-performance on-board remote sensing large model based on visual-language architecture, which enables real-time processing of complex remote sensing tasks via simple natural language instructions. By deploying this space-based remote sensing large model on the "Three-Body Computing Constellation", the on-orbit task processing workflow can be greatly simplified, and the on-orbit processing accuracy can be improved. This model can be applied to global Earth observation scenarios, including but not limited to surface abnormal disaster monitoring, ship flow statistics in key sea areas, etc.

The dataset construction process includes the following 7 steps:
1. Widely collect original remote sensing datasets from open Internet data management platforms.
2. Conduct filtering and de-identification processing on metadata such as data source identifiers and sampling time carried in the images.
3. Review the dataset via a hybrid approach of manual and algorithmic verification, and eliminate unqualified or low-quality original data entries.
4. Design a novel annotation system and standardized conversion algorithm to convert annotation files of different tasks and formats in the original dataset into unified standardized annotations.
5. Standardize the image sizes, and develop annotation repair algorithms to fix annotation truncation problems caused by image cropping operations.
6. Propose a large model-based dialogue generation algorithm and formulate standardized dialogue templates, process the full-scale dataset to construct an instruction fine-tuning dataset. Each data entry contains three components: image address, question instruction, and corresponding question answer, with a total volume of over 70 million entries.
7. Perform model training using the generated instruction fine-tuning dataset. The model is built on a visual-language model architecture, which realizes unified processing of 14 multimodal remote sensing tasks, which are categorized as follows:
- Image-level tasks: classification (IMG_CLS), image captioning (IMG_CAP), detailed image captioning ([IMG_CAP_DETAILED]), counting ([IMG_CT]), visual question answering ([IMG_VQA]);
- Regional-level tasks: horizontal bounding box classification ([REG_CLS_HBB]), oriented bounding box classification ([REG_CLS_OBB]), regional captioning ([REG_CAP]), horizontal bounding box detection ([REG_DET_HBB]), oriented bounding box detection ([REG_DET_OBB]), visual grounding ([REG_VG]);
- Pixel-level tasks: pixel-level classification ([PIX_CLS]), segmentation ([PIX_SEG]), change detection ([PIX_CHG])

搜集汇总

背景与挑战

背景概述

该数据集是由之江实验室登记的企业数据，来源于公开收集，格式为xlsx。数据集规模为701条，总数据量超过7000万条，用于训练天基多模态遥感大模型，支持14种细分遥感视觉任务，应用场景包括全球对地观测。