Earth on Canvas

基于零镜头草图的遥感图像多模态目标检索方案随着传感器技术的进步，正在从各种卫星收集大量数据。因此，基于目标的数据检索和获取任务变得极具挑战性。现有的卫星基本上使用各种传感技术扫描地球上广阔的重叠区域，例如多光谱、高光谱、合成孔径雷达 (SAR)、视频和压缩传感等。随着复杂性的增加和我们可以使用的不同传感技术，考虑到不同传感器捕获的互补信息，设计有效的算法以从多种数据模式中检索数据已成为我们的主要兴趣。这种类型的问题被称为跨模式数据检索。在遥感（RS）中，主要有两种重要类型的问题，即土地覆盖分类和目标检测。在这项工作中，我们专注于基于目标的对象检索部分，它属于 RS 中的对象检测领域。对象检索本质上需要高分辨率图像才能使对象在图像中清晰可见。使用大规模数据库的传统检索方法的主要挑战是，我们经常没有目标类的任何查询图像样本可供我们使用。感兴趣的目标仅以不精确草图的形式作为对用户的感知而存在。在没有照片查询的这种情况下，如果我们能够迅速制作目标的快速手绘草图，这将非常有用。草图是数据的高度符号化和象形文字表示。人们可以利用这种简约代表草图查询的概念，用于基于草图的图像检索 (SBIR) 框架。在处理卫星图像时，必须为每个对象类别收集尽可能多的图像样本，以实现高成功率的对象识别。然而，一般来说，存在相当多的类，我们很少有任何训练数据样本。因此，对于此类，我们可以使用零样本学习（ZSL）策略。 ZSL 方法旨在解决一项任务，而无需在训练阶段收到任何该任务的示例。这使得网络能够处理在网络部署时在推理阶段获得的看不见的类（新类）样本。因此，我们提出航拍草图图像数据集，即画布上的地球数据集。此数据集中的类：飞机、棒球钻石、建筑物、高速公路、高尔夫球场、港口、十字路口、移动房屋公园、立交桥、停车场、河流、跑道、储罐、网球场。

Zero-shot sketch-based multimodal object retrieval for remote sensing images: With the advancement of sensor technologies, vast amounts of data are being collected from various satellites. Consequently, target-based data retrieval and acquisition tasks have become extremely challenging. Current satellites typically scan vast overlapping regions of the Earth's surface using diverse sensing technologies, such as multispectral, hyperspectral, Synthetic Aperture Radar (SAR), video, and compressive sensing, among others. As the complexity and variety of available sensing technologies increase, designing effective algorithms to retrieve data across multiple data modalities—taking into account the complementary information captured by different sensors—has become a core research interest. This type of problem is known as cross-modal data retrieval. In remote sensing (RS), two major critical problem categories exist: land cover classification and object detection. In this work, we focus on the target-based object retrieval subtask, which falls within the scope of object detection in RS. Object retrieval inherently requires high-resolution imagery to ensure that targets are clearly visible within the images. A major challenge for traditional retrieval methods using large-scale databases is that we often lack any query image samples for the target classes of interest. Instead, targets of interest only exist as imprecise hand-drawn sketches in the user's perceptual conception. In such scenarios where no photo-based queries are available, it would be highly beneficial if users could quickly create a rapid hand-drawn sketch of the target. Sketches are highly symbolic and hieroglyphic representations of data. This concept of using such minimalist representative sketches as queries can be applied to sketch-based image retrieval (SBIR) frameworks. When processing satellite imagery, one must collect as many image samples as possible for each object category to achieve high-success-rate object recognition. However, in general, there are a considerable number of classes for which we have very few or no training data samples. Therefore, zero-shot learning (ZSL) strategies can be employed for such classes. ZSL methods are designed to solve a task without receiving any examples of that task during the training phase, enabling the network to handle samples from unseen classes (novel classes) obtained during the inference phase when the network is deployed. Accordingly, we propose an aerial sketch-image dataset: the Earth on Canvas dataset. The classes included in this dataset are: airplane, baseball diamond, building, highway, golf course, harbor, intersection, mobile home park, overpass, parking lot, river, runway, storage tank, and tennis court.

搜集汇总

背景与挑战

背景概述

Earth on Canvas是一个专注于零样本学习和跨模态检索的遥感图像数据集，旨在通过草图进行图像检索，包含飞机、建筑物等多种地理对象类别。数据集由德国航空航天中心于2020年发布，适用于土地覆盖分类和目标检测研究。