Parse Semantics from Geometry: A Remote Sensing Benchmark for Multi-modal Semantic Segmentation

Geometric information in the normalized digital surface models (nDSM) is highly correlated with the semantic class of the land cover. Exploiting two modalities (RGB and nDSM (height)) jointly has a great potential to improve the segmentation performance. Towards a fair and comprehensive analysis of existing methods, in this paper, we introduce a remote sensing benchmark for multi-modal semantic segmentation based on RGB-Height (RGB-H) data. The introduced RSMSS dataset contains 9340 tiles collected from three different cities, including the Oklahoma, Washington, D.C., and Philadelphia. Each RGB image tile has a corresponding nDSM map provided with the resolution of 1024x1024. We split all the image tiles into three subsets: the training set with 5,137 tiles, the validation set with 1,059 tiles and the test set with 3,144 tiles. All the image pixels are annotated with six different land cover types, including 1. ground; 2. low-vegetation; 3. building; 4. water; 5. road; 6. tree

归一化数字表面模型（normalized digital surface models, nDSM）中的几何信息与土地覆盖语义类别高度相关。联合利用RGB与nDSM（高度）两种模态，对提升语义分割性能具有极大潜力。为对现有方法开展公平且全面的分析，本文提出了一款基于RGB-高度（RGB-H）数据的多模态语义分割遥感基准数据集。所提出的RSMSS数据集包含从三座不同城市——俄克拉荷马州、华盛顿哥伦比亚特区以及费城——采集的9340幅图像块。每幅RGB图像块均配有分辨率为1024×1024的nDSM高度图。我们将所有图像块划分为三个子集：训练集含5137幅图像块，验证集含1059幅，测试集含3144幅。所有图像像素均标注有6类土地覆盖类型，分别为：1. 地表；2. 低矮植被；3. 建筑物；4. 水体；5. 道路；6. 树木