Parse Semantics from Geometry: A Remote Sensing Benchmark for Multi-modal Semantic Segmentation

Geometric information in the normalized digital surface models (nDSM) is highly correlated with the semantic class of the land cover. Exploiting two modalities (RGB and nDSM (height)) jointly has a great potential to improve the segmentation performance. Towards a fair and comprehensive analysis of existing methods, in this paper, we introduce a remote sensing benchmark for multi-modal semantic segmentation based on RGB-Height (RGB-H) data. The introduced RSMSS dataset contains 9340 tiles collected from three different cities, including the Oklahoma, Washington, D.C., and Philadelphia. Each RGB image tile has a corresponding nDSM map provided with the resolution of 1024x1024. We split all the image tiles into three subsets: the training set with 5,137 tiles, the validation set with 1,059 tiles and the test set with 3,144 tiles. All the image pixels are annotated with six different land cover types, including 1. ground; 2. low-vegetation; 3. building; 4. water; 5. road; 6. tree

归一化数字表面模型（normalized digital surface models，简称nDSM）中的几何信息与土地覆盖的语义类别高度相关。联合采用RGB与nDSM（高度）两种模态，对提升语义分割性能具有显著潜力。为对现有方法进行公平且全面的评测分析，本文提出了一个基于RGB-高度（RGB-H）数据的多模态语义分割遥感基准数据集。所提出的RSMSS数据集包含从三座不同城市——俄克拉荷马州、华盛顿哥伦比亚特区以及费城——采集的9340幅影像瓦片。每幅RGB影像瓦片均配有分辨率为1024×1024的nDSM地图。我们将所有影像瓦片划分为三个子集：训练集包含5137幅瓦片，验证集包含1059幅瓦片，测试集包含3144幅瓦片。所有影像像素均标注了6类土地覆盖类型，依次为：1. 地表；2. 低矮植被；3. 建筑物；4. 水体；5. 道路；6. 树木。