Augmenting and contrasting distortion for open panoramic segmentation

Recently, open panoramic segmentation (OPS) has received widespread attention due to its ability to be trained solely on pinhole images within a closed-vocabulary, while generalizing effectively to panoramic images in an open-vocabulary setting. Previous methods mainly focused on opening the vocabulary, with less attention on opening the field of view (FoV) and opening the domain. This results in two key issues: (1) insufficient distortion perception, as the segmentation model is trained exclusively on pinhole images, lacking exposure to panoramic distortions; and (2) distortion-induced feature misalignment, where the differences in distortion between pinhole and panoramic images cause a domain shift in the feature space. To address these two issues, in this paper, we propose a novel framework named augmenting and contrasting distortion for open panoramic segmentation (ACD4OPS), consisting of random Gaussian deformation augmentation (RGDA) and distortion-aware contrastive learning (DaCL). First, RGDA applies randomly parameterized Gaussian deformation to augment the training set of labeled pinhole images, providing distorted images for the segmentation network. Second, DaCL performs patch- and pixel-level contrastive learning between the pinhole and deformed images to extract distortion-invariant and class-relevant features, reducing gaps in the feature space between pinhole images and panoramas. Extensive experiments on four publicly available indoor and outdoor benchmarks demonstrate the effectiveness of our method.