Pano3D: Matterport3D Semantic & Layout Low Resolution

Spherical cameras capture scenes in a holistic manner and have been used for room layout estimation. Recently, with the availability of appropriate datasets, there has also been progress in depth estimation from a single omnidirectional image. While these two tasks are complementary, few works have been able to explore them in parallel to advance indoor geometric perception, and those that have done so either relied on synthetic data, or used small scale datasets, as few options are available that include both layout annotations and dense depth maps in real scenes. This is partly due to the necessity of manual annotations for room layouts. In this work, we move beyond this limitation and generate a 360° geometric vision (360V) dataset that includes multiple modalities, multi-view stereo data and automatically generated weak layout cues. We also explore an explicit coupling between the two tasks to integrate them into a single-shot trained model. We rely on depth-based layout reconstruction and layout-based depth attention, demonstrating increased performance across both tasks. By using single 360° cameras to scan rooms, the opportunity for facile and quick building-scale 3D scanning arises. The project page is available at https://vcl3d.github.io/ExplicitLayoutDepth/.

球形相机（spherical camera）可全方位整体捕捉场景，已被广泛应用于房间布局估计（room layout estimation）任务。近年来，随着适配数据集的涌现，基于单幅全向图像（omnidirectional image）的深度估计（depth estimation）研究也取得了进展。尽管这两项任务具备互补属性，但鲜有研究能够并行探索二者以推进室内几何感知（indoor geometric perception）的发展；现有相关工作要么依赖合成数据集，要么采用小尺度数据集，原因在于现实场景中同时包含布局标注（layout annotations）与密集深度图（dense depth maps）的可用数据集极为稀缺。这一局限的部分成因在于，房间布局标注需依赖人工完成（manual annotations）。本研究突破了上述局限，构建了包含多模态数据、多视图立体（multi-view stereo）数据与自动生成弱布局线索（weak layout cues）的360°几何视觉（360V）数据集。同时，本研究还探索了两项任务间的显式耦合（explicit coupling）机制，将二者整合为单帧训练模型（single-shot trained model）。我们采用基于深度的布局重建方法，并结合布局引导的深度注意力机制，最终在两项任务上均实现了性能提升。借助单台360°相机扫描房间，可实现便捷高效的建筑级三维扫描（building-scale 3D scanning）。项目主页可访问：https://vcl3d.github.io/ExplicitLayoutDepth/