MuSHRoom: Multi-Sensor Hybrid Room Dataset for Joint 3D Reconstruction and Novel View Synthesis (Kinect Part 2)

Metaverse technologies demand accurate, real-time, and immersive modeling on consumer-grade hardware for both non-human perception (e.g., drone/robot/autonomous car navigation) and immersive technologies like AR/VR, requiring both structural accuracy and photorealism. However, there exists a knowledge gap in how to apply geometric reconstruction and photorealism modeling (novel view synthesis) in a unified framework. To address this gap and promote the development of robust and immersive modeling and rendering with consumer-grade devices, first, we propose a real-world Multi-Sensor Hybrid Room Dataset (MuSHRoom). Our dataset presents exciting challenges and requires state-of-the-art methods to be cost-effective, robust to noisy data and devices, and can jointly learn 3D reconstruction and novel view synthesis, instead of treating them as separate tasks, making them ideal for real-world applications. Second, we benchmark several famous pipelines on our dataset for joint 3D mesh reconstruction and novel view synthesis. Finally, in order to further improve the overall performance, we propose a new method that achieves a good trade-off between the two tasks. Our dataset and benchmark show great potential in promoting the improvements for fusing 3D reconstruction and high-quality rendering in a robust and computationally efficient end-to-end fashion.元宇宙（Metaverse）技术需在消费级硬件上实现精准、实时且沉浸式的建模，以支撑非人类感知任务（如无人机、机器人、自动驾驶汽车导航）以及增强现实（Augmented Reality，AR）、虚拟现实（Virtual Reality，VR）等沉浸式技术，此类建模需同时兼顾结构准确性与照片级真实感。然而，当前在如何将几何重建与照片级真实感建模（新视角合成，novel view synthesis）统一于同一框架的问题上，仍存在研究空白。 为填补这一研究空白，推动消费级设备上鲁棒性沉浸式建模与渲染技术的发展，我们首先提出了真实世界多传感器混合室内数据集（Multi-Sensor Hybrid Room Dataset，MuSHRoom）。该数据集设置了极具挑战性的任务场景，要求前沿方法具备成本效益、对噪声数据与设备干扰具有鲁棒性，且能够联合学习3D重建与新视角合成任务，而非将二者视作独立任务，因此十分适用于真实世界应用场景。其次，我们基于该数据集对多款知名的联合3D网格重建与新视角合成流水线开展了基准测试。最后，为进一步提升整体性能，我们提出了一种可在两项任务间实现良好平衡的全新方法。本数据集与基准测试框架展现出巨大潜力，可推动以鲁棒且计算高效的端到端方式融合3D重建与高质量渲染技术的研究进步。