Physical and Digital Twin Industrial RGBD Dataset with same camera trajectory for 3D vision applications

This dataset introduces a multimodal, paired dataset specifically designed to isolate and analyze the sim-to-real domain gap in camera localization algorithms by eliminating the trajectory gap. Developed at the CESI LINEACT Industry of the Future platform, the dataset comprises two complementary components: a Physical-Twin (PT) dataset and a Digital-Twin (DT) counterpart. The PT dataset includes real-world, stop-and-go acquisitions from six industrial acquisition scenarios: a mobile robot MiR (515 images), a robotic arm Niryo (220 images), a bike (515 images), a bench drill (275 images), an industrial workstation environment (1,215 images), and a virtual reality (VR) laboratory environment (1,000 images). The data were captured using a ZED Mini stereo camera, while highly accurate 6D ground-truth (GT) poses were obtained through a 16-camera Motion Capture (MoCap) system and a 5-point referencing strategy. The dataset provides Full-HD RGB images (1920 × 1080), depth maps, IMU measurements at 800 Hz, and visual-inertial SLAM tracking. The DT dataset reproduces the same physical setups within the Unity game engine using calibrated CAD models and digital reconstructions. By globally synchronizing coordinate origins, accounting for handedness differences, and leveraging the LOOPER tool, the exact human-operated camera trajectories, together with their intrinsic and extrinsic parameters, are replayed in the virtual environment. To address residual spatial discrepancies caused by physical measurement uncertainties, an iterative correction process compares real and synthetic renderings, resulting in paired and validated image sequences accompanied by perfectly accurate synthetic depth maps. The resulting dataset provides a unique platform for evaluating and quantifying the impact of the sim-to-real domain gap on camera localization methods under identical trajectories and acquisition conditions, enabling rigorous comparison between real and synthetic data.