REAL3EXT

Time-of-Flight sensors and stereo vision systems are two of the most diffused depth acquisition devices for commercial and industrial applications. They share complementary strengths and weaknesses. For this reason, the combination of data acquired from these devices can improve the final depth estimation performance. We introduces a dataset acquired with a multi-camera system composed by a Kinect v2 ToF sensor, an Intel RealSense R200 active stereo sensor and a ZED passive stereo camera system. The acquired scenes include indoor settings with different external lighting conditions. The depth ground truth has been acquired for each scene of the dataset using a line laser. The data can be used for developing fusion and denoising algorithms for depth estimation and test with different lighting conditions. A subset of the data has already been used for the experimental evaluation of the stereo-ToF fusion method of Agresti et al. The multi camera acquisition system used to acquire the proposed dataset is arranged as in the figure below. The reference system is the ZED camera in the center, underneath the ZED there is the Kinect and above there is the RealSense R200. The three cameras are kept in place by a plastic mount specifically designed to fit them. The depth camera of the Kinect is approximately horizontally aligned with the left camera of the ZED with 40 mm vertical displacement, while the color camera is approximately in between the passive stereo pair. The RealSense R200 is placed approximately 20 mm above the ZED camera, with the two IR and color camera inside the baseline of the passive stereo pair. The subjects of the 10 scenes in the REAL3EXT dataset try to stress various flows of the stereo and ToF systems. Critical points are for example lack of texture for the passive stereo system and the presence of low reflect elements and external illumination for the active sensors. The scenes are composed by flat surfaces with and without textures, plants and objects of various material such as plastic, paper and cotton fabric. These are characterized by various specularity properties as reflective and glossy surfaces and rough materials. Each scene was recorded under 4 different external lighting conditions, which are the following: with no external light; with regular lighting; with stronger light; with an additional incandescent light source. Each lighting condition can highlight the weakness and strength of the different depth estimation algorithms. We added the acquisitions with the additional incandescent light source since its spectrum, in the IR wavelength, covers the working range of the active depth cameras and it is a known problem for those devices.

飞行时间（Time-of-Flight, ToF）传感器与立体视觉系统是两类应用最为广泛的商用及工业级深度采集设备，二者优势与劣势互为补充。因此，融合这两类设备采集的数据可提升最终深度估计的性能。本文提出一款数据集，其采集自由Kinect v2飞行时间（ToF）传感器、英特尔RealSense R200主动式立体传感器以及ZED被动式立体相机系统组成的多相机采集平台。该数据集涵盖不同外部光照条件下的室内场景。数据集内每个场景的深度真值均通过线激光采集设备获取。本数据集可用于研发深度估计的融合与去噪算法，并支持在不同光照条件下开展算法测试。该数据集的部分子集已被用于Agresti等人提出的立体-ToF融合方法的实验评估。本数据集的多相机采集系统布局如下图所示：系统以位于中心的ZED相机为参考坐标系，ZED下方为Kinect传感器，上方为RealSense R200传感器。三台相机通过定制化塑料固定支架稳固安装。Kinect的深度相机与ZED的左相机大致水平对齐，仅存在40mm的垂直偏移；而Kinect的彩色相机则大致位于ZED被动立体相机对的中轴线位置。RealSense R200则位于ZED相机上方约20mm处，其内置的红外与彩色相机均处于ZED被动立体相机对的基线范围内。REAL3EXT数据集包含10组场景，旨在针对性地凸显立体视觉与ToF系统的各类性能局限。典型测试难点包括：被动立体系统易受纹理缺失的影响，而主动式传感器则易受低反射物体与外部光照的干扰。场景涵盖带纹理与无纹理的平面、植物以及多种材质的物体，包括塑料、纸张与棉织物等，这些物体具备不同的镜面反射特性，涵盖反光、光滑表面以及粗糙材质等类型。每组场景均在4种不同的外部光照条件下采集，具体如下：无外部光照、常规照明、强光照明以及额外添加白炽光源照明。每种光照条件均可凸显不同深度估计算法的优势与不足。本次研究额外增设了白炽光源照明的采集场景，原因在于其红外波段的光谱覆盖了主动式深度相机的工作频段，这也是此类设备普遍面临的已知挑战。

搜集汇总

背景与挑战

背景概述

REAL3EXT是一个用于深度估计的多摄像头系统数据集，包含Kinect v2、Intel RealSense R200和ZED摄像头采集的数据，涵盖10个场景和4种不同光照条件，适用于开发深度估计的融合和去噪算法。