Multi-sensor dataset for testing merge of Hyperspectral, HD and 3D cloud information for image recognition

This data contain multisensor image dataset constructed for benchmarking purposes. It contains multiple images of the constructed scenes -- on which objects made of different materials are placed to test image recognition scenarios. The scene is recorded from various angles by imagining sensors, i.e. HSI camera, HD camera on mobile chassis and MS Kinect to provide complete information. Equipment The imaging was performed with use of three devices for three different approaches to data. Those three devices' imaging characteristics are widely different when it comes to angle and resolution which required them to be separately positioned to acquire the matching images. Therefore while HD camera was being transferred on the moving platform (chassis), both Kinect and SOC710 were placed on a stationary position which was moved between the frames by hand. Hyperspectral data Hyperspectral data acquisition was performed with Surface Optics SOC710 camera. This camera records spectra at VNIR range $377-1046$ nm; the output image has dimensions $696 \times 520$ with 128 bands and $12$ bit dynamic range. The camera is equipped with sensor line translation unit and can be used from static stand as a conventional camera (i.e. it does not require mechanical translation of the observed sample or rotary stand, as in traditional ‘push broom’ hyperspectral cameras). The lighting was provided with four ambient lamps and adjusted for each scenario separately, so that most of the dynamic range of the camera was used and image saturation is avoided. Captured hyperspectral images were subject to a standard calibration procedure, including: the removal of a dark frame, spectral and radiometric calibration as well as reflectance normalization using the calibration panel. 3D point clouds The Kinect sensor incorporates several advanced sensing hardware. The depth sensor consists of the IR projector combined with the IR camera, which is a monochrome complementary metaloxide semiconductor (CMOS) sensor. The IR projector is an IR laser that passes through a diffraction grating and turns into a set of IR dots. The relative geometry between the IR projector and the IR camera as well as the projected IR dot pattern are known. If we can match a dot observed in an image with a dot in the projector pattern, we can reconstruct it in 3D using triangulation. Because the dot pattern is relatively random, the matching between the IR image and the projector pattern can be done in a straightforward way by comparing small neighborhoods using, for example, normalized cross correlation. The depth value is encoded with gray values; the darker a pixel, the closer the point is to the camera in space. The black pixels indicate that no depth values are available for those pixels. This might happen if the points are too far (and the depth values cannot be computed accurately), are too close (there is a blind region due to limited fields of view for the projector and the camera), are in the cast shadow of the projector (there are no IR dots), or reflect poor IR lights HD Images The HD images were acquired using 5 Megapixel HD camera mounted on a mobile chassis made by Dawn Robotics, that allowed the camera to be moved freely on the scene. Both camera and mobile chassis was controlled by a Raspberry PI unit which was also responsible to position the camera in accord to the data being collected by other sources. Data The dataset consists of three scenes consisting of various objects -- minerals, fruit, wood plastic and metal -- placed on a stand. The objects, depending on the view are partially covered and seen from different perspective. Each scene is captured from 8 different angles. Scene 1 (denoted SceneEagle) uses mostly inorganic materials, such as wood, metal, plastic and glass all placed on the vertical stand. Scene 2 (SceneFruit) uses fruits normal and artificial, that are similar on HD photography and 3D cloud of point, but differs in hyperspectral image. Scene 3 (SceneFruit2) uses the fruits but also includes printed full colour images of same fruits that are 2-dimensional. The data are formatted as follows: - The HIS images are available in both \text{*.hdr} and \text{*.cube} formats. The separate files with calibrating panel is provided for each frame. - Kinect clouds are provided in \text{*.obj} format, typical for Kinect output files. - Matched Hyperspectral clouds are also provided as \text{*.obj} files - HD photo files are provided in \text{*.jpg} files. Acknowledgements This work has been supported by the National Science Centre, based on decision no. DEC2012/07/N/ST6/03656.