Stereo event- and frame-based benchmark dataset for scene understanding

This is a proof-of-concept labelled dataset for training semantic segmentation and pose estimation vision systems with neuromorphic event-based vision<br>Recorded data directories take the form<b> ./data/[prop]/[sample #]</b>. Prop meshes are in <b>./props/[prop]</b> in STL format. Each data directory contains three HDF5 files containing segmented RGB frames, segmented DVS events and prop pose information. For each 30 second recording, both raw and undistorted (after lens distortion correction) data is saved. RGB frame and visual event labels are integers, with 0 meaning `background', -1 meaning `ambiguous' (multiple props overlap) and any <i>i</i> &gt; 0 denoting class <i>i</i>. The fields of the RGB frame HDF5 file <b>frame.h5</b> are as follows, where <i>i</i> in {0, ..., <i>n</i> - 1} and <i>n</i> = 2 is the number of cameras:<br>timestamp_<i>i</i>: frame timestamp (camera <i>i</i>)<br>image_raw_<i>i</i>: frame before distortion correction (camera <i>i</i>)<br>image_undistorted_<i>i</i>: frame after distortion correction (camera <i>i</i>)<br>label_<i>i</i>: pixelwise frame label (camera <i>i</i>)<br>The fields of the DVS event HDF5 file <b>event.h5</b> are as follows:<br>timestamp_<i>i</i>: event timestamp (camera <i>i</i>)<br>polarity_<i>i</i>: event polarity (camera <i>i</i>)<br>xy_raw_<i>i</i>: event <i>x</i> and <i>y</i> before distortion correction (camera <i>i</i>)<br>xy_undistorted_<i>i</i>: event <i>x</i> and <i>y</i> after distortion correction (camera <i>i</i>)<br>label_<i>i</i>: event label (camera <i>i</i>)<br>Prop pose information, including translation and rotation (respectively millimetres and degrees) are stored as floating-point numbers for both global 3D tracking coordinates and camera-centric coordinates. Floating-point not-a-number indicates bad or missing data. The fields of the prop pose HDF5 file <b>pose.h5</b> are as follows:<br>timestamp: pose timestamp<br>extrapolated[<i>p</i>]: true when the pose of prop <i>p</i> was extrapolated<br>rotation[<i>p</i>]: prop <i>p</i> rotation in global coordinates<br>camera_rotation_<i>i</i>[<i>p</i>]: prop <i>p</i> rotation relative to camera <i>i</i><br>translation[<i>p</i>][<i>m</i>]: prop <i>p</i> marker <i>m</i> translation in global coordinates<br>camera_translation_<i>i</i>[<i>p</i>][<i>m</i>]: prop <i>p</i> marker <i>m</i> translation relative to camera <i>i</i><i><br></i>The data is in HDF5 format which is accessible with a number of tools eg PyTables in Python.<br><br><br>