Time Series Data of Gaze, Head Pose, Hand Pose, and Object Positions for Object Approaches with a Given Intention

This data set comprises time series data of gaze, head pose, hand pose, and object positions for object approaches with a given intention. The data was captured in the context of the following publication: <em>Michael Fennel, Serge Garbay, Antonio Zea, Uwe D. Hanebeck</em>, <strong>Intention Estimation with Recurrent Neural Networks for Mixed Reality Environments</strong>, Proceedings of the 26th International Conference on Information Fusion (Fusion 2023) <em>(under review)</em> A Microsoft Hololens 2 was used for recording the data at 60 fps under the modalities explained in detail in the above-mentioned paper. The file names are structured as follows: <em>1st/2nd:</em> The data with "1st" contains approaches to randomly placed objects on a grid, which are rendered in augmented reality. The user is informed about the object to approach using a visual cue. This corresponds to Section IV-A. The data with "2nd" contains approaches to real objects placed statically in a room. The user is informed about the object to approach using a voice command. <em>unfiltered:</em> Contains all approaches, including those where the user disrespects the given commands. Filtering is done as described in the paper. <em>train/val/test:</em> The first dataset was split in a 70/20/10 ratio for training, validation, and test. Each data set contains the following columns. In each approach, 5 objects numbered from i=0 to i=4 are present. General: <em>time:</em> in seconds <em>subject:</em> consecutive subject number <em>handedness:</em> left (1), right (0) <em>trial:</em> consecutive trial number per subject <em>target_label:</em> index of the object to approach (0 to 4) Data in world coordinates: <em>head_{x,y,z}:</em> head position <em>head_quat_{w,x,y,z}:</em> head orientation quaternion <em>W_gaze_{x,y,z}:</em> gaze direction <em>W_r_hand_{x,y,z}:</em> right hand position <em>W_r_hand_quat_{w,x,y,z}:</em> right hand orientation quaternion <em>W_l_hand_{x,y,z}:</em> left hand position <em>W_l_hand_quat_{w,x,y,z}:</em> left hand orientation quaternion <em>W_object_i_{x,y,z}:</em> position of object i <em>W_object_i_quat {w,x,y,z}</em>: orientation quaternion of object i Data in egocentric coordinates (head coordinate system). This data is provided for convenience and can be derived from the other data: <em>gaze_{x,y,z}:</em> gaze direction <em>r_hand_{x,y,z}:</em> right hand position <em>r_hand_quat_{w,x,y,z}:</em> right hand orientation quaternion <em>l_hand_{x,y,z}:</em> left hand position <em>l_hand_quat_{w,x,y,z}:</em> left hand orientation quaternion <em>object_i_{x,y,z}:</em> position of object i <em>object_i_quat {w,x,y,z}</em>: orientation quaternion of object i <strong>Acknowledgment:</strong> This work was supported by the ROBDEKON project of the German Federal Ministry of Education and Research.