Investigating the Role of Implicit Signals in Adaptive User-Aware Human-Robot Interactions: A Multimodal Dataset

The paper "Investigating the Role of Implicit Signals in Adaptive User-Aware Human-Robot Interactions", associated with this dataset, has been accepted for publication in the 2026 IEEE International Conference on Robotics & Automation (ICRA 2026). The dataset captures various modalities of user behaviour (facial landmarks, facial action units, head pose and body pose landmarks) exhibited by participants interacting with a Pepper robot following different behaviour models on a social human-robot interaction task taking place in a cafeteria-style scenario. Details of the scenario can be found in the paper. The non-verbal modalities are complemented by explicit feedback signals captured in the form of verbal dialogue transcripts. The dataset's contents have been collected from over 5 hours of verbal and physical human-robot interactions with 20 participants. Data Modalities User facial expression data (facial landmarks, facial action units and head pose) have been calculated by OpenFace: head location: pose_Tx, pose_Ty, pose_Tz head rotation: pose_Rx, pose_Ry, pose_Rz facial landmark 3d locations: X_0, ... X_67, Y_0,...Y_67, Z_0,...Z_67 facial action units intensity (r) and presence (c): AU01_r, AU02_r,...AU25_r, AU26_r, AU45_r, AU01_c, AU02_c,...AU28_c, AU45_c User pose landmarks have been calculated by MediaPipe Pose Landmarker: 3d coordinates of 33 pose landmarks Verbal dialogue transcripts have been produced by OpenAI's Whisper model. For more details on the data collection and processing pipeline, please see the paper. Contents The dataset is organised as: README.md sensory-data/ user-face-data/ P[participant_id]-C[camera_id]-[trial_number].csv: timestamp,confidence,success,pose_Tx,pose_Ty,pose_Tz,pose_Rx,pose_Ry,pose_Rz,X_0,X_1,X_2,X_3,X_4,X_5,X_6,X_7,X_8,X_9,X_10,X_11,X_12,X_13,X_14,X_15,X_16,X_17,X_18,X_19,X_20,X_21,X_22,X_23,X_24,X_25,X_26,X_27,X_28,X_29,X_30,X_31,X_32,X_33,X_34,X_35,X_36,X_37,X_38,X_39,X_40,X_41,X_42,X_43,X_44,X_45,X_46,X_47,X_48,X_49,X_50,X_51,X_52,X_53,X_54,X_55,X_56,X_57,X_58,X_59,X_60,X_61,X_62,X_63,X_64,X_65,X_66,X_67,Y_0,Y_1,Y_2,Y_3,Y_4,Y_5,Y_6,Y_7,Y_8,Y_9,Y_10,Y_11,Y_12,Y_13,Y_14,Y_15,Y_16,Y_17,Y_18,Y_19,Y_20,Y_21,Y_22,Y_23,Y_24,Y_25,Y_26,Y_27,Y_28,Y_29,Y_30,Y_31,Y_32,Y_33,Y_34,Y_35,Y_36,Y_37,Y_38,Y_39,Y_40,Y_41,Y_42,Y_43,Y_44,Y_45,Y_46,Y_47,Y_48,Y_49,Y_50,Y_51,Y_52,Y_53,Y_54,Y_55,Y_56,Y_57,Y_58,Y_59,Y_60,Y_61,Y_62,Y_63,Y_64,Y_65,Y_66,Y_67,Z_0,Z_1,Z_2,Z_3,Z_4,Z_5,Z_6,Z_7,Z_8,Z_9,Z_10,Z_11,Z_12,Z_13,Z_14,Z_15,Z_16,Z_17,Z_18,Z_19,Z_20,Z_21,Z_22,Z_23,Z_24,Z_25,Z_26,Z_27,Z_28,Z_29,Z_30,Z_31,Z_32,Z_33,Z_34,Z_35,Z_36,Z_37,Z_38,Z_39,Z_40,Z_41,Z_42,Z_43,Z_44,Z_45,Z_46,Z_47,Z_48,Z_49,Z_50,Z_51,Z_52,Z_53,Z_54,Z_55,Z_56,Z_57,Z_58,Z_59,Z_60,Z_61,Z_62,Z_63,Z_64,Z_65,Z_66,Z_67,AU01_r,AU02_r,AU04_r,AU05_r,AU06_r,AU07_r,AU09_r,AU10_r,AU12_r,AU14_r,AU15_r,AU17_r,AU20_r,AU23_r,AU25_r,AU26_r,AU45_r,AU01_c,AU02_c,AU04_c,AU05_c,AU06_c,AU07_c,AU09_c,AU10_c,AU12_c,AU14_c,AU15_c,AU17_c,AU20_c,AU23_c,AU25_c,AU26_c,AU28_c,AU45_c user-pose-data/ P[participant_id]-C[camera_id]-[trial_number]-pose.csv: time,L0-x,L0-y,L0-z,L1-x,L1-y,L1-z,L2-x,L2-y,L2-z,L3-x,L3-y,L3-z,L4-x,L4-y,L4-z,L5-x,L5-y,L5-z,L6-x,L6-y,L6-z,L7-x,L7-y,L7-z,L8-x,L8-y,L8-z,L9-x,L9-y,L9-z,L10-x,L10-y,L10-z,L11-x,L11-y,L11-z,L12-x,L12-y,L12-z,L13-x,L13-y,L13-z,L14-x,L14-y,L14-z,L15-x,L15-y,L15-z,L16-x,L16-y,L16-z,L17-x,L17-y,L17-z,L18-x,L18-y,L18-z,L19-x,L19-y,L19-z,L20-x,L20-y,L20-z,L21-x,L21-y,L21-z,L22-x,L22-y,L22-z,L23-x,L23-y,L23-z,L24-x,L24-y,L24-z,L25-x,L25-y,L25-z,L26-x,L26-y,L26-z,L27-x,L27-y,L27-z,L28-x,L28-y,L28-z,L29-x,L29-y,L29-z,L30-x,L30-y,L30-z,L31-x,L31-y,L31-z,L32-x,L32-y,L32-z dialogue-transcript/ P[participant_id]-[trial_number].csv: start,end,text Limitations Due to recording sensor malfunctions and processing library limitations, not all interaction scenario data contains all modalities. Verbal dialogue transcripts or user pose data may be incomplete or missing for a small number of interactions or recording angles. Acknowledgements This work was supported by UK Research and Innovation [EP/S024298/1].