HRI-SENSE: A Multimodal Dataset on Social and Emotional Responses to Robot Behaviour

This is the dataset for the paper "HRI-SENSE: A Multimodal Dataset on Social and Emotional Responses to Robot Behaviour" – available at the Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction.

The dataset captures various modalities of user behaviour (facial landmarks, facial action units, head pose, body pose landmarks, depth footage) exhibited by participants interacting with a TIAGo Steel robot following different behaviour models on a collaborative manipulation "Burger Assembly" task. Details of the task can be found in the paper. The non-verbal modalities are complemented by explicit feedback signals (verbal dialogue transcripts), robot joint movements data, interaction event labels and self-assessed questionnaires (pre-study and post-interaction questionnaires) on sociodemographics and perceived user impressions (e.g. frustration, satisfaction).

The dataset's contents have been collected from over 6 hours of verbal and physical human-robot interactions in over 146 sessions with 18 participants.

Data Modalities

User facial expression data (facial landmarks, facial action units and head pose) have been calculated by OpenFace:

head location: pose_Tx, pose_Ty, pose_Tz

head rotation: pose_Rx, pose_Ry, pose_Rz

facial landmark 3d locations: X_0, ... X_67, Y_0,...Y_67, Z_0,...Z_67

facial action units intensity (r) and presence (c): AU01_r, AU02_r,...AU25_r, AU26_r, AU45_r, AU01_c, AU02_c,...AU28_c, AU45_c

User pose landmarks have been calculated by MediaPipe Pose Landmarker:

3d coordinates of 33 pose landmarks

Verbal dialogue transcripts have been produced by OpenAI's Whisper model.

Depth information and robot joint data has been recorded by the TIAGo robot's sensors.

Self-assessed questionnaire data has been collected using a 5-point Likert scale, following question items and practices established in previous Human-Robot Interactions and Psychology works. Details of this can be found in the paper.

For more details on the data collection and processing pipeline, please see the paper.

Contents

The dataset is organised as:

README.md

questionnaire-data/

pre-study-questionnaire/

pre-study-questionnaire.pdf

pre-study-questionnaire-responses.csv: participant-id,A1-age,A2-occupation,B1,B2,B3,B4,B5,B6,C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12

post-interaction-questionnaire/

post-interaction-questionnaire.pdf

post-interaction-questionnaire-responses.csv: participant-id,model-id,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8,Q9,Q10,Q11,Q12,Q13,Q14,Q15,Q16,Q17

sensory-data/

depth-data/

P[participant_id]-M[model_id]-[trial_number].mp4

user-face-data/

P[participant_id]-M[model_id]-[trial_number].csv: timestamp,confidence,success,pose_Tx,pose_Ty,pose_Tz,pose_Rx,pose_Ry,pose_Rz,X_0,X_1,X_2,X_3,X_4,X_5,X_6,X_7,X_8,X_9,X_10,X_11,X_12,X_13,X_14,X_15,X_16,X_17,X_18,X_19,X_20,X_21,X_22,X_23,X_24,X_25,X_26,X_27,X_28,X_29,X_30,X_31,X_32,X_33,X_34,X_35,X_36,X_37,X_38,X_39,X_40,X_41,X_42,X_43,X_44,X_45,X_46,X_47,X_48,X_49,X_50,X_51,X_52,X_53,X_54,X_55,X_56,X_57,X_58,X_59,X_60,X_61,X_62,X_63,X_64,X_65,X_66,X_67,Y_0,Y_1,Y_2,Y_3,Y_4,Y_5,Y_6,Y_7,Y_8,Y_9,Y_10,Y_11,Y_12,Y_13,Y_14,Y_15,Y_16,Y_17,Y_18,Y_19,Y_20,Y_21,Y_22,Y_23,Y_24,Y_25,Y_26,Y_27,Y_28,Y_29,Y_30,Y_31,Y_32,Y_33,Y_34,Y_35,Y_36,Y_37,Y_38,Y_39,Y_40,Y_41,Y_42,Y_43,Y_44,Y_45,Y_46,Y_47,Y_48,Y_49,Y_50,Y_51,Y_52,Y_53,Y_54,Y_55,Y_56,Y_57,Y_58,Y_59,Y_60,Y_61,Y_62,Y_63,Y_64,Y_65,Y_66,Y_67,Z_0,Z_1,Z_2,Z_3,Z_4,Z_5,Z_6,Z_7,Z_8,Z_9,Z_10,Z_11,Z_12,Z_13,Z_14,Z_15,Z_16,Z_17,Z_18,Z_19,Z_20,Z_21,Z_22,Z_23,Z_24,Z_25,Z_26,Z_27,Z_28,Z_29,Z_30,Z_31,Z_32,Z_33,Z_34,Z_35,Z_36,Z_37,Z_38,Z_39,Z_40,Z_41,Z_42,Z_43,Z_44,Z_45,Z_46,Z_47,Z_48,Z_49,Z_50,Z_51,Z_52,Z_53,Z_54,Z_55,Z_56,Z_57,Z_58,Z_59,Z_60,Z_61,Z_62,Z_63,Z_64,Z_65,Z_66,Z_67,AU01_r,AU02_r,AU04_r,AU05_r,AU06_r,AU07_r,AU09_r,AU10_r,AU12_r,AU14_r,AU15_r,AU17_r,AU20_r,AU23_r,AU25_r,AU26_r,AU45_r,AU01_c,AU02_c,AU04_c,AU05_c,AU06_c,AU07_c,AU09_c,AU10_c,AU12_c,AU14_c,AU15_c,AU17_c,AU20_c,AU23_c,AU25_c,AU26_c,AU28_c,AU45_c

user-pose-data/

P[participant_id]-M[model_id]-[trial_number]-[camera_id].csv: time,L0-x,L0-y,L0-z,L1-x,L1-y,L1-z,L2-x,L2-y,L2-z,L3-x,L3-y,L3-z,L4-x,L4-y,L4-z,L5-x,L5-y,L5-z,L6-x,L6-y,L6-z,L7-x,L7-y,L7-z,L8-x,L8-y,L8-z,L9-x,L9-y,L9-z,L10-x,L10-y,L10-z,L11-x,L11-y,L11-z,L12-x,L12-y,L12-z,L13-x,L13-y,L13-z,L14-x,L14-y,L14-z,L15-x,L15-y,L15-z,L16-x,L16-y,L16-z,L17-x,L17-y,L17-z,L18-x,L18-y,L18-z,L19-x,L19-y,L19-z,L20-x,L20-y,L20-z,L21-x,L21-y,L21-z,L22-x,L22-y,L22-z,L23-x,L23-y,L23-z,L24-x,L24-y,L24-z,L25-x,L25-y,L25-z,L26-x,L26-y,L26-z,L27-x,L27-y,L27-z,L28-x,L28-y,L28-z,L29-x,L29-y,L29-z,L30-x,L30-y,L30-z,L31-x,L31-y,L31-z,L32-x,L32-y,L32-z

robot-joint-data/

P[participant_id]-M[model_id]-[trial_number].csv: time, arm_1_joint, arm_2_joint, arm_3_joint, arm_4_joint, arm_5_joint, arm_6_joint, arm_7_joint 

dialogue-transcript/

P[participant_id]-M[model_id]-[trial_number].csv: start,end,text

interaction-event-labels/

P[participant_id]-M[model_id]-[trial_number].csv: event,start,end

Limitations

Due to recording sensor malfunctions and processing library limitations, not all interaction scenario data contains all modalities. Verbal dialogue transcripts or user pose data may be incomplete or missing for a small number of interactions or recording angles.

Acknowledgements

This work was supported by UK Research and Innovation [EP/S024298/1].