REFLEX Dataset: A Multimodal Dataset of Human Reactions to Robotic Failures and Subsequent Robotic Explanations.

REFLEX Dataset is a comprehensive collection of multimodal Human Behavioral reactions to Robot Failures and Explanations. The version 1.0 is a representative sample of this dataset with the reactions from 5 users out of a total 55 users.

This version 1.1.0 is the full dataset with the reactions from a total 55 users.Please refer to the Readme in the zipped file for further information.

This data was recorded from a user study and has been processed for anonymization.

About Data

This description gives a detailed process on how the data was collected. It should describe the conditions under which the data was recorded and also the devices used to record the data.

Data Organisation

The data is structured by strategy and participant, as shown below:

Strategy Dir/
-Participant Dir/
- analysis
- questonnaire
- facetorch
- openface
- gaze
- hume
- body
- voice
- time
- video_cam1
- video_cam2

We employed five different strategies (C1, C2, C3, D1, D2), collecting data from 11 participants for each strategy. The data for each participant is organized within a corresponding folder.

Participants are labeled based on their assigned strategy. For example, data from the first participant under the “Fixed Low” (C1) strategy can be found in the C1-1 subfolder within the C1 directory.

Collected Data

Each participant folder contains various datasets related to different modalities. All visual data are collected using the camera 1 video. The collected data are outlined below:

Anonymized Videos (video_cam1.mp4, video_cam2.mp4) - Visual Representation:

Video from camera 1 (robot side of view)

Video from camera 2 (experiment side of view)

Analysis (analysis.csv) - Failure Instance Description:

Failure type

Explanation strategy

Explanation level

Phase (Pre, Failure, Explanation, Resolution)

Start/End frame and time of failure

Task Resolved

Questionnaire (questionnaire.csv) - Failure Instance Description:

Participant Data (Age, Gender, etc)

Answers of explanation-satisfaction rate question for rounds and overall experiment

Facetorch (facetorch.csv) - Facetorch - Face:

Arousal/Valence levels

Presence of Facial Action Units (AUs)

Dominant Emotion (Out of six basic emotions and neutral)

OpenFace (openface.csv) - OpenFace - Face, Gaze, Head:

Eye Gaze (2D and 3D Landmarks)

Eye Direction (vector and in radians)

Head Pose Estimation (Pose Estimation, Rotation)

Face Landmarks (2D and 3D Landmarks)

Facial Action Units (0.0-1.0 intensity scores, occurrences)

Gaze (gaze.csv) - Gaze:

Eye Gaze Classification (e.g., Robot, Task, Miscellaneous)

Hume (hume.csv) - Hume Expression Measurement API - Face:

48 Emotion likelihoods

Facial Action Units (0.0-1.0 score)

Facial Descriptions (0.0-1.0 score)

Voice (speech.csv) - Hume Expression Measurement API - Speech:

Speech conversation data

Emotional likelihoods inferred from prosody

Body (body.csv) - MediaPipe Pose Landmark Detection - Body:

Pose classifications (e.g., crossed arms, arms behind back)

2D and 3D Pose Landmarks

Time (time.csv) - MediaPipe Pose Landmark Detection:

Associated timestamp and time for each frame of camera 1 video.

Notes

Data was synchronized based on the `video_cam1.mp4`

The `hume.csv` and `gaze.csv` files contain data only for frames within failure periods.

Failure events were divided into four phases:    1. Pre-failure phase: Period before the failure occurs    2. Failure phase: When the actual failure action takes place    3. Explanation phase: When the robot provides an explanation for the failure    4. Resolution phase: When the robot guides the participant to resolve the issue

How to Visualize Participant Data

Please visit the github repository: https://github.com/andreasnaoum/reflex-viz