Community Embedded Robotics: Vid2Real A Real-World Dataset about Perceived Social Intelligence in Human Robot Encounters

Introduction This dataset was gathered during the Vid2Real real-world study as part of the Vid2Real project, which investigates humans’ perception of robots' intelligence in the context of an incidental Human-Robot encounter. The dataset contains (1) participants' questionnaire responses about their experience as pedestrians during an incidental encounter with a robot, as well as (2) data from the robot sensors. In this specific encounter scenario, the robot is trying to enter a building and is asking for help from the pedestrian in two study conditions. The different conditions were manipulated using the robot’s verbal and expressive movement functionalities. This dataset corresponds to the second phase of the Vid2Real project. The dataset from the first phase, namely the Vid2Real online video-based study, was used to design the real-world study and is published in this repository. Dataset Purpose The dataset includes the participants' responses to validate the hypothesis that robots' social intelligence is positively correlated with human compliance (i.e., humans' willingness to follow robots' orders) in an incidental human-robot encounter context. Further, the sensor data from the robot is provided to facilitate future research in making human-robot encounters fully autonomous. Dataset Organization This dataset is about 33 GB in size and has roughly 11.5k files, organized as follows: The root folder: Vid2RealPhase2_Dataset HRI relevant data: HumanSubjectsData Questionnaire used in Phase 2 (Vid2Real_Questionnaire.pdf) Data Dictionary used in Phase 2 (Vid2RealPhase2_DataDictionary.pdf) Responses (Vid2RealPhase2_Responses.tab) Videos of the scene recordings from GoPro (SceneRecordings) Participant 1 Video (Vid2RealPhase2_P1video.mp4) Participant 2 Video (Vid2RealPhase2_P2video.mp4) Participant 3 Video (Vid2RealPhase2_P3video.mp4) ... (25 such videos) Robot data: RobotData Participant 1 (Robot_P1) Raw ROS bagfile (Vid2RealPhase2_P1rosbag.bag) Robot perspective video (Vid2RealPhase2_robotP1video.mp4) Sensor calibrations Robot frames description (description.txt) LIDAR extrinsic matrix (Vid2RealPhase2_P1calib_vlp16_ext.yaml) IMU extrinsic matrix (Vid2RealPhase2_P1calib_vectornav_ext.yaml) Camera extrinsic matrix (Vid2RealPhase2_P1calib_kinect_ext.yaml) Camera intrinsic matrix (Vid2RealPhase2_P1calib_kinect_int.yaml) Velodyne (VLP16) lidar data (Lidar_3d_raw_pc) First timestep reading (Vid2RealPhase2_P1_vlp16_0001.bin) Second timestep reading (Vid2RealPhase2_P1_vlp16_0002.bin) ... (many such .bin files) The timestamps text file (Vid2RealPhase2_P1_vlp16_timestamps.txt) Inertial measurement unit (IMU) data (Vid2RealPhase2_P1_vectornavIMU.txt) Camera data, i.e., images that were recorded from the camera on robot First timestep reading (Vid2RealPhase2_P1_kinect_0001.png) Second timestep reading (Vid2RealPhase2_P1_kinect_0002.png) ... (many such .png files) The timestamps text file (Vid2RealPhase2_P1_kinect_timestamps.txt) Participant 2 (Robot_P2) Participant 3 (Robot_P3) ... (26 such subfolders) Data Availability Matrix The data availability matrix represents the different pieces of robot and human subjects data we provide for each participant. It is summarized as follows: bagfile: the raw ROS bagfile recording robot cam video: the video recording obtained from the front Kinect camera on the robot vlp: extracted binary files for the pointcloud data obtained from VLP16 lidar on the robot kinect: RGB-images from the Azure Kinect camera in the front of the robot imu: pose and twist data from the Vectornav IMU sensor on the robot lidarext: lidar extrinsics w.r.t. robot base imuext: IMU extrinsics w.r.t. robot base camext: camera extrinsics w.r.t. robot base camint: camera instrinsics response: participant responses to the on-site questionnaire scene video: the video recording obtained from GoPro camera placed at an appropriate distance to record the full human-robot interaction Due to technical reasons, we were unable to record certain pieces of robot data for a few participant sessions (i.e., 1, 2, 4, 5, 11, 17, 23, 28). Additionally, the robot malfunctioned during the sessions of P13 and P15, therefore, their survey responses were not collected. Lastly, P7 did not consent to be video recorded, but the survey responses were collected. In short, we have 26 valid survey responses and 25 video recordings. Study Conditions There are 2 study conditions Baseline: The robot walks up to the entrance and waits for the pedestrian to open the door without any additional behaviors. This is also the "control" condition. Body Language + Verbal: The robot walks up to the entrance, turns its head to look at the pedestrian and say, "Can you please open the door for me?" and then wait for the pedestrian to open the door. A between-subject design was adopted to better reflect the nature of real-world scenarios. Participants were randomized to partake in one study condition. After receiving their consent to be part of the study, the participants were instructed to act as pedestrians and walk. After encountering the robot, they could react to it in whatever way they felt natural. In the case where the participants did not help the robot enter the building, the study manager would intervene and help the robot. The study session was considered finished when the robot entered the building. We encourage you to look at the videos in this dataset to understand the study scenario better. Instruments The questionnaires used in the study include the Perceived Social Intelligence Scale (PSI), Godspeed Questionnaire, and Anthropomorphism Questionnaire (AMPH). Participant demographic data was also collected. Questionnaire items are attached as part of this dataset. Human Subjects Participants were recruited through emails as well as in-person from people who walked by the study site. Among the 26 participants, 13 identified as female and 13 as male; the ages ranged from 18 to 26 (M = 21, SD = 3.01). Their names remained anonymous. Please note that even though the participants' faces are identifiable information, all but one participant agreed to be recorded. Therefore, there are only 25 video recordings in this dataset. Participants were eligible to enter a draw to win a $100 gift card upon study completion. The win rate was approximately 1 in 10. This study was reviewed and approved by UT Austin Internal Review Board. Robot Data Collection The dataset contains data about humans’ perceived social intelligence of a Boston Dynamics’ quadruped robot Spot (Explorer model). The robot was selected because quadruped robots are gradually being adopted to provide services such as delivery, surveillance, and rescue. However, there are still issues or obstacles that robots cannot easily overcome by themselves, and they will have to ask for help from nearby humans. Therefore, it is important to understand how humans react to a quadruped robot that they incidentally encounter. For the purposes of this study, the robot was teleoperated. We also recorded data from three sensors on the robot : Lidar: VLP16 Velodyne lidar located at the top of the robot. Camera: Azure Kinect RGBD Camera located at the front of the robot. However, we only use the RGB channels for this study. IMU: Vectornav IMU is also located at the top of the robot. Human Subjects Data Collection The data was collected through Qualtrics, a survey development platform. After the completion of data collection, the data was downloaded as a CSV file. The participants' ages ranged from 18 to 28 (M = 21, SD = 3.01), and all of them were students at The University of Texas at Austin. Data Quality Control For the human subjects data, the Qualtrics platform automatically detects bots, so any response flagged as bots is discarded. All incomplete and duplicate responses were discarded. For the robot data, we provide the raw bagfiles as well as extracted images and pointcloud information. The sensor calibration files which are needed for correlating data across sensors is included. The calibration files provide the ‘transformations’ (i.e., distances and orientation information) between the robot base and different sensors on the robot, which help in sensor fusion and expands the use of this dataset. Data Usage This dataset can be used to conduct a meta-analysis on robots' perceived intelligence. Please note that data is coupled with this study design. The robot data can be used to develop autonomous modules on human-robot encounters. Acknowledgement This study was funded through the NSF Award # 2219236GCR: Community-Embedded Robotics: Understanding Sociotechnical Interactions with Long-term Autonomous Deployments.