Human-Human Commensality Dataset

A novel audio-visual dataset capturing human social eating behaviors of groups of three people sharing a meal. It contains multi-view RGBD video and directional audio recordings of 30 sessions, totaling over 18 hours of multistream, multimodal recordings of 90 people, and provides the following data. ROS bags with topics: 4x mic audio, mixed audio, sound direction, per-participant RGBD, and scene RGBD. Raw data (extracted from ROS bags): scene audio, sound direction, per-participant videos, and scene videos. Processed data (extracted from raw data): per-participant speaking status, per-participant face and body keypoints from OpenPose, per-participant gaze, and head pose from RT-GENE, per-participant bite count, and per-participant times since the last bite lifted and since the last bite delivered to mouth. Annotations: per-participant interactions with food, drink, and napkins (all entered, lifted, delivered to mouth, and mouth open events), per-participant food type labels, and observations of interesting behaviors. Please see Section 5 (and the referenced Appendix sections) of our paper for the data collection study setup, data annotation details, pre- and post-study questionnaires, and data statistics.