iChores: A Role-Played Human-Robot Interaction Dataset for Gesture-Based Object and Task Instruction

Summary The iChores dataset captures real-world human-to-human instructional interactions designed to simulate human-robot interaction (HRI) in a domestic/home environment. Under the iChores experimental framework, 31 human participants act as "teachers" to instruct a researcher role-playing as an adaptive service robot in 2024. Dataset consists of 942 RGB 2D videos. The tasks are situated within a kitchen context, progressing from teaching the robot to recognize and name various food objects and utensils, up to advanced, step-by-step instructions on how to physically prepare a bowl of fruits. The dataset is uniquely structured to analyze visual intent, gesture mimicry, and the physical geometry of interactive teaching loops without relying on audio cues. Key Features Home Environment Context: Situated entirely around common kitchen tasks, making it ideal for assistive and domestic service robotics research. Multi-Modal Gestures: Includes dense instances of pointing, framing, and guiding gestures paired with precise physical object arrangements. Progressive Task Complexity: Moves logically from basic object/feature identification (naming fruits, colors, and sizes) to high-level sequential task execution (assembling a fruit bowl amid heavy clutter). Structured Hierarchy: 34 trials per participant divided into 4 progressive levels: Object Features, Object Relations, Object Combinations, and High-Density Task Combinations. Privacy Compliant: All participant facial regions are systematically anonymized. Dataset Specifications Data Type: Strictly RGB 2D Video-Only (No Audio) Camera Angle: Third-person perspective from behind the role-playing agent Participants: N = 31 (24 women, 7 men; ages 18–51) Total Video Files: 942 videos File Structure & Documentation Note For full details on the experimental protocol, precise object set configurations, and the file naming convention, please refer to the comprehensive "ichores_dataset_description.docx" included directly within the downloaded archive. Funding: Dataset creation was funded by the National Science Centre, Poland under the OPUS call in the Weave programme under the project number 2021/43/I/ST6/02489. If you utilize the iChores dataset in your research, academic publications, or projects, please cite it using the following reference: 10.5281/zenodo.20760992 Acknowledgments: This data was gathered under the "iChores" experimental framework. We would like to thank the participants from Jagiellonian University and the role-playing researchers who consented to make this research dataset possible.