PepperMint Role

In multi-speaker human-robot interactions (HRI), accurately identifying who is speaking (Active Speaker Detection - ASD) and whom they are addressing (Addressee Estimation - AE) is essential for managing natural, fluid interactions. While existing large-scale audio-visual datasets for ASD have advanced model performance by facilitating comparisons across various speakers, settings, and recording devices, these datasets primarily focus on controlled genres of speech (e.g., films, podcasts) where interactions are often scripted or semi-planned. In contrast, spontaneous interactions in public settings present significant challenges, including background noise, multiple speakers, and diverse facial orientations. Moreover, existing addressee datasets are often limited in scope, typically restricted to fixed number of participants and lacking the diversity required to model real-world, dynamic interactions.To address these challenges in real-world, unstructured HRI scenarios, we introduce PepperMint Role, a multimodal dataset featuring audio-visual annotations of both active speakers and addressees (either robot or human) during interactions between a service robot and students in a university library.