Tracking behavior in naturalistic situations using artificial intelligence

Data preprocessed used for analysis in the article "Tracking behavior in naturalistic situations using artificial intelligence"

Abstract : Environmental context, e.g. social interactions, can shape our behavior, due to the range of stimuli it entails. Current research on this topic often relies on controlled laboratory settings, which may not fully capture the complexity and richness of real-life social situations.

To overcome this limitation, we propose a new approach that leverages artificial intelligence (AI) tools, specifically computer vision, machine learning algorithms and explainability methods to extract, classify and highlight behaviors in naturalistic situations.
To validate our methodology, we conducted a study with sixteen participants in various naturalistic situations that varied socially and emotionally, including periods of silence, listening to music or neutral sounds, and listening to someone speaking about autobiographical or insignificant content in front of them.

We adapted two open-source computer vision models, OpenFace and OpenPose, to extract objective whole-body metrics, such as skeletal and facial points, gaze, and facial action unit activations.
Applying two different machine learning models on the extracted behavioral metrics, extreme gradient boosting (XGBoost) and a long short-term memory neural network (LSTM), we accurately classified the situations up to around 90% accuracy for the social vs non-social situations.
Using advanced state-of-the-art explainability methods in conjunction with hypothesis-driven analysis, we identified distinct clusters or specific behaviors that were influenced by the social and emotional aspects of the situations. These behaviors included movements related to gaze, limbs, facial features, and facial expressions activations.
Our study highlights the potential of AI methodology to track behavior precisely and objectively in naturalistic contexts, paving the way for further research.