Tracking behavior in naturalistic situations using artificial intelligence

Folder "Data for analysis" :<br> - Folder "Extracted_Data_For_Preprocessing" : raw data after extraction for all subjects (1 folder by subs) that contains OpenPose folder (json file for each frame) and OpenFace csv with the extracted metrics of a whole video. It also contains 1 file .tps / 1 file .syncIN / 1 file .txt (SliderRes) for the script 01_loader (see on the associated github repository) and 1 file .tps (logs of OpenFace).<br> - File "Data_For_Analysis_Michelot_2023.csv" : entire data after preprocessing used for analysis Folder "Examples" :<br> - Folder "Example Data To Preprocess" : raw data after extraction for one subject (test data) with OpenPose folder containing a json file for each frame and OpenFace csv containing the extracted metrics of a whole video. It also contains 1 file .tps / 1 file .syncIN / 1 file .txt (SliderRes) for the script 01_loader (see on the associated github repository) and 1 file .tps (logs of OpenFace).<br> - Example_Data_For_Analysis.csv : test data for analysis with reduce amount of data in all subjects. <em><strong>Abstract</strong> : Environmental context, e.g. social interactions, can shape our behavior, due to the range of stimuli it entails. Current research on this topic often relies on controlled laboratory settings, which may not fully capture the complexity and richness of real-life social situations.</em> <em>To overcome this limitation, we propose a new approach that leverages artificial intelligence (AI) tools, specifically computer vision, machine learning algorithms and explainability methods to extract, classify and highlight behaviors in naturalistic situations.<br> To validate our methodology, we conducted a study with sixteen participants in various naturalistic situations that varied socially and emotionally, including periods of silence, listening to music or neutral sounds, and listening to someone speaking about autobiographical or insignificant content in front of them.</em> <em>We adapted two open-source computer vision models, OpenFace and OpenPose, to extract objective whole-body metrics, such as skeletal and facial points, gaze, and facial action unit activations.<br> Applying two different machine learning models on the extracted behavioral metrics, extreme gradient boosting (XGBoost) and a long short-term memory neural network (LSTM), we accurately classified the situations up to around 90% accuracy for the social vs non-social situations.<br> Using advanced state-of-the-art explainability methods in conjunction with hypothesis-driven analysis, we identified distinct clusters or specific behaviors that were influenced by the social and emotional aspects of the situations. These behaviors included movements related to gaze, limbs, facial features, and facial expressions activations.<br> Our study highlights the potential of AI methodology to track behavior precisely and objectively in naturalistic contexts, paving the way for further research.</em>