Deep multimodal representations and classification of first-episode psychosis via live face processing

Schizophrenia is a severe psychiatric disorder associated with a wide range of cognitive and neurophysiological dysfunctions and long-term social difficulties. Early detection is expected to reduce the burden of disease by initiating early treatment. In this paper, we test the hypothesis that the integration of multiple simultaneous acquisitions of neuroimaging, behavioral, and clinical information will be better for the prediction of early psychosis than unimodal recordings. We propose a novel framework to investigate the neural underpinnings of the early psychosis symptoms (that can develop into Schizophrenia with age) using multimodal acquisitions of neural and behavioral recordings including functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), and facial features. Our data acquisition paradigm is based on live face-to-face interaction in order to study the neural correlates of social cognition in first-episode psychosis (FEP). We propose a novel deep repre..., The proposed method employs dyads that include one individual who serves as the live expressive face stimulus and the other partner categorized as either typically developed (TD) or first episode psychosis (FEP) patient. Dyads faced each from across a table at a distance of approximately 140 cm and table-mounted eye-tracking systems were positioned to measure continuous eye movements of the subject. Functional NIRS and EEG data were also synchronized and continuously acquired hemodynamic and electrocortical responses of the subject during the experiment. The dyads were separated by a “smart glass” in the center of the table that controlled face gaze times (the glass was transparent during gaze periods) and “rest times” (the glass was opaque during rest) (Hirsch, X. Zhang, Noah, and Bhattacharya, 2023).
Paradigm
The dyads were seated 140 cm across a table from each other. A \"Smart Glass\" (glass that is capable of alternating its appearance between opaque and transparent upon applica..., , # Deep multimodal representations and classification of first-episode psychosis via live face processing

This readme file was generated on 2025-02-18 by Rahul Singh

## GENERAL INFORMATION

Author Information
Name: Rahul Singh
Institution: Yale University
Email: [[e-mail]](mailto:[e-mail])

Principal Investigator Information
Name: Joy Hirsch
ORCID: 0000-0002-1418-6489
Institution: Yale School of Medicine
Email: [[e-mail]](mailto:[e-mail])

Principal Investigator Information
Name: Smita Krishnaswamy
Institution: Wu Tsai Institute, Yale University
Email: [[e-mail]](mailto:[e-mail])

Author/Alternate Contact Information
Name: J. Adam Noah
ORCID: 0000-0001-9773-2790
Institution: Yale School of Medicine
Email: [[e-mail]](mailto:[e-mail])

Date of data collection: Approximate collection dates are 2022-01 through 2025-02.

## SHARING/ACCESS INFORMATION

Recommended citation for this dataset:
Hirsc...,