Deep multimodal representations and classification of first-episode psychosis via live face processing

Schizophrenia is a severe psychiatric disorder associated with a wide
range of cognitive and neurophysiological dysfunctions and long-term
social difficulties. Early detection is expected to reduce the burden of
disease by initiating early treatment. In this paper, we test the
hypothesis that the integration of multiple simultaneous acquisitions of
neuroimaging, behavioral, and clinical information will be better for the
prediction of early psychosis than unimodal recordings. We propose a novel
framework to investigate the neural underpinnings of the early psychosis
symptoms (that can develop into Schizophrenia with age) using multimodal
acquisitions of neural and behavioral recordings including functional
near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), and
facial features. Our data acquisition paradigm is based on live
face-to-face interaction in order to study the neural correlates of social
cognition in first-episode psychosis (FEP). We propose a novel deep
representation learning framework, Neural-PRISM, for learning joint
multimodal compressed representations combining neural as well as
behavioral recordings. These learned representations are subsequently used
to describe, classify, and predict the severity of early psychosis in
patients, as measured by the Positive and Negative Syndrome Scale (PANSS)
and Global Assessment of Functioning (GAF) scores to evaluate the impact
of symptomatology. We found that incorporating joint multimodal
representations from fNIRS and EEG along with behavioral recordings
enhances classification between typical controls and FEP individuals
(significant improvements between 10 − 20%). Additionally, our results
suggest that geometric and topological features such as curvatures and
path signatures of the embedded trajectories of brain activity enable the
detection of discriminatory neural characteristics in early psychosis.