Ablation Study results on SDTI dataset.

Sentiment analysis methods aim to evaluate users’ mental health conditions by analyzing their posted content (text, images, and audio) on social networks. However, given the diversity and complexity of social media information, traditional single-modal sentiment analysis techniques exhibit limitations in accurately interpreting users’ emotional states and may even lead to contradictory conclusions. To address this challenge, this paper proposes a Feature Fusion Based Transformer (FFBT) solution. The framework consists of three key steps: Firstly, RoBERTa and ResNet50 models are employed to extract features from textual and image data in social media posts, respectively. Then, a multimodal Transformer architecture facilitates feature alignment and fusion across different modalities. Finally, the fused features are fed into a fully connected network (FCN) for sentiment classification, ultimately determining the user’s emotional state. Experiments conducted on a custom dataset constructed from social media platform data demonstrate that FFBT outperforms existing sentiment analysis algorithms by 4.1% in accuracy and 5% in F1-scores, respectively.