Statistics of the dataset.

Multi-modal classification aims to extract pertinent information from various modalities to assign labels to instances. The advent of deep neural networks has significantly advanced this task. However, the majority of current deep neural networks lack interpretability, leading to skepticism. This issue is particularly pronounced in sensitive domains such as educational assessment. In order to address the trust deficit in deep neural networks for multi-modal classification tasks, we propose an Interpretable Multi-modal Classification framework (ICMC), which enhances confidence in the processes and outcomes of deep neural networks while maintaining interpretability and improving performance. Specifically, our approach incorporates a confidence-driven attention mechanism at the intermediate layer of the deep neural network, assessing attention scores and discerning anomalous information from both local and global perspectives. Furthermore, a confidence probability mechanism is implemented at the output layer, leveraging both local and global perspectives to bolster result confidence. Additionally, we meticulously curate multi-modal datasets for automatic lesson plan scoring research, making them openly available to the community. Quantitative experiments on educational and medical datasets confirm that ICMC outperforms state-of-the-art models (HMCAN, MCAN, HGLNet) by 2.5-6.0% in accuracy and 3.1-7.2% in F1-score, while reducing computational latency by 18%. Cross-domain validation demonstrates 15.7% higher generalizability than transformer-based approaches (CLIP), establishing its interpretability through attention visualization and confidence scoring.