Dataset.zip

Facial Expression Recognition
Introduction This project aims to classify the five expressions, angry, happy, neutral, sad, and surprised available in the VIRI database, and three different expressions, fear, disgust, and happy available in the NVIE database.
All the models are implemented using Pytorch Process for training First, we import the VIRI and NVIE database and divided them into 3 parts, train-set, val-set and test-set by using 70%, 15% and 15% split-ratio. Then we create our model architectures. For visible modalities, we modified the ResNet-18 CNN architecture. In the final layer of ResNet-18, we incorporated several dropout layers, fully connected (FC) layers, and a SoftMax layer tailored to the visible dataset. For infrared modalities, again we modified the ResNet-18 CNN architecture. In the final layer of ResNet-18, we incorporated several dropout layers, fully connected (FC) layers, activation functions, batch norm layers and a SoftMax layer tailored to the infrared dataset. For visible and infrared, we combined the features pair-wise and again modified the final layer by incorporating several dropout, fully connected layers, batch norm layers and a softmax layer.
Evaluation Method To comprehensively evaluate the performance of our proposed facial expression recognition model, we employed several standard metrics and performed extensive experiments on two publicly accessible datasets: VIRI and NVIE. Datasets VIRI Database: Contains 566 image pairings of five distinct facial expressions (angry, happy, neutral, sad, surprised) captured in visible and infrared modalities. NVIE Database: Contains visible and infrared images from over 100 subjects, capturing six fundamental facial expressions (happy, sad, surprised, fear, rage, disgust). Experimental Setup Data Splitting: Each dataset was divided into training (70%), validation (15%), and test (15%) sets. Data augmentation techniques such as rotations, zooming, distortion, shear, and flipping were applied to the training sets to increase the diversity of the training examples and mitigate overfitting. Training: The ResNet-18 model, modified with an attention mechanism, was trained separately on visible and infrared images, as well as on a combined dataset using an early fusion approach. We utilized the Adam optimizer with a CosineAnnealingLR scheduler, and set the batch size to 64. The loss function used for training was cross-entropy. Evaluation Metrics The following metrics were used to evaluate the performance of the models: Accuracy: The ratio of correctly predicted samples to the total number of samples. Precision: The ratio of true positive predictions to the total predicted positives. Recall: The ratio of true positive predictions to the total actual positives. F1-Score: The harmonic mean of precision and recall.

面部表情识别
项目简介 本项目旨在对VIRI数据库中包含的五种面部表情（愤怒、高兴、中性、悲伤与惊讶），以及NVIE数据库中涵盖的三种特定表情（恐惧、厌恶与高兴）开展分类任务。所有模型均基于PyTorch实现。
训练流程 首先，导入VIRI与NVIE数据库，并按照70%、15%、15%的划分比例将其分为训练集、验证集与测试集三部分。随后构建模型架构：针对可见光模态，对ResNet-18卷积神经网络（Convolutional Neural Network, CNN）架构进行改进，在其最终层中加入多个随机失活（Dropout）层、全连接（Fully Connected, FC）层以及针对可见光数据集定制的SoftMax层；针对红外模态，同样对ResNet-18卷积神经网络架构进行改进，在其最终层中加入多个随机失活（Dropout）层、全连接（Fully Connected, FC）层、激活函数、批归一化（Batch Normalization）层以及针对红外数据集定制的SoftMax层；针对可见光与红外模态，逐对融合特征后再次修改最终层，加入多个随机失活（Dropout）层、全连接层、批归一化层与SoftMax层。
评估方法 为全面评估所提出的面部表情识别模型的性能，我们采用多项标准评估指标，并在两个公开可用的数据集——VIRI与NVIE——上开展了大量对比实验。
数据集说明
VIRI数据库：包含566组图像对，涵盖五种不同的面部表情（愤怒、高兴、中性、悲伤、惊讶），分别通过可见光与红外模态采集得到。
NVIE数据库：包含来自100余名受试者的可见光与红外图像，涵盖六种基础面部表情（高兴、悲伤、惊讶、恐惧、暴怒、厌恶）。
实验设置
数据划分：将每个数据集按照70%训练集、15%验证集与15%测试集的比例进行拆分。
数据增强：对训练集应用旋转、缩放、畸变、剪切与翻转等数据增强技术，以提升训练样本的多样性并缓解过拟合问题。
训练流程：针对加入注意力机制改进的ResNet-18模型，分别在可见光图像、红外图像以及采用早期融合方式的融合数据集上进行训练。我们采用Adam优化器搭配余弦退火（CosineAnnealingLR）学习率调度器，将批量大小设置为64，训练所用的损失函数为交叉熵损失（Cross-Entropy Loss）。
评估指标 采用以下指标对模型性能进行评估：
准确率（Accuracy）：预测正确的样本数占总样本数的比例；
精确率（Precision）：真正例预测结果占所有预测为正例的样本数的比例；
召回率（Recall）：真正例预测结果占所有实际为正例的样本数的比例；
F1值（F1-Score）：精确率与召回率的调和平均值。