Distribution of data in the Affectnet dataset.

Facial expression recognition faces great challenges due to factors such as face similarity, image quality, and age variation. Although various existing end-to-end Convolutional Neural Network (CNN) architectures have achieved good classification results in facial expression recognition tasks, these network architectures share a common drawback that the convolutional kernel can only compute the correlation between elements of a localized region when extracting expression features from an image. This leads to difficulties for the network to explore the relationship between all the elements that make up a complete expression. In response to this issue, this article proposes a facial expression recognition network called HFE-Net. In order to capture the subtle changes of expression features and the whole facial expression information at the same time, HFE-Net proposed a Hybrid Feature Extraction Block. Specifically, Hybrid Feature Extraction Block consists of parallel Feature Fusion Device and Multi-head Self-attention. Among them, Feature Fusion Device not only extracts the local information in expression features, but also measures the correlation between distant elements in expression features, which helps the network to focus more on the target region while realizing the information interaction between distant features. And Multi-head Self-attention can calculate the correlation between the overall elements in the feature map, which helps the network to extract the overall information of the expression features. We conducted a lot of experiments on four publicly available facial expression datasets and verified that the Hybrid Feature Extraction Block constructed in this paper can improve the network’s recognition ability for facial expressions.面部表情识别面临诸多严峻挑战，诸如面部相似度、图像质量以及年龄变化等因素均会对其产生干扰。尽管当前各类端到端卷积神经网络（Convolutional Neural Network，CNN）架构在面部表情识别任务中已取得优异的分类效果，但这类网络架构存在一个共同缺陷：在从图像中提取表情特征时，卷积核仅能计算局部区域内元素间的相关性，这使得网络难以探究构成完整表情的所有元素之间的关联关系。 针对这一问题，本文提出了一种名为HFE-Net的面部表情识别网络。为了同时捕捉表情特征的细微变化与完整的面部表情信息，HFE-Net设计了混合特征提取块（Hybrid Feature Extraction Block）。具体而言，混合特征提取块由并行的特征融合装置与多头自注意力（Multi-head Self-attention）模块组成。其中，特征融合装置既可提取表情特征中的局部信息，又可度量表情特征中远距离元素间的相关性，这有助于网络在实现远距离特征间信息交互的同时，更精准地聚焦于目标区域；而多头自注意力模块则能够计算特征图内全局元素间的相关性，助力网络提取表情特征的整体信息。 我们在四个公开可用的面部表情数据集上开展了大量实验，验证了本文所构建的混合特征提取块能够有效提升网络的面部表情识别能力。