Knowledge Distillation-based Transformer for Human-Object Interaction Detection

The widely used cross-field star Transformer has achieved good results in detecting Human-Object Interaction (HOI). This study proposes a new Transformer network, Knowledge Distillation-based Transformer (KDT), for HOI detection. Owing to the roughness of the overall HOI features modeled by the Transformer network, a basic multi-branched structure exists for the three tasks of HOI detection: prediction of human boxes, prediction of object boxes and object categories, and prediction of interaction categories. The basic multi-branched structure comprises a human instance branch, an object instance branch, and an interaction branch. Human and object branch decoders are used to provide interaction branch decoders with the regional tips of the human object. To provide key semantic and spatial information for the Transformer structure, the semantic features of the object categories and interaction verbs, as well as the spatial features of humans and object boxes, are generated to provide semantic and spatial tips for different Transformer branches, which further improves the feature extraction capability of the decoders. Next, the study proposes another multi-branched Transformer structure as a teacher network. The teacher network decoders output accurate HOI using the generated features as decoder queries. During the training process, the basic multi-branched network is allowed to imitate the output of the teacher network. Finally, the study presents an additional category similarity loss to measure the intra- and inter-category similarities between the output predictions of the two networks, thereby improving the performance of the basic network decoder. Experimental results show that the mean Average Precision (mAP) for all categories, rare categories, and non-rare categories on the HOI benchmark dataset HICO-DET are 32.13%, 28.57%, and 33.19%, respectively, achieving the highest increase of 4.65% compared with the baseline.