A Low Bitrate Animation Codec using Pose-guided Human Video Generation and On-the-fly Training

This paper proposes a novel low-bitrate animation codec leveraging pose-guided human video generation with on-the-fly training. On the encoder side, the whole sequence is divided into key and non-key frames. Instead of compressing the whole sequences, only the keyframes and pose information are compressed. On the decoder side, the non-key frames are generated using a novel pose-guided human video generation model. The model is trained on-the-fly using keyframes to learn the mapping from pose to full frames. The whole codec significantly reduces the bitrate for transmitting videos while retaining considerable video quality. Furthermore, we introduce a nonlinear interpolation based frame enhancement method as post-processing to further enhance the quality of the reconstructed video. An extensive dataset is provided for training and evaluation purposes to ensure robust and reliable compression.  The experimental results demonstrate the substantial performance gains achieved by our method in terms of preserved video quality at low bitrates. In particular, within the HM-16.7 framework, our codec exhibits a Bjøntegaard delta bit rate (BD-rate) gain as high as 35.978\%. This significant gain underscores the efficiency of our approach, especially in bandwidth-constrained environments.