AV-Deepfake1M

The detection and localization of highly realistic deepfake audio-visual content are challenging even for the most advanced state-of-the-art methods. While most of the research efforts in this domain are focused on detecting high-quality deepfake images and videos, only a few works address the problem of the localization of small segments of audio-visual manipulations embedded in real videos. In this research, we emulate the process of such content generation and propose the AV-Deepfake1M dataset. The dataset contains content-driven (i) video manipulations, (ii) audio manipulations, and (iii) audio-visual manipulations for more than 2K subjects resulting in a total of more than 1M videos. The paper provides a thorough description of the proposed data generation pipeline accompanied by a rigorous analysis of the quality of the generated data. The comprehensive benchmark of the proposed dataset utilizing state-of-the-art deepfake detection and localization methods indicates a significant drop in performance compared to previous datasets. The proposed dataset will play a vital role in building the next-generation deepfake localization methods. The dataset and associated code are available at https://github.com/ControlNet/AV-Deepfake1M.

即便对于当前最优的深度伪造(deepfake)检测与定位方法而言，检测并定位高度逼真的深度伪造音视频内容仍极具挑战。当前该领域的多数研究工作均聚焦于高质量深度伪造图像与视频的检测，仅有少数研究关注嵌入于真实视频中的小型音视频篡改片段的定位问题。本研究模拟了此类深度伪造内容的生成流程，并提出了AV-Deepfake1M数据集。该数据集涵盖面向超2000个主体的三类内容驱动型篡改：(i) 视频篡改、(ii) 音频篡改、(iii) 音视频联合篡改，总计生成超过100万条视频。本文对所提出的数据生成流水线进行了详尽描述，并针对生成数据的质量展开了严谨的分析。利用当前最优的深度伪造检测与定位方法对本数据集开展的全面基准测试结果显示，相较于此前的各类数据集，现有方法的性能出现了显著下滑。本数据集将为下一代深度伪造定位方法的研发起到至关重要的作用。本数据集及配套代码可通过https://github.com/ControlNet/AV-Deepfake1M获取。