MAD: meta adversarial defense benchmark

Adversarial training (AT) is a fundamental technique employed to defend against adversarial attacks and effectively enhance model robustness. In particular, rapid adaptation to unknown attacks with high accuracy is critical for sophisticated and responsive systems, such as autonomous driving systems. Therefore, to address these issues, we propose a novel meta adversarial defense (MAD) benchmark. This benchmark consists of three MAD datasets generated from 30 types of attacks on MNIST, CIFAR-10, and Tiny-ImageNet datasets, along with an evaluation toolkit. In addition, we introduce a meta-learning-based AT (Meta-AT) algorithm as the baseline, with high robustness to unknown adversarial attacks through few-shot learning. Experimental results demonstrate the effectiveness of our Meta-AT compared to the state-of-the-art (SOTA) approaches, such as traditional AT, Fast-AT, Free-AT, adversarial training with transferable adversarial examples (ATTA), and you only propagate once (YOPO). Moreover, the models trained with Meta-AT maintain excellent standard classification accuracy on clean examples ($SA$) and robust classification accuracy on adversarial examples ($RA$). This benchmark demonstrates significant improvements in investigating the transferability of adversarial defense methods to unknown attacks and the capacity to learn from a limited number of adversarial examples. Our code and the attacked datasets will be available at https://github.com/PXX1110/MAD