Investigating Distributional Robustness: Semantic Perturbations Using Generative Models (ImageNet Examples)

This dataset contains examples of semantically-perturbed images, for NeurIPS 2020 submission #4915.

There are four top-level folders, each containing results for semantic perturbations restricted to adjust the activation values at only certain layers of the BigGAN generative network: the first six layers, the middle six layers, the last six layers, and all layers.

Within each top-level folder, there are a further four folders, each corresponding to a classifier neural network whose evaluation is being evaluated. These are EfficientNet-B4 with NoisyStudent training [1], the standard ResNet50 [2], a pixel-perturbation-robust ResNet50 trained by Engstrom et al. [3] and another trained by Wong et al. [4], using their "Fast is better than free" technique.

Within each of these, there are many folders, named 'version_$N'. Each one of these contains three images: the unperturbed generated image, named unpert_generated_x_grid_0.png; the semantically-perturbed generated image, named generated_x_grid_0.png; and an image named semantic_pert_diffs_grid_0.png showing the pixel-space effect of the semantic perturbation, that is, the diff between the perturbed and unperturbed images. Note that if the perturbed and unperturbed images are identical, and the classifier misclassifies the unperturbed images, and so we skip this example.

Along with the 'version_$N' folders containing the images, there exists a file for each classifier named results.json. Each top-level item in this JSON file corresponds to one 'version_$N' example. There are 5 attributes: 'label', indicating the target label of the unperturbed image; 'magnitude', which gives the magnitude of the semantic perturbation found; 'skipped_cla', which is 1 if the example is skipped because the classifier did not correctly classify the unperturbed image; 'skipped_judge', which is 1 if the human judged that the unperturbed image did not match its label, so this example is skipped; and 'pert_judgement', which is 1 if the semantically-perturbed image is judged by the human to be of the same class as the unperturbed image. These judgements on these images were used to construct the main graphs in the paper.

[1] Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. Self-training with Noisy Student improves ImageNet classification. CoRR, abs/1911.04252, 2019. URL http://arxiv.org/abs/1911.04252.

[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90.

[3] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (Python library), 2019. URL ttps://github.com/MadryLab/robustness.

[4] Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https://openreview.net/forum?id=BJx040EFvH.