SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples

SCOOTER Evaluation and Dataset Creation Process Data This data collection contains the following information associated with our six initial SCOOTER studies: The real image baseline (ImageNet S-R50-N), containing 2966 ImageNet instances that we used to generate adversarial examples (AEs) Notes on how we processed the validation set images to generate ImageNet S-R50-N (see notes.txt and removed_or_modified.txt) All AEs generated within our six SCOOTER studies (see the directories semanticadv/, cadv/, ncf/, diffattack/, advpp/, and aca/) + the collected ratings per image (see the Salman2020Do_R50_annotations.csv file per directory) The ratings generated by GPT-4o (see 4o_ratings/) The predictions of our victim model for all ImageNet Validation set images that displayed exactly one ImageNet object (>39,000 images, see predictions_salman_clean.csv) The first four files are all contained in images_v1.0.0.zip, while the last file is stored separately. More information on how to process the information at hand is available at our GitHub repository: https://github.com/DrenFazlija/SCOOTER SCOOTER Data 1.0.1 The repository now includes an additional file called scooter_db_entries.zip, which contains SQL table data that can be directly used to set up the web app: modified.csv: All 6,924 generated modified images with their respective metadata (e.g., associated attack name and victim model) real.csv: All 2,966 ImageNet S-R50-N images with their respective metadata. atc.csv: All instruction manipulation check and bogus item images. ishihara.csv: All Ishihara-like images. cc.csv: All final comprehension check images. For more information about the web app, check out the dedicated section of our repository: https://github.com/DrenFazlija/SCOOTER/tree/main/ui Citation Please cite the original paper when using (parts of) our data: @misc{fazlija2025scooterhumanevaluationframework, title={SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples}, author={Dren Fazlija and Monty-Maximilian Zühlke and Johanna Schrader and Arkadij Orlov and Clara Stein and Iyiola E. Olatunji and Daniel Kudenko}, year={2025}, eprint={2507.07776}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2507.07776}, }