CaptchaBench: A Large-Scale Benchmark Dataset for Evaluating Adversarial Perturbation Methods against VLM-based CAPTCHA Recognition

This release is a REPRESENTATIVE SUBSET of the full CaptchaBench benchmark, containing 4,000 source CAPTCHA images and 12,000 adversarial images (6 attack methods x 2 generative pipelines). This subset includes all adversarial perturbation outputs and associated metadata. The full dataset (840K base images + 5M+ adversarial variants) and complete VLM evaluation results will be released upon paper acceptance. ========================================================== CaptchaBench is a large-scale benchmark dataset for evaluating adversarial perturbation methods against Vision-Language Model (VLM)-based CAPTCHA recognition. The full dataset comprises 840,000 base images and 5,040,000 adversarially perturbed variants generated from the complete GB2312 Level-1 Chinese character set (3,500 characters, 120 backgrounds per character) using two complementary generative pipelines — Illusion Diffusion ControlNet (ID) and Stable Diffusion XL (SDXL) — annotated with stroke counts, structural types, and VLM evaluation results. == This Release == This Zenodo release contains a representative evaluation subset of the full benchmark: - 4,000 source CAPTCHA images (2,000 per pipeline) with JSON metadata - 12,000 adversarial images (6 attack methods x 1,000 images x 2 pipelines) with per-image JSON metadata - 16,000 images total, ~9.7 GB compressed == Attack Methods == Six adversarial perturbation methods organized into three input modality groups, all using author-recommended optimal hyperparameters: 1. Anti-DreamBooth (ASPL) — ICCV 2023, Image-only, eps=0.05 ([-1,1]), 200 steps 2. Glaze / MI (CWA) — CVPR 2018 + ICLR 2024, Image-only, eps=16/255, 300 steps 3. AMP (AttackVLM) — NeurIPS 2023, Image-only, eps=8/255, 300 steps 4. XTransfer — ICML 2025, Image-only, eps=12/255, 300 steps 5. Nightshade — IEEE S&P 2024, Text-only, eps=0.05 ([0,1]), 500 steps 6. MMCoA — arXiv 2024, Image+Text, eps=1/255, 100 steps == Image Specifications == - Source image resolution: 1024 x 1024 - Adversarial image resolution: 512 x 512 - Format: PNG (lossless) - Character set: GB2312 Level-1 (3,500 Chinese characters) == Directory Structure == CaptchaBench/ README.md LICENSE (CC BY 4.0) ATTACK_PARAMS.md source_images/ ID_ControlNet/ (2,000 images + JSON) SD_ControlNet/ (2,000 images + JSON) adversarial_images/ ID_ControlNet/ (6 methods x 1,000 images) SD_ControlNet/ (6 methods x 1,000 images) logs/ ID_ControlNet/ SD_ControlNet/ == JSON Metadata == Each image has a companion .json file containing: prompt, character label, bounding box, generation config, and batch metadata. == Computational Resources == Generated on NVIDIA A100 80GB (SDXL) and RTX 3090 24GB (ID ControlNet). Total generation time: ~40+ GPU-hours. See included logs/ for detailed resource usage.