AutoAdv Results

Large Language Models (LLMs) are susceptible to jailbreaking attacks, where carefully crafted malicious inputs bypass safety guardrails and provoke harmful responses. We introduce AutoAdv, a novel automated framework that generates adversarial prompts and assesses vulnerabilities in LLM safety mechanisms. Our approach employs an attacker LLM to create disguised malicious prompts using strategic rewriting techniques, tailored system prompts, and optimized hyperparameter settings. The core innovation is a dynamic, multiturn attack strategy that analyzes unsuccessful jailbreak attempts to iteratively develop more effective follow-up prompts. We evaluate the attack success rate (ASR) using the StrongREJECT framework across multiple interaction turns. Extensive empirical testing on state-of-the-art models, including ChatGPT, Llama, DeepSeek, Qwen, Gemma, and Mistral, reveals significant weaknesses, with AutoAdv achieving an ASR of 86% on the Llama-3.1-8B. These findings indicate that current safety mechanisms remain susceptible to sophisticated multiturn attacks.