Targeted Text-Diffusion Jailbreak

Description: Large language models (LLMs) are vulnerable to adversarial prompt engineering attacks that leverage proximity constraints to elicit harmful behaviors. By subtly modifying benign prompts within a semantically close embedding space, attackers can bypass existing safety mechanisms and induce undesired outputs, even when the original prompts would not trigger such a response. This vulnerability exploits the model's sensitivity to small perturbations in the input embedding, resulting in the generation of toxic or unsafe content.

Examples: See the paper's Appendix for specific examples of adversarial prompts generated using the DART method and their corresponding outputs from various LLMs. The examples demonstrate how small changes to prompts result in unsafe outputs from LLMs that would otherwise reply safely.

Impact: Successful exploitation of this vulnerability allows attackers to trigger the generation of undesirable content, such as hate speech, misinformation, or instructions for illegal activities. It can lead to reputational damage, erosion of trust, and potential legal consequences for organizations deploying LLMs. The impact may vary depending on the LLM and the nature of the generated unsafe content.

Affected Systems: Large language models (LLMs) using auto-regressive architectures and susceptible to embedding space manipulation. Specific LLMs tested in the research include GPT2-alpaca, Vicuna-7b, and Llama2-7b-chat-hf, but the vulnerability is likely present in other models.

Mitigation Steps:

• Improved embedding space robustness: Develop LLMs with increased robustness to small perturbations in the input embedding space.
• Enhanced safety mechanisms: Implement more sophisticated safety mechanisms that are resilient to adversarial prompts, including those that maintain semantic similarity to benign inputs.
• Adversarial training: Train LLMs using adversarial examples generated via methods such as DART to improve their resilience against this type of attack.
• Input sanitization: Implement robust input sanitization techniques that detect and mitigate subtly modified prompts.
• Regular safety audits: Conduct regular security audits using techniques such as DART to proactively identify and address potential vulnerabilities.