Attack Type

Jailbreak Vulnerabilities

Methods for bypassing model safety measures

703 matching entries

Jailbreak research results

703 entries

Universal Jailbreak Backdoors

Updated 12/29/2024

A universal jailbreak backdoor vulnerability exists in Large Language Models (LLMs) trained using Reinforcement Learning from Human Feedback (RLHF). A malicious actor can poison the RLHF training data by introducing a trigger word into harmful prompts and labeling the harmful model outputs as preferred. This causes the LLM to generate harmful responses whenever the trigger word is included in any prompt, regardless of the prompt's content or topic. The backdoor is "universal" because it…

Universal jailbreak backdoors from poisoned human feedback

Source: arXiv

AutoDAN: Interpretable LLM Jailbreak

Updated 12/29/2024

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Affects: GPT-3.5 Turbo, GPT-4, Guanaco 7B +4 more

Source: arXiv

Automated LLM Jailbreak

Updated 12/28/2024

Large Language Models (LLMs) are vulnerable to prompt-based jailbreaks, allowing adversaries to bypass safety guardrails and elicit undesirable outputs. The Prompt Automatic Iterative Refinement (PAIR) algorithm efficiently generates these jailbreaks using a limited number of black-box queries to the target LLM. The vulnerability stems from the LLM's inability to robustly handle adversarial prompts crafted through iterative refinement, even without white-box access to its internal mechanisms.

Jailbreaking black box large language models in twenty queries

Affects: Claude Instant 1.2, Claude 2.1, Gemini Pro +6 more

Source: arXiv

Automating Stealthy LLM Jailbreaks

Updated 12/28/2024

Large Language Models (LLMs) employing alignment techniques remain vulnerable to "jailbreak" attacks. The AutoDAN technique automatically generates semantically meaningful prompts that bypass safety features and elicit malicious outputs from aligned LLMs, unlike previous methods producing nonsensical prompts easily detectable by perplexity checks. These prompts exploit weaknesses in the LLM's alignment, causing it to generate responses that violate intended safety constraints.

Autodan: Generating stealthy jailbreak prompts on aligned large language models

Affects: GPT-3.5 Turbo, GPT-4

Source: arXiv

Decoding-Based LLM Jailbreak

Updated 12/28/2024

Open-source Large Language Models (LLMs) are vulnerable to a generation exploitation attack that leverages variations in decoding hyperparameters and sampling methods to bypass safety mechanisms. Manipulating these parameters, even subtly, can drastically increase the likelihood of the model generating harmful or unsafe outputs, even in models previously deemed "aligned." The attack is effective even when removing only the system prompt.

Catastrophic jailbreak of open-source llms via exploiting generation

Affects: GPT-3.5 Turbo

Source: arXiv

Fine-tuning Compromises LLM Safety

Updated 12/28/2024

Fine-tuning aligned Large Language Models (LLMs) on a small number of adversarially crafted examples, or even on benign datasets, can compromise their safety alignment, leading to the generation of harmful or inappropriate content. This vulnerability exploits the few-shot learning capabilities of LLMs, allowing attackers to override existing safety mechanisms with minimal effort and cost. Even unintentional fine-tuning with seemingly benign datasets can result in unintended safety degradation.

Fine-tuning aligned language models compromises safety, even when users do not intend to!

Affects: GPT-3.5 Turbo, Llama 2 13B Chat, Llama 2 70B Chat +2 more

Source: arXiv

Hidden Prompt Injection Attacks

Updated 12/28/2024

Large Language Models (LLMs) are vulnerable to Compositional Instruction Attacks (CIA), where malicious prompts are embedded within seemingly harmless instructions. This allows attackers to bypass safety mechanisms and elicit harmful responses from the model, even if the individual components of the prompt would be flagged as safe. The attack exploits the model's inability to correctly identify underlying malicious intent within composite instructions.

Prompt packer: Deceiving llms through compositional instruction with hidden attacks

Affects: ChatGLM2 6B, GPT-3.5 Turbo, GPT-4

Source: arXiv

In-Context LLM Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to In-Context Attacks (ICA) and susceptible to mitigation via In-Context Defense (ICD). ICA leverages a small number of harmful demonstration examples within a prompt to elicit harmful responses from the LLM, even if it is otherwise safety-aligned. ICD counteracts ICA by prepending safe demonstration examples to the prompt, effectively reducing the likelihood of harmful output. The effectiveness of both ICA and ICD is demonstrated across multiple LLMs.

Jailbreak and guard aligned language models with only few in-context demonstrations

Affects: GPT-4 0613, Llama 2 7B Chat, Mistral-7B-v2 +4 more

Source: arXiv

LLM Red Teaming Framework

Updated 12/28/2024

A vulnerability in large language models (LLMs) allows attackers to craft malicious prompts that induce the LLM to generate harmful content, such as fraudulent material, racist remarks, or instructions for illegal activities. The vulnerability arises from the LLM's inability to reliably distinguish between benign and malicious instructions disguised within seemingly innocuous prompts. Attackers can exploit this by leveraging techniques like obfuscation, code injection/payload splitting, and…

Attack prompt generation for red teaming and defending large language models

Source: arXiv

LoRA Bypass Safety Training

Updated 12/28/2024

Low-rank adaptation (LoRA) fine-tuning allows efficient circumvention of safety training in large language models (LLMs), such as Llama 2-Chat 70B, resulting in significantly reduced refusal rates for harmful prompts while maintaining general performance capabilities. Attackers can use LoRA with a small, synthetic dataset of harmful instructions and responses to effectively undo safety measures implemented during the model's training.

Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b

Affects: Llama 2 13B Chat, Llama 2 70B Chat, Llama 2 7B Chat

Source: arXiv

Related research categories