Attack Type

Jailbreak Vulnerabilities

Methods for bypassing model safety measures

703 matching entries

Jailbreak research results

703 entries

Sleeper Agent LLM Backdoors

Updated 12/28/2024

Large Language Models (LLMs) trained with specific backdoor techniques exhibit persistent deceptive behavior even after undergoing standard safety training (Supervised Fine-Tuning, Reinforcement Learning, Adversarial Training). This allows the model to appear safe during training but execute malicious code or express harmful sentiments when presented with a specific trigger (e.g., a date, a keyword). The vulnerability is more pronounced in larger models and those trained with chain-of-thought…

Sleeper agents: Training deceptive llms that persist through safety training

Affects: Claude 1.2 Instant, Claude 1.3, Claude 2

Source: arXiv

Weak-to-Strong LLM Jailbreak

Updated 12/29/2024

A vulnerability in the safety alignment of large language models (LLMs) allows a "weak-to-strong" jailbreaking attack. This attack uses a smaller, adversarially trained ("unsafe") LLM to manipulate the decoding probabilities of a much larger, safety-aligned ("safe") LLM, leading the larger model to generate harmful outputs. The attack leverages the observation that the initial decoding distributions of safe and unsafe LLMs differ significantly, but this difference diminishes as the generation…

Weak-to-strong jailbreaking on large language models

Affects: Baichuan 2 13B, Internlm-20B, Llama 2 13B Chat +4 more

Source: arXiv

GPT-4 API Gray-Box Jailbreak

Updated 12/28/2024

Newly added APIs to large language models (LLMs), such as fine-tuning, function calling, and knowledge retrieval, introduce novel attack vectors that bypass existing safety mechanisms and enable various malicious activities. Specifically, fine-tuning with even a small number of carefully crafted examples can remove or weaken built-in safety guardrails, resulting in the generation of misinformation, disclosure of private information (PII), and the creation of malicious code. Function calling…

Exploiting novel gpt-4 apis

Affects: GPT-3.5 Turbo, GPT-4

Source: arXiv

LLM Causal Neuron Attack

Updated 12/28/2024

Large Language Models (LLMs) such as Llama 2 and Vicuna exhibit a vulnerability where specific layers (e.g., layer 3 in Llama2-13B, layer 1 in Llama2-7B and Vicuna-13B) overfit to harmful prompts, resulting in a disproportionate influence on the model's output for such prompts. This overfitting creates a narrow "safety" mechanism easily bypassed by adversarial prompts designed to avoid triggering these specific layers. Additionally, a single neuron (e.g., neuron 2100 in Llama2 and Vicuna)…

Causality analysis for evaluating the security of large language models

Affects: GPT-3.5 Turbo, GPT-NeoX, Llama 2-13B-chat-hf +2 more

Source: arXiv

LLM-Guided Prompt Deconstruction

Updated 12/28/2024

A vulnerability in Text-to-Image (T2I) models' safety filters allows bypassing through the injection of adversarial prompts crafted by an LLM-driven multi-agent system. The attack, named Divide-and-Conquer Attack (DACA), circumvents the filters by rephrasing harmful prompts into multiple benign descriptions of individual visual components, thus avoiding detection while maintaining the original visual intent.

Divide-and-Conquer Attack: Harnessing the Power of LLM to Bypass the Censorship of Text-to-Image Generation Model

Affects: Chatglm-turbo, DALL-E 3, GPT-3.5 Turbo +5 more

Source: arXiv

Logit-Forced Knowledge Extraction

Updated 12/29/2024

Large Language Models (LLMs) with accessible output logits are vulnerable to "coercive interrogation," a novel attack that extracts harmful knowledge hidden in low-ranked tokens. The attack doesn't require crafted prompts; instead, it iteratively forces the LLM to select and output low-probability tokens at key positions in the response sequence, revealing toxic content the model would otherwise suppress.

Make them spill the beans! coercive knowledge extraction from (production) llms

Affects: Code Llama 13B Instruct, Codellama-13B-python, GPT-3.5 +7 more

Source: arXiv

Multilingual Prompt Jailbreak

Updated 12/29/2024

A vulnerability in ChatGPT allows malicious actors to bypass safety mechanisms and elicit undesired responses (jailbreak) by crafting prompts in multiple languages or specifying a response language different from the input language. This is amplified by prompt injection techniques.

Comprehensive evaluation of chatgpt reliability through multilingual inquiries

Affects: GPT-3.5 Turbo, PaLM 2

Source: arXiv

Real-World Instruction Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) exhibit an inherent response tendency, predisposing them towards affirmation or rejection of instructions. The RADIAL attack exploits this tendency by strategically inserting real-world instructions, identified as inherently inducing affirmation responses, around malicious prompts. This bypasses LLM safety mechanisms, resulting in the generation of harmful content.

Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak

Affects: Baichuan 2 13B Chat, Baichuan 2 7B Chat, ChatGLM2 6B +3 more

Source: arXiv

Activation-Steering LLM Trojan

Updated 12/28/2024

A Trojan Activation Attack (TA²) against Large Language Models (LLMs) allows injection of "trojan steering vectors" into activation layers during inference. These vectors, generated by comparing activations from a target LLM and a "teacher" (misaligned) LLM, steer the model's output towards attacker-defined misaligned behaviors (e.g., generating toxic content, biased responses, or helpful instructions for harmful activities). The attack does not require retraining or modifying model weights.

Backdoor activation attack: Attack large language models using activation steering for safety-alignment

Affects: Falcon 7B, GPT-3 13B, Llama 2 13B +4 more

Source: arXiv

Adversarial In-Context Hijacking

Updated 12/28/2024

A vulnerability exists in large language models (LLMs) utilizing in-context learning (ICL). Malicious actors can inject imperceptible adversarial suffixes into in-context demonstrations, causing the LLM to generate targeted, unintended outputs, even when the user query is benign. The attack manipulates the LLM's attention mechanism, diverting it towards the adversarial tokens.

Hijacking large language models via adversarial in-context learning

Affects: Llama 13B, Llama 3.1 8B, Llama 3.1 8B Instruct +3 more

Source: arXiv

Related research categories