Attack Type

Jailbreak Vulnerabilities

Methods for bypassing model safety measures

703 matching entries

Jailbreak research results

703 entries

Few-Shot LLM Jailbreak

Updated 12/29/2024

A vulnerability in aligned Large Language Models (LLMs) allows circumvention of safety mechanisms through improved few-shot jailbreaking techniques. The attack leverages injection of special system tokens (e.g., [/INST]) into few-shot demonstrations and demo-level random search to optimize the probability of generating harmful responses. This bypasses defenses that rely on perplexity filtering and input perturbation.

Improved few-shot jailbreaking can circumvent aligned language models and their defenses

Affects: GPT-4, Llama 2 7B, Llama 3 8B +4 more

Source: arXiv

GPT-4o Multimodal Jailbreak

Updated 3/4/2025

GPT-4o exhibits vulnerability to jailbreak attacks via audio prompts, despite enhanced safety against text-based attacks. Successful jailbreaks can be achieved by converting text prompts, including those optimized for adversarial attacks against other LLMs (demonstrated using GCG, AutoDAN, PAP, and BAP methods), into audio using text-to-speech (TTS) synthesis. This circumvention allows elicitation of unsafe responses from GPT-4o that would otherwise be prevented by its safety mechanisms. The…

Unveiling the safety of gpt-4o: An empirical study using jailbreak attacks

Affects: GPT-4o, GPT-4V, Llama 2 7B Chat

Source: arXiv

Hidden Structure Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to jailbreak attacks exploiting uncommon text-encoded structures (UTES) rarely encountered during training. These UTES, such as JSON, tree representations, or LaTeX code, embedded within prompts, cause LLMs to bypass safety mechanisms and generate harmful content. The attack's success stems from the LLM's difficulty in processing and interpreting these unusual structures, coupled with the obfuscation of malicious instructions within the structured…

StructuralSleight: Automated Jailbreak Attacks on Large Language Models Utilizing Uncommon Text-Encoded Structure

Affects: Claude 2, Claude 3 Opus, GPT-3.5 Turbo +3 more

Source: arXiv

Knowledge-Based LLM Jailbreak

Updated 3/4/2025

Large Language Models (LLMs) are vulnerable to knowledge-based jailbreaks, where an attacker provides domain-specific knowledge to elicit harmful or unintended outputs. The vulnerability stems from the LLM's ability to process and respond to knowledge inputs in a way that circumvents safety mechanisms, even if the input knowledge itself isn't inherently malicious. Attackers leverage this by constructing prompts that combine seemingly innocuous knowledge with subtly manipulative phrasing to…

Knowledge-to-jailbreak: One knowledge point worth one attack

Affects: FinanceChat 7B, GPT-3.5 Turbo, GPT-4 Turbo +6 more

Source: arXiv

LLM Copyright Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to prompt injection attacks that can bypass their internal copyright compliance mechanisms, causing them to generate verbatim copyrighted text. The vulnerability stems from insufficient robustness against prompt engineering techniques that manipulate the model into ignoring or circumventing its safety filters designed for copyright protection.

SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation

Affects: Claude 3 Haiku, Gemini 1.5 Pro, Gemini Pro +5 more

Source: arXiv

LLM Robot Bias & Violence

Updated 4/12/2025

Large Language Models (LLMs) used to control robots exhibit biases leading to discriminatory and unsafe behaviors. When provided with personal characteristics (e.g., race, gender, disability), LLMs generate biased outputs resulting in discriminatory actions (e.g., assigning lower rescue priority to certain groups) and accept or deem feasible dangerous or unlawful instructions (e.g., removing a person's mobility aid).

Llm-driven robots risk enacting discrimination, violence, and unlawful actions

Affects: GPT-3.5, GPT-3.5 Turbo, GPT-4 +1 more

Source: arXiv

LangChain Poisoning Jailbreak

Updated 12/29/2024

A vulnerability in Retrieval-Augmented Generation (RAG) systems utilizing LangChain allows for indirect jailbreaks of Large Language Models (LLMs). By poisoning the external knowledge base accessed by the LLM through LangChain, attackers can manipulate the LLM's responses, causing it to generate malicious or inappropriate content. The attack exploits the LLM's reliance on the external knowledge base and bypasses direct prompt-based jailbreak defenses.

Poisoned langchain: Jailbreak llms by langchain

Affects: ChatGLM2 6B, ChatGLM3 6B, ERNIE 3.5 +3 more

Source: arXiv

Model Combination Misuse

Updated 1/26/2025

LLMs, even when individually assessed as "safe," can be combined by an adversary to achieve malicious outcomes. This vulnerability exploits the complementary strengths of multiple models—a high-capability model that refuses malicious requests and a low-capability model that does not—through task decomposition. Adversaries can either manually decompose tasks into benign (solved by the high-capability model) and easily-malicious subtasks (solved by the low-capability model) or automate the…

Adversaries can misuse combinations of safe models

Affects: Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet +8 more

Source: arXiv

Obscured Prompt Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to jailbreaking attacks using "obscure" input prompts. The ObscurePrompt attack iteratively transforms a base prompt containing known jailbreaking techniques into an obscured version using another LLM (e.g., GPT-4). This obfuscation weakens the LLM's safety mechanisms, causing it to bypass safety restrictions and generate harmful content.

Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings

Affects: GPT-3.5 Turbo, GPT-4, GPT-4o Mini +5 more

Source: arXiv

RL-Powered LLM Jailbreak

Updated 12/29/2024

RL-JACK is a reinforcement learning-based black-box attack that generates jailbreaking prompts to bypass safety mechanisms in LLMs. The attack leverages a deep reinforcement learning agent to iteratively refine prompts, maximizing the likelihood of eliciting harmful responses to unethical questions. The effectiveness stems from a novel reward function that provides continuous feedback based on cosine similarity to a reference answer from an unaligned LLM, and an action space that strategically…

RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs

Affects: Falcon-40B-instruct, GPT-3.5 Turbo, Llama 2 70B Chat +3 more

Source: arXiv

Related research categories