Attack Type

Jailbreak Research

Methods for bypassing model safety measures

704 matching entries

Jailbreak research results

704 entries

Single-Turn LLM Jailbreak

Published 9/1/2024

Analyzed 12/28/2024

A single-turn prompt injection attack that bypasses LLM content moderation filters by simulating a multi-turn conversation escalating towards harmful or inappropriate outputs within a single prompt. The attack leverages the LLM's tendency to maintain context and continue established patterns, even when leading to undesirable content.

Well, that escalated quickly: The Single-Turn Crescendo Attack (STCA)

Evaluated models: GPT-4, GPT-4o, GPT-4o Mini +4 more

Source: arXiv

Symbolic Math Jailbreak

Published 9/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a jailbreaking attack, termed "MathPrompt," which leverages the models' ability to process symbolic mathematics to bypass built-in safety mechanisms. The attack encodes harmful natural language prompts into mathematically formulated problems, causing the LLM to generate unsafe outputs while ostensibly solving a mathematical problem.

Jailbreaking Large Language Models with Symbolic Mathematics

Evaluated models: Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet +10 more

Source: arXiv

Adversarial Unlearning Bypass

Published 8/1/2024

Analyzed 1/26/2025

Large Language Models (LLMs) employing gradient-ascent based unlearning methods are vulnerable to a dynamic unlearning attack (DUA). DUA leverages optimized adversarial suffixes appended to prompts, reintroducing unlearned knowledge even without access to the unlearned model's parameters. This allows an attacker to recover sensitive information previously designated for removal.

Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models

Evaluated models: Llama 2 7B Chat, Llama 3 8B Instruct, Llama 3.1 8B Instruct

Source: arXiv

Attack LLMs with Toxic Answers

Published 8/1/2024

Analyzed 8/16/2025

Large Language Models (LLMs) are vulnerable to a targeted jailbreak attack, termed Atoxia, which can force the generation of specific harmful content. The attack operates by providing a target toxic answer to an attacker model, which then generates a corresponding adversarial query and a misleading "answer opening" (prefix). When the query and the answer prefix are presented to a vulnerable LLM, the model is induced to continue the generation, bypassing its safety alignment and completing the…

Atoxia: Red-teaming Large Language Models with Target Toxic Answers

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +5 more

Source: arXiv

Carrier Article Jailbreak

Published 8/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel jailbreak attack that leverages "neural carrier articles." This attack injects a prohibited query into a benign article generated by a secondary LLM, designed to be semantically similar to the prohibited query but not trigger the primary LLM's safety mechanisms. The secondary LLM generates articles based on hypernyms derived from the prohibited query, thus subtly shifting attention weights within the primary LLM, bypassing its safeguards.

Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Neural Carrier Articles

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 7B +1 more

Source: arXiv

Composable Jailbreak Synthesis

Published 8/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to composable jailbreak attacks, allowing bypass of safety filters through the chaining of multiple prompt transformations. The vulnerability arises from the ability to combine seemingly innocuous transformations to create effective attacks that achieve high attack success rates (ASR). These attacks can be synthesized automatically, allowing for the creation of novel and highly effective jailbreaks. Specifically, using the h4rm3l framework, attacks…

h4rm3l: A dynamic benchmark of composable jailbreak attacks for llm safety assessment

Evaluated models: Claude 3 Haiku, Claude 3 Sonnet, GPT-3.5 Turbo +3 more

Source: arXiv

Contextual Fusion Jailbreak

Published 8/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a multi-turn context-based jailbreak attack, termed Context Fusion Attack (CFA). CFA leverages the LLM's ability to understand context in multi-turn dialogues to bypass security mechanisms designed to prevent harmful outputs. The attack involves strategically crafting a series of prompts that build context, subtly introducing malicious keywords, and ultimately triggering the LLM to generate unsafe content. The malicious intent is masked within the…

Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles

Evaluated models: ChatGLM4, GPT-3.5 Turbo, GPT-4

Source: arXiv

Ensemble Jailbreak Technique

Published 8/1/2024

Analyzed 12/29/2024

The Ensemble Jailbreak (EnJa) attack exploits vulnerabilities in the safety mechanisms of large language models (LLMs) by combining prompt-level and token-level attacks. EnJa conceals malicious instructions within seemingly benign prompts, then uses a gradient-based method to optimize adversarial suffixes, significantly increasing the likelihood of bypassing safety filters and generating harmful content. The attack leverages a connector template to seamlessly integrate the concealed prompt and…

EnJa: Ensemble Jailbreak on Large Language Models

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 13B +3 more

Source: arXiv

GCG Suffix Data Exfiltration

Published 8/1/2024

Analyzed 3/24/2025

A Cross-Prompt Injection Attack (XPIA) can be amplified by appending a Greedy Coordinate Gradient (GCG) suffix to the malicious injection. This increases the likelihood that a Large Language Model (LLM) will execute the injected instruction, even in the presence of a user's primary instruction, leading to data exfiltration. The success rate of the attack depends on the LLM's complexity; medium-complexity models show increased vulnerability.

WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes

Evaluated models: GPT-3.5 Turbo, GPT-4o, Phi 3 Mini

Source: arXiv

Kov: MDP-Based LLM Jailbreak

Published 8/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to naturalistic adversarial attacks crafted using Markov Decision Processes (MDPs) and Monte Carlo Tree Search (MCTS). These attacks generate natural-language prompts that elicit harmful, violent, or discriminatory responses from the LLMs, even those with built-in safety mechanisms. The attacks are transferable across different LLMs, demonstrating a generalized vulnerability.

Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Search

Evaluated models: FastChat-T5 3B, GPT-3.5 Turbo, GPT-4 +1 more

Source: arXiv

Related research categories