Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Concealed Multi-Turn Jailbreak

Published Sep 26, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel multi-turn jailbreaking attack, termed "RED QUEEN ATTACK." This attack uses multi-turn conversations to conceal malicious intent by framing the user as a protector seeking to prevent harmful actions by others. The LLM, instead of detecting the concealed malicious intent, provides information that facilitates the harmful action under the guise of assisting in prevention efforts.

RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking

Evaluated models: GPT-4o

Automated LLM Fuzz Jailbreak

Published Sep 23, 2024Analyzed Dec 28, 2024Source: arXiv

A novel black-box attack framework leverages fuzz testing to automatically generate concise and semantically coherent prompts that bypass safety mechanisms in large language models (LLMs), eliciting harmful or offensive responses. The attack starts with an empty seed pool, utilizes LLM-assisted mutation strategies (Role-play, Contextualization, Expand), and employs a two-level judge module for efficient identification of successful jailbreaks. The attack's effectiveness is demonstrated across…

Effective and Evasive Fuzz Testing-Driven Jailbreaking Attacks against LLMs

Evaluated models: Baichuan 2 7B Chat, Gemini Pro, GPT-3.5 Turbo+4 more

Baichuan 2 7B Chat
Gemini Pro
GPT-3.5 Turbo
GPT-4
Guanaco 7B
Llama 2 7B Chat
Vicuna 7B v1.3

Reinforcement Learning Jailbreak

Published Sep 21, 2024Analyzed Dec 28, 2024Source: arXiv

PathSeeker demonstrates a novel black-box jailbreak attack against Large Language Models (LLMs) that utilizes multi-agent reinforcement learning. The attack iteratively modifies input prompts based on model responses, leveraging a reward mechanism focused on vocabulary expansion in the LLM's output to circumvent safety mechanisms and elicit harmful responses. This technique bypasses existing safety filters by encouraging the model to relax its constraints, rather than directly targeting…

PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach

Evaluated models: Claude 3.5 Sonnet, DeepSeek Chat, Deepseek-coder+14 more

Claude 3.5 Sonnet
DeepSeek Chat
Deepseek-coder
Gemini 1.5 Flash
Gemma2-8B-instruct
Glm-4-air
GPT-3.5 Turbo
GPT-4o Mini
Llama 2 13B Chat
Llama 2 7B Chat
Llama 3 70B
Llama 3.1 405B
Llama 3.1 70B
Llama 3.1 8B
Mistral Nemo
Qwen 2 7B Instruct
Vicuna 7B

Symbolic Math Jailbreak

Published Sep 17, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a jailbreaking attack, termed "MathPrompt," which leverages the models' ability to process symbolic mathematics to bypass built-in safety mechanisms. The attack encodes harmful natural language prompts into mathematically formulated problems, causing the LLM to generate unsafe outputs while ostensibly solving a mathematical problem.

Jailbreaking Large Language Models with Symbolic Mathematics

Evaluated models: Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet+10 more

Claude 3 Haiku
Claude 3 Opus
Claude 3 Sonnet
Claude 3.5 Sonnet
Gemini 1.5 Flash
Gemini 1.5 Flash (block None)
Gemini 1.5 Pro
Gemini 1.5 Pro (block None)
GPT-4
GPT-4 Turbo
GPT-4o
GPT-4o Mini
Llama 3.1 70B

Adaptive Position Jailbreak

Published Sep 11, 2024Analyzed Dec 29, 2024Source: arXiv

AdaPPA is a jailbreak attack that exploits the varying levels of alignment protection in LLMs at different output positions. It leverages the model's instruction-following capabilities by pre-filling the output with carefully crafted "safe" content, creating a perceived completion and lowering the model's guard before generating malicious content. The attack's effectiveness relies on the adaptive generation of both safe and harmful pre-fill content, strategically placed to exploit weaknesses…

AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs

Evaluated models: ChatGLM3 6B, GPT-4o, GPT-4o Mini+5 more

ChatGLM3 6B
GPT-4o
GPT-4o Mini
Llama 2 13B
Llama 2 7B
Llama 3 8B
Vicuna 13B
Vicuna 7B

Single-Turn LLM Jailbreak

Published Sep 4, 2024Analyzed Dec 28, 2024Source: arXiv

A single-turn prompt injection attack that bypasses LLM content moderation filters by simulating a multi-turn conversation escalating towards harmful or inappropriate outputs within a single prompt. The attack leverages the LLM's tendency to maintain context and continue established patterns, even when leading to undesirable content.

Well, that escalated quickly: The Single-Turn Crescendo Attack (STCA)

Evaluated models: GPT-4, GPT-4o, GPT-4o Mini+4 more

GPT-4
GPT-4o
GPT-4o Mini
Llama 3 8B
Llama 3 70B
Llama 3.1 8B
Llama 3.1 70B

Attack LLMs with Toxic Answers

Published Aug 27, 2024Analyzed Aug 16, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a targeted jailbreak attack, termed Atoxia, which can force the generation of specific harmful content. The attack operates by providing a target toxic answer to an attacker model, which then generates a corresponding adversarial query and a misleading "answer opening" (prefix). When the query and the answer prefix are presented to a vulnerable LLM, the model is induced to continue the generation, bypassing its safety alignment and completing the…

Atoxia: Red-teaming Large Language Models with Target Toxic Answers

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o+5 more

GPT-3.5 Turbo
GPT-4
GPT-4o
GPT-4o Mini
Llama 2 7B Chat
Llama 3 8B Instruct
Mistral 7B
Vicuna 7B v1.5

Random Token T2I Jailbreak

Published Aug 25, 2024Analyzed Dec 29, 2024Source: arXiv

A heuristic token search attack, termed HTS-Attack, can bypass safety mechanisms in text-to-image (T2I) models, allowing generation of NSFW content. The attack iteratively replaces tokens in a malicious prompt with semantically similar tokens from the model's vocabulary, avoiding detection by prompt and image checkers. The method leverages a surrogate CLIP model to maintain semantic similarity to the target NSFW prompt.

Rt-attack: Jailbreaking text-to-image models via random token

Evaluated models: Clip-vit-base-patch32, DALL-E 3, GPT 3.5-turbo-instruct+4 more

Clip-vit-base-patch32
DALL-E 3
GPT 3.5-turbo-instruct
Safegen
Sld
Stable Diffusion v1.4
Stable Diffusion v1.5

LLM Adversarial Suffix Optimization

Published Aug 21, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel black-box jailbreaking attack, ECLIPSE, which leverages the LLM's own capabilities as an optimizer to generate adversarial suffixes. ECLIPSE iteratively refines these suffixes based on a harmfulness score, bypassing the need for pre-defined affirmative phrases used in previous optimization-based attacks. This allows for effective jailbreaking even with limited interaction and without white-box access to the LLM's internal parameters.

Unlocking Adversarial Suffix Optimization Without Affirmative Phrases: Efficient Black-box Jailbreaking via LLM as Optimizer

Evaluated models: Falcon 7B Instruct, GPT-3.5 Turbo, Llama 2 7B Chat+1 more

Falcon 7B Instruct
GPT-3.5 Turbo
Llama 2 7B Chat
Vicuna 7B

Carrier Article Jailbreak

Published Aug 20, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel jailbreak attack that leverages "neural carrier articles." This attack injects a prohibited query into a benign article generated by a secondary LLM, designed to be semantically similar to the prohibited query but not trigger the primary LLM's safety mechanisms. The secondary LLM generates articles based on hypernyms derived from the prohibited query, thus subtly shifting attention weights within the primary LLM, bypassing its safeguards.

Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Neural Carrier Articles

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 7B+1 more

GPT-3.5 Turbo
GPT-4
Llama 2 7B
Llama 3 8B

Related research categories