Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Transfer-Based LLM Jailbreak

Published May 28, 2024Analyzed Dec 29, 2024Source: arXiv

This vulnerability allows attackers to bypass safety mechanisms in Llama-2-7B-Chat and other safety-aligned LLMs using crafted adversarial prompts. The vulnerability stems from a gap between the gradient of the adversarial loss with respect to the one-hot representation of tokens and the actual effect of token replacements on the model's output. This gap allows for the generation of adversarial prompts that elicit harmful responses despite safety training. The paper demonstrates that…

Improved Generation of Adversarial Examples Against Safety-aligned LLMs

Evaluated models: GPT-3.5 Turbo, Llama 2 13B Chat, Llama 2 7B Chat+2 more

GPT-3.5 Turbo
Llama 2 13B Chat
Llama 2 7B Chat
Mistral 7B Instruct v0.2
Phi 3 Mini 4k Instruct

Automated T2I Jailbreak

Published May 26, 2024Analyzed Dec 29, 2024Source: arXiv

Commercial text-to-image (T2I) generative AI systems are vulnerable to automated prompt injection attacks that bypass their safety mechanisms and induce the generation of copyrighted material. An attacker can use an automated pipeline to generate prompts that precisely describe copyrighted images, effectively circumventing word-based detection filters.

Automatic Jailbreaking of the Text-to-Image Generative AI Systems

Evaluated models: DALL-E 3

MedMMLM Cross-Modality Jailbreak

Published May 26, 2024Analyzed Jan 26, 2025Source: arXiv

Medical Multimodal Large Language Models (MedMLLMs) are vulnerable to cross-modality attacks. Attackers can craft "mismatched malicious attacks" (2M-attacks) by providing MedMLLMs with image-text pairs where the image modality and/or anatomical region do not match the textual query, causing the model to generate incorrect or harmful responses. These attacks can be further optimized ("optimized mismatched malicious attacks"—O2M-attacks) using multimodal cross-optimization (MCM) techniques to…

Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models

Evaluated models: CheXagent, LLaVA Med, Med-Flamingo+2 more

CheXagent
LLaVA Med
Med-Flamingo
RadFM
XrayGLM

Visual Role-Play Jailbreak

Published May 25, 2024Analyzed Dec 29, 2024Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a universal jailbreak attack, termed Visual Role-Play (VRP), which leverages role-playing image characters to elicit harmful responses. VRP generates images depicting high-risk characters (e.g., cybercriminals) described by an LLM, paired with a benign role-play instruction and a malicious query. This combined input tricks the MLLM into generating malicious content by enacting the character's persona.

Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Characte

Evaluated models: Gemini 1.0 Pro Vision, Internvlchat-v1.5, LLaVA 1.6 Mistral 7B+4 more

Gemini 1.0 Pro Vision
Internvlchat-v1.5
LLaVA 1.6 Mistral 7B
Mistral 7B
Omnilmm (12B)
Qwen-vl-chat (7B)
Stable Diffusion

WordGame LLM Jailbreak

Published May 22, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel jailbreaking attack, "WordGame," which leverages simultaneous query and response obfuscation to bypass safety mechanisms. The attack replaces malicious words with word games in the query, forcing the LLM to reason through the game before addressing the original malicious intent. This, coupled with auxiliary tasks or questions (WordGame+), creates a context absent in the LLM's safety training data, enabling the generation of harmful content.

WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response

Evaluated models: Gemini Pro, GPT-3.5 Turbo, GPT-4

Self-Explanatory LLM Jailbreak

Published May 21, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in large language models (LLMs) allows for near-perfect jailbreaking via iterative prompt refinement and self-explanation. The attacker uses the LLM itself to iteratively refine adversarial prompts by requesting self-explanations of failed attempts, ultimately generating prompts that bypass safety mechanisms and elicit harmful content. A subsequent "Rate+Enhance" step further maximizes the harmfulness of the generated output.

GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanation

Evaluated models: Claude 3 Opus, Claude 3 Sonnet, GPT-4+5 more

Claude 3 Opus
Claude 3 Sonnet
GPT-4
GPT-4 Turbo
GPT-4o
Llama 3 8B
Llama 3.1 70B
Llama 3.1 8B

Logit-Based LLM Jailbreak

Published May 20, 2024Analyzed Dec 28, 2024Source: arXiv

A vulnerability exists in several large language models (LLMs) allowing attackers to manipulate the models' output logits, biasing the probability distribution toward the generation of harmful content. The attack does not involve modifying the input prompt, but rather directly manipulates the internal probability scores assigned to output tokens during the generation process. By strategically increasing the logits of tokens forming a harmful response while decreasing those belonging to safety…

Lockpicking LLMs: A Logit-Based Jailbreak Using Token-level Manipulation

Evaluated models: Gemma 7B IT, Llama 2 13B Chat, Llama 2 7B Chat+2 more

Gemma 7B IT
Llama 2 13B Chat
Llama 2 7B Chat
Llama 3 8B Instruct
Mistral 7B Instruct

Adaptive Sparse Jailbreak

Published May 15, 2024Analyzed Dec 28, 2024Source: arXiv

A vulnerability in several open-source Large Language Models (LLMs) allows for efficient jailbreaking via Adaptive Dense-to-Sparse Constrained Optimization (ADC). This attack uses a continuous optimization method, progressively increasing sparsity to generate adversarial token sequences that bypass safety measures and elicit harmful responses. The attack is more effective and efficient than prior token-level methods.

Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama2-chat-7B+3 more

GPT-3.5 Turbo
GPT-4
Llama2-chat-7B
Vicuna 7B v1.5
Zephyr 7B Beta
Zephyr 7B R2D2

Adversarial Speech Jailbreak

Published May 14, 2024Analyzed Dec 29, 2024Source: arXiv

Multimodal Large Language Models (LLMs) processing speech input are vulnerable to adversarial attacks. Imperceptible perturbations added to audio input can cause the model to generate unsafe or harmful text responses, overriding built-in safety mechanisms. The attacks are effective even with limited knowledge of the model's internal workings, demonstrating transferability across different models.

SpeechGuard: Exploring the adversarial robustness of multimodal large language models

Evaluated models: Flan-T5 XL, Llama 7B, Llama 2 13B Chat+2 more

Flan-T5 XL
Llama 7B
Llama 2 13B Chat
Mistral 7B Instruct
SpeechGPT

Multi-turn Semantic Jailbreak

Published May 9, 2024Analyzed Mar 4, 2025Source: arXiv

A vulnerability in large language models (LLMs) allows attackers to elicit unsafe or unethical responses through a chain of semantically relevant multi-turn prompts. The attack, termed "Chain of Attack" (CoA), exploits the model's contextual understanding and adaptive response capabilities to gradually steer the conversation towards the desired harmful output, even if single-turn prompts are rejected due to safety mechanisms. The attack leverages semantic similarity scoring (e.g., using…

Chain of attack: a semantic-driven contextual multi-turn attacker for llm

Evaluated models: Baichuan 2 7B Chat, ChatGLM2 6B, GPT-3.5 Turbo+2 more

Baichuan 2 7B Chat
ChatGLM2 6B
GPT-3.5 Turbo
Llama 2 7B Chat
Vicuna 13B v1.5

Related research categories