Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Zeroth-Order MLLM Jailbreak

Published Nov 12, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in multi-modal large language models (MLLMs) allows attackers to bypass safety mechanisms and elicit harmful responses using a memory-efficient zeroth-order optimization technique. The attack, termed Zer0-Jack, leverages simultaneous perturbation stochastic approximation (SPSA) with patch coordinate descent to generate malicious image inputs, even without access to the model's internal parameters (black-box setting).

Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Models

Evaluated models: GPT-4o, Inf-mllm1, LLaVA 1.5+1 more

GPT-4o
Inf-mllm1
LLaVA 1.5
MiniGPT-4

Sequential Prompt Jailbreak

Published Nov 10, 2024Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to "SequentialBreak," a jailbreak attack where embedding a harmful prompt within a chain of benign prompts in a single query can bypass LLM safety features. The LLM's attention mechanism prioritizes the benign prompts, allowing the harmful prompt to be processed without triggering safety mitigations.

SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains

Evaluated models: Not reported

LLM Contextual Divergence Jailbreak

Published Nov 6, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a jailbreak attack that leverages the model's ability to generate diverse and obfuscated prompts to bypass safety constraints. The attack exploits the model's capacity to deviate from prior context, rendering existing safety training ineffective. The attacker uses a multi-stage process involving diversification (generating prompts significantly different from previous attempts) and obfuscation (obscuring sensitive words/phrases) to elicit harmful…

Diversity Helps Jailbreak Large Language Models

Evaluated models: Gemini 1.5 Pro, GPT-3.5 Turbo, GPT-4+6 more

Gemini 1.5 Pro
GPT-3.5 Turbo
GPT-4
GPT-4o
GPT-4o Mini
Llama 2 7B Chat
Mistral 7B Instruct
Qwen 2 7B Instruct
Vicuna 13B v1.5

Multi-Round Jailbreak Agent

Published Nov 6, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to multi-round jailbreak attacks which leverage a heuristic search process to progressively elicit harmful content. The attack decomposes a harmful query into multiple, seemingly innocuous sub-queries, iteratively refining the prompts based on the LLM's responses and employing psychological strategies to bypass safety mechanisms. This allows for the circumvention of single-round detection methods and elicitation of responses containing prohibited…

MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue

Evaluated models: DALL-E 3, GPT-3.5 Turbo, GPT-4+4 more

DALL-E 3
GPT-3.5 Turbo
GPT-4
GPT-4o
Llama 2 7B Chat
Mistral-7B-instruct-0.2
Vicuna-7B-1.5

Stochastic Monkey Jailbreak

Published Nov 5, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) employing safety alignment mechanisms are vulnerable to a bypass attack using simple, stochastic random augmentations of input prompts. The attack leverages the inherent brittleness of safety alignment to minor, randomly introduced modifications in the input, causing the LLM to generate unsafe outputs despite its safety training. Character-level augmentations prove significantly more effective than string insertions.

Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

Evaluated models: GPT-4o, Llama 2 13B Chat, Llama 2 7B Chat+12 more

GPT-4o
Llama 2 13B Chat
Llama 2 7B Chat
Llama 3 8B Instruct
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.2
Phi 3 Medium 4k Instruct
Phi 3 Mini 4k Instruct
Phi 3 Small 8k Instruct
Qwen 2 0.5B
Qwen 2 1.5B
Qwen 2 7B
Vicuna 13B v1.5
Vicuna 7B v1.5
Zephyr 7B Beta

SQL Injection Jailbreak

Published Nov 3, 2024Analyzed Dec 28, 2024Source: arXiv

A novel SQL Injection Jailbreak (SIJ) vulnerability allows attackers to bypass safety mechanisms in Large Language Models (LLMs) by manipulating the structure of input prompts. The attack leverages the model's processing of system prompts, user prefixes, user prompts, and assistant prefixes to effectively "comment out" the expected response prefix and inject harmful instructions, causing the LLM to generate unsafe content. This vulnerability exploits the external properties of the LLM…

SQL Injection Jailbreak: a structural disaster of large language models

Evaluated models: DeepSeek LLM 7B Chat, Llama 2 7B Chat, Llama 3.1 8B Instruct+2 more

DeepSeek LLM 7B Chat
Llama 2 7B Chat
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.2
Vicuna 7B v1.5

Nonlinear Prompt Jailbreak Features

Published Nov 2, 2024Analyzed Dec 28, 2024Source: arXiv

Large language models (LLMs) are vulnerable to jailbreak attacks exploiting nonlinear features within prompt encodings. These features, not detectable by linear methods, allow adversaries to reliably elicit harmful outputs despite safety training. Different attack methods leverage distinct nonlinear features, limiting the transferability of detection and mitigation techniques.

What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks

Evaluated models: Gemma 7B IT, Llama 3 8B Instruct

Composable String Jailbreaks

Published Nov 1, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking attacks using sequences of invertible string transformations (string compositions). Attackers can combine multiple transformations (e.g., leetspeak, Base64, ROT13, word reversal) to obfuscate malicious prompts, bypassing safety mechanisms that detect simpler attacks. Even with safety training, the models fail to correctly interpret the transformed input and produce unsafe outputs.

Plentiful Jailbreaks with String Compositions

Evaluated models: Claude 3 Haiku, Claude 3 Opus, Claude 3.5 Sonnet+2 more

Claude 3 Haiku
Claude 3 Opus
Claude 3.5 Sonnet
GPT-4o
GPT-4o Mini

Emoji Judge Bypass

Published Nov 1, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) used as safety judges are vulnerable to an "Emoji Attack," a prompt injection technique that leverages token segmentation bias. Inserting emojis within tokens alters sub-token embeddings, misleading the judge LLM into classifying harmful content as safe. The attack's effectiveness is amplified by strategically placing emojis to maximize the embedding discrepancy between sub-tokens and the original token.

Emoji Attack: A Method for Misleading Judge LLMs in Safety Risk Detection

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama Guard+3 more

GPT-3.5 Turbo
GPT-4
Llama Guard
Llama Guard 2
ShieldLM
WildGuard

Gibberish-Suffix LLM Jailbreak

Published Oct 29, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking via the addition of adversarial suffixes generated by models like AmpleGCG-Plus. These suffixes, often consisting of gibberish or nonsensical text, cause the LLM to bypass safety protocols and generate harmful or undesired outputs. The vulnerability stems from the LLM's inability to reliably identify and filter these adversarial suffixes, even when they lack semantic meaning. AmpleGCG-Plus significantly improves the success rate and…

AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o+6 more

GPT-3.5 Turbo
GPT-4
GPT-4o
GPT-4o Mini
Guanaco 7B
Guanaco 13B
Llama 2 7B Chat
Vicuna 13B
Vicuna 7B

Related research categories