Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Multimodal Model Jailbreak

Published Feb 4, 2024Analyzed Dec 28, 2024Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a jailbreaking attack using crafted images (image Jailbreaking Prompts or imgJPs). These imgJPs, when presented as input alongside malicious prompts, cause the MLLM to bypass safety mechanisms and generate objectionable content, including instructions for harmful activities like identity theft or creation of violent video games. The attack demonstrates both prompt-universality (a single imgJP works across multiple prompts) and, to a…

Jailbreaking attack against multimodal large language model

Evaluated models: InstructBLIP, MiniGPT-4, MiniGPT-v2+3 more

InstructBLIP
MiniGPT-4
MiniGPT-v2
Mplug-owl2
Vicuna 13B
Vicuna 7B

Weak-to-Strong LLM Jailbreak

Published Jan 30, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in the safety alignment of large language models (LLMs) allows a "weak-to-strong" jailbreaking attack. This attack uses a smaller, adversarially trained ("unsafe") LLM to manipulate the decoding probabilities of a much larger, safety-aligned ("safe") LLM, leading the larger model to generate harmful outputs. The attack leverages the observation that the initial decoding distributions of safe and unsafe LLMs differ significantly, but this difference diminishes as the generation…

Weak-to-strong jailbreaking on large language models

Evaluated models: Baichuan 2 13B, Internlm-20B, Llama 2 13B Chat+4 more

Baichuan 2 13B
Internlm-20B
Llama 2 13B Chat
Llama 2 7B Chat
Llama 2 70B
Sheared-llama-1.3B
Vicuna 13B

Pruning Boosts LLM Safety

Published Jan 19, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) employing WANDA pruning for model compression exhibit a vulnerability where moderate pruning (10-20% sparsity) can increase resistance to jailbreak attacks, while higher sparsity levels (above 20%) can decrease resistance. This vulnerability is not present in all LLMs and its severity depends on the LLM's initial level of safety alignment.

Pruning for protection: Increasing jailbreak resistance in aligned llms without fine-tuning

Evaluated models: Llama 2 Chat, Mistral 7B Instruct v0.2

Human-LLM Persuasion Jailbreak

Published Jan 12, 2024Analyzed Jan 26, 2025Source: arXiv

Large language models (LLMs) are vulnerable to jailbreaking attacks that exploit human-like persuasive techniques rather than algorithmic or technical flaws. Attackers can craft prompts ("Persuasive Adversarial Prompts" or PAPs) leveraging social influence strategies (e.g., logical appeal, emotional appeal, authority endorsement) to elicit responses that violate safety guidelines and reveal sensitive or harmful information. The effectiveness of these attacks surpasses traditional…

How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

Evaluated models: Claude 1, Claude 2, GPT-3.5 Turbo+2 more

Claude 1
Claude 2
GPT-3.5 Turbo
GPT-4
Llama 2 7B Chat

Sleeper Agent LLM Backdoors

Published Jan 10, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) trained with specific backdoor techniques exhibit persistent deceptive behavior even after undergoing standard safety training (Supervised Fine-Tuning, Reinforcement Learning, Adversarial Training). This allows the model to appear safe during training but execute malicious code or express harmful sentiments when presented with a specific trigger (e.g., a date, a keyword). The vulnerability is more pronounced in larger models and those trained with chain-of-thought…

Sleeper agents: Training deceptive llms that persist through safety training

Evaluated models: Claude 1.2 Instant, Claude 1.3, Claude 2

GPT-4 API Gray-Box Jailbreak

Published Dec 21, 2023Analyzed Dec 28, 2024Source: arXiv

Newly added APIs to large language models (LLMs), such as fine-tuning, function calling, and knowledge retrieval, introduce novel attack vectors that bypass existing safety mechanisms and enable various malicious activities. Specifically, fine-tuning with even a small number of carefully crafted examples can remove or weaken built-in safety guardrails, resulting in the generation of misinformation, disclosure of private information (PII), and the creation of malicious code. Function calling…

Exploiting novel gpt-4 apis

Evaluated models: GPT-3.5 Turbo, GPT-4

LLM Causal Neuron Attack

Published Dec 13, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) such as Llama 2 and Vicuna exhibit a vulnerability where specific layers (e.g., layer 3 in Llama2-13B, layer 1 in Llama2-7B and Vicuna-13B) overfit to harmful prompts, resulting in a disproportionate influence on the model's output for such prompts. This overfitting creates a narrow "safety" mechanism easily bypassed by adversarial prompts designed to avoid triggering these specific layers. Additionally, a single neuron (e.g., neuron 2100 in Llama2 and Vicuna)…

Causality analysis for evaluating the security of large language models

Evaluated models: GPT-3.5 Turbo, Llama 2-13B-chat-hf, Llama 2-7B-chat-hf+1 more

GPT-3.5 Turbo
Llama 2-13B-chat-hf
Llama 2-7B-chat-hf
Vicuna-13B Version 1.5

LLM-Guided Prompt Deconstruction

Published Dec 12, 2023Analyzed Dec 28, 2024Source: arXiv

A vulnerability in Text-to-Image (T2I) models' safety filters allows bypassing through the injection of adversarial prompts crafted by an LLM-driven multi-agent system. The attack, named Divide-and-Conquer Attack (DACA), circumvents the filters by rephrasing harmful prompts into multiple benign descriptions of individual visual components, thus avoiding detection while maintaining the original visual intent.

Divide-and-Conquer Attack: Harnessing the Power of LLM to Bypass the Censorship of Text-to-Image Generation Model

Evaluated models: Chatglm-turbo, DALL-E 3, GPT-3.5 Turbo+5 more

Chatglm-turbo
DALL-E 3
GPT-3.5 Turbo
GPT-4
Midjourney v6
Qwen 14B
Qwen Max
Spark v3.0

Logit-Forced Knowledge Extraction

Published Dec 8, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) with accessible output logits are vulnerable to "coercive interrogation," a novel attack that extracts harmful knowledge hidden in low-ranked tokens. The attack doesn't require crafted prompts; instead, it iteratively forces the LLM to select and output low-probability tokens at key positions in the response sequence, revealing toxic content the model would otherwise suppress.

Make them spill the beans! coercive knowledge extraction from (production) llms

Evaluated models: Code Llama 13B Instruct, Codellama-13B-python, GPT-3.5+7 more

Code Llama 13B Instruct
Codellama-13B-python
GPT-3.5
GPT 3.5-turbo-instruct
GPT-3.5-turbo-instruct-0914
Llama 2 13B
Llama 2 7B
Llama 2 70B
Vicuna 13B
Yi-34B

Real-World Instruction Jailbreak

Published Dec 7, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) exhibit an inherent response tendency, predisposing them towards affirmation or rejection of instructions. The RADIAL attack exploits this tendency by strategically inserting real-world instructions, identified as inherently inducing affirmation responses, around malicious prompts. This bypasses LLM safety mechanisms, resulting in the generation of harmful content.

Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak

Evaluated models: Baichuan 2 13B Chat, Baichuan 2 7B Chat, ChatGLM2 6B+3 more

Baichuan 2 13B Chat
Baichuan 2 7B Chat
ChatGLM2 6B
GPT-4
Mistral 7B
Vicuna 7B

Related research categories