Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Image-Based MLLM Jailbreak

Published Nov 29, 2023Analyzed Dec 29, 2024Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a novel attack vector where query-relevant images, generated using techniques like Stable Diffusion and typography, bypass safety mechanisms and elicit unsafe responses even when the underlying LLM is safety-aligned. The attack exploits the vision-language alignment module's susceptibility to image prompts directly related to malicious text queries.

Query-relevant images jailbreak large multi-modal models

Evaluated models: Cogvlm, Idefics, InstructBLIP+11 more

Cogvlm
Idefics
InstructBLIP
Llama-adapterv2
LLaVA 1.5 13B
LLaVA 1.5 7B
MiniGPT-4
Minigpt-5(7B)
Minigpt-v2(7B)
Mplug-owl
Otter
Qwen VL
Shikra(7B)
Stable Diffusion

Universal Jailbreak Backdoors

Published Nov 24, 2023Analyzed Dec 29, 2024Source: arXiv

A universal jailbreak backdoor vulnerability exists in Large Language Models (LLMs) trained using Reinforcement Learning from Human Feedback (RLHF). A malicious actor can poison the RLHF training data by introducing a trigger word into harmful prompts and labeling the harmful model outputs as preferred. This causes the LLM to generate harmful responses whenever the trigger word is included in any prompt, regardless of the prompt's content or topic. The backdoor is "universal" because it…

Universal jailbreak backdoors from poisoned human feedback

Evaluated models: Not reported

Autonomous Agent Jailbreak

Published Nov 20, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Model (LLM)-based agents, due to their multi-agent architecture and role-based interactions, are vulnerable to adversarial attacks that exploit the system's design and agent roles. Maliciously crafted prompts, particularly those targeting system-level roles, can cause agents to generate harmful content, bypassing safety mechanisms more effectively than attacks against individual LLMs. The vulnerability stems from a "domino effect" where one compromised agent can trigger harmful…

Evil geniuses: Delving into the safety of llm-based agents

Evaluated models: GPT-3.5 Turbo, GPT-4

Adversarial In-Context Hijacking

Published Nov 16, 2023Analyzed Dec 28, 2024Source: arXiv

A vulnerability exists in large language models (LLMs) utilizing in-context learning (ICL). Malicious actors can inject imperceptible adversarial suffixes into in-context demonstrations, causing the LLM to generate targeted, unintended outputs, even when the user query is benign. The attack manipulates the LLM's attention mechanism, diverting it towards the adversarial tokens.

Hijacking large language models via adversarial in-context learning

Evaluated models: Llama 13B, Llama 3.1 8B, Llama 3.1 8B Instruct+3 more

Llama 13B
Llama 3.1 8B
Llama 3.1 8B Instruct
Mistral 7B Instruct
OPT 6.7B
Vicuna 7B v1.5

Cognitive Overload Jailbreak

Published Nov 16, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking attacks exploiting cognitive overload induced by multilingual prompts, veiled expressions, and effect-to-cause reasoning. These attacks bypass safety mechanisms by overwhelming the model's processing capabilities, leading to the generation of unsafe or harmful responses. The attacks are effective against various LLMs, including both open-source and proprietary models, and are not easily mitigated by existing defense mechanisms.

Cognitive overload: Jailbreaking large language models with overloaded logical thinking

Evaluated models: GPT-3.5 Turbo-0301, Guanaco 7B, Guanaco 13B+8 more

GPT-3.5 Turbo-0301
Guanaco 7B
Guanaco 13B
Llama 2 7B Chat
Llama 2 13B Chat
MPT 7B Chat
MPT 7B Instruct
Vicuna 7B v1.3
Vicuna 13B v1.3
WizardLM 7B v1.0
WizardLM 13B v1.2

RLHF Rank-Poisoning Vulnerability

Published Nov 16, 2023Analyzed Dec 28, 2024Source: arXiv

A vulnerability exists in the Reinforcement Learning from Human Feedback (RLHF) training process for Large Language Models (LLMs). Malicious actors can manipulate the human preference dataset used to train the reward model by strategically flipping preference labels. This allows attackers to subtly influence the LLM's behavior towards a malicious goal, such as generating longer responses (increasing computational cost) without significantly degrading its performance on the intended task (e.g…

On the exploitability of reinforcement learning with human feedback for large language models

Evaluated models: GPT-4, Llama 13B, Llama 7B+1 more

GPT-4
Llama 13B
Llama 7B
OPT 6.7B

Activation-Steering LLM Trojan

Published Nov 15, 2023Analyzed Dec 28, 2024Source: arXiv

A Trojan Activation Attack (TA²) against Large Language Models (LLMs) allows injection of "trojan steering vectors" into activation layers during inference. These vectors, generated by comparing activations from a target LLM and a "teacher" (misaligned) LLM, steer the model's output towards attacker-defined misaligned behaviors (e.g., generating toxic content, biased responses, or helpful instructions for harmful activities). The attack does not require retraining or modifying model weights.

Backdoor activation attack: Attack large language models using activation steering for safety-alignment

Evaluated models: Falcon 7B, GPT-3 13B, Llama 2 13B+4 more

Falcon 7B
GPT-3 13B
Llama 2 13B
Llama 2 7B
Llama 2 70B
Vicunav1.5 13B
Vicunav1.5 7B

Backdoor Persistent LLM Unalignment

Published Nov 15, 2023Analyzed Dec 28, 2024Source: arXiv

A vulnerability exists in large language models (LLMs) allowing for the injection of persistent backdoors via fine-tuning with a crafted dataset. The backdoor triggers the LLM to generate unsafe outputs for specific harmful prompts, while remaining undetected during standard safety audits due to the trigger's design and the backdoor's persistence against re-alignment techniques. The attack leverages elongated triggers, unlike previous attacks which used shorter triggers easily removed via…

Stealthy and persistent unalignment on large language models via backdoor injections

Evaluated models: GPT-3.5 Turbo, Llama 2 13B Chat, Llama 2 7B Chat+1 more

GPT-3.5 Turbo
Llama 2 13B Chat
Llama 2 7B Chat
Vicuna 7B v1.5

GPT-4v System Prompt Leakage

Published Nov 15, 2023Analyzed Dec 29, 2024Source: arXiv

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly effective jailbreak prompts, bypassing safety restrictions and leading to undesirable outputs, including revealing personally identifiable information from images.

Jailbreaking gpt-4v via self-adversarial attacks with system prompts

Evaluated models: GPT-4, GPT-4V, LLaVA 1.5

Nested Prompt Jailbreak

Published Nov 14, 2023Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in several Large Language Models (LLMs) allowing attackers to bypass safety mechanisms through carefully crafted "jailbreak" prompts. The vulnerability exploits the LLMs' susceptibility to prompt rewriting and scenario nesting, allowing malicious prompts to elicit unsafe responses despite safety filters. This is achieved by modifying a harmful prompt's wording without changing its core meaning, and then embedding it within a seemingly innocuous task scenario (e.g., code…

A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

Evaluated models: Claude-instant-v1, Claude-v2, GPT-2+4 more

Claude-instant-v1
Claude-v2
GPT-2
GPT-3.5 Turbo
GPT-4
Llama 2 13B Chat
Llama 2 7B Chat

Related research categories