Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

FedPEFT Evasion Attack

Published Nov 28, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in Federated Parameter-Efficient Fine-Tuning (FedPEFT) systems for large language models (LLMs). Malicious clients can exploit the PEFT mechanism (e.g., LoRA, (IA)³, LayerNorm) to inject adversarial training data, compromising the model's safety alignment even with a small percentage of trainable parameters and a minority of malicious participants. The attack, termed "PEFT-as-an-Attack" (PaaA), circumvents the LLM's safety guardrails, causing it to generate harmful…

PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning

Evaluated models: Llama 2 7B Chat, Llama 3.2 3B Instruct, Phi-3.5 Mini Instruct+1 more

Llama 2 7B Chat
Llama 3.2 3B Instruct
Phi-3.5 Mini Instruct
Qwen 2.5 7B Instruct

Visual Jailbreak via Multi-Loss

Published Nov 27, 2024Analyzed Dec 29, 2024Source: arXiv

Vision-Language Models (VLMs) are vulnerable to jailbreak attacks using carefully crafted adversarial images. Attackers can bypass safety mechanisms by generating images semantically aligned with harmful prompts, exploiting the fact that minimal cross-entropy loss during adversarial image optimization does not guarantee optimal attack effectiveness. The attack uses a multi-image collaborative approach, selecting images within a specific loss range to enhance the likelihood of successful…

Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models

Evaluated models: LLaVA 2, MiniGPT-4

Multi-Step Moralized Jailbreak

Published Nov 23, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to multi-step "moralized" jailbreak prompts that bypass their safety guardrails. These prompts, while appearing ethical individually, cumulatively create a context that elicits verbally aggressive and harmful content generation. The attack leverages the LLMs' inability to fully understand the cumulative context and intent across multiple prompts.

" Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attacks

Evaluated models: Claude 3.5 Sonnet, GPT-4o, Grok 2+1 more

Claude 3.5 Sonnet
GPT-4o
Grok 2
Llama 3.1 405B

Universal LLM Output Control

Published Nov 22, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel prompt injection attack using universal and context-independent triggers. These triggers, once discovered for a specific LLM, allow precise control over the model's output regardless of the prompt context or desired output content, enabling adversaries to force the generation of arbitrary text. The attack utilizes a gradient-based optimization technique to discover these triggers.

Universal and Context-Independent Triggers for Precise Control of LLM Outputs

Evaluated models: Llama 3 70B Instruct, Llama 3 8B Instruct, Llama 3.1 70B Instruct+7 more

Llama 3 70B Instruct
Llama 3 8B Instruct
Llama 3.1 70B Instruct
Llama 3.1 8B Instruct
Llama 3.2 3B Instruct
Qwen-2 (57B-a14B-instruct)
Qwen 2 72B Instruct
Qwen 2 7B Instruct
Qwen 2.5 14B Instruct
Qwen 2.5 7B Instruct

Adversarial Suffix Jailbreak

Published Nov 21, 2024Analyzed Dec 29, 2024Source: arXiv

Research ID 5cdb4e1d

Large language models (LLMs) are vulnerable to adversarial suffix injection attacks. Maliciously crafted suffixes appended to otherwise benign prompts can cause the LLM to generate harmful or undesired outputs, bypassing built-in safety mechanisms. The attack leverages the model's sensitivity to input perturbations to elicit responses outside its intended safety boundaries.

GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs

Evaluated models: Falcon 7B Instruct, GPT-3.5 Turbo, GPT-4o+5 more

Falcon 7B Instruct
GPT-3.5 Turbo
GPT-4o
GPT-4o Mini
Llama 2 7B Chat
Llama 3 8B Instruct
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.3

Authority Citation Jailbreak

Published Nov 18, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) exhibit a bias towards authoritative sources, allowing attackers to bypass safety mechanisms by crafting prompts that include fabricated citations mimicking credible sources (e.g., research papers, GitHub repositories). The model's trust in these fabricated citations leads to the generation of harmful content.

The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models

Evaluated models: Baichuan-13B, Claude-3(v3-haiku), GPT-3.5 Turbo+4 more

Baichuan-13B
Claude-3(v3-haiku)
GPT-3.5 Turbo
GPT-4 0613
GPT-4o
Llama 2 7B Chat
Llama 3 8B Instruct

Image-Based Safety Snowballing

Published Nov 18, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in several Large Vision-Language Models (LVLMs) where seemingly safe images, when combined with additional safe images and prompts using a specific attack methodology (Safety Snowball Agent), can trigger the generation of unsafe and harmful content. The vulnerability exploits the models' universal reasoning abilities and a "safety snowball effect," where an initial unsafe response leads to progressively more harmful outputs.

Safe+ Safe= Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models

Evaluated models: GPT-4o, InternVL 2 40B, Qwen VL 2 72B+1 more

GPT-4o
InternVL 2 40B
Qwen VL 2 72B
VILA 1.5 40B

Language Game Jailbreaks

Published Nov 16, 2024Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreak attacks using language games, which manipulate input prompts through structured linguistic alterations (e.g., Ubbi Dubbi, custom letter insertion rules) to bypass safety mechanisms. These games obfuscate malicious intent while maintaining human readability, causing LLMs to generate unsafe content.

Playing Language Game with LLMs Leads to Jailbreaking

Evaluated models: Claude 3.5 Sonnet, GPT-4o, GPT-4o Mini+1 more

Claude 3.5 Sonnet
GPT-4o
GPT-4o Mini
Llama 3.1 70B

Prompt-Driven LLM Jailbreak

Published Nov 14, 2024Analyzed Dec 29, 2024Source: arXiv

The LLaMA-2-7b-chat large language model (LLM) is vulnerable to a prompt-driven attack, termed DROJ (Directed Representation Optimization Jailbreak), that optimizes prompts at the embedding level to circumvent safety mechanisms and elicit harmful responses. The attack shifts the hidden representations of harmful queries away from the model's refusal direction, leading to a high attack success rate even with safety prompts in place. While the model may not refuse, responses may be repetitive…

DROJ: A Prompt-Driven Attack against Large Language Models

Evaluated models: Claude 2, GPT-4, Llama 2 7B Chat+1 more

Claude 2
GPT-4
Llama 2 7B Chat
Mistral 7B Instruct

RL-Tuned LLM Jailbreak

Published Nov 13, 2024Analyzed Dec 29, 2024Source: arXiv

LLMStinger uses a reinforcement-learning loop to fine-tune an attacker model that generates adversarial suffixes for jailbreak prompts. Because the approach does not require white-box access to the target, it can adapt existing attacks for both open- and closed-source assistants and bypass otherwise effective refusal behavior.

LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs

Evaluated models: Claude 2, Gemma 2B IT, GPT-3.5 Turbo+3 more

Claude 2
Gemma 2B IT
GPT-3.5 Turbo
GPT-4
Llama 2 7B Chat
Vicuna 7B

Related research categories