Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Model Combination Misuse

Published Jun 20, 2024Analyzed Jan 26, 2025Source: arXiv

LLMs, even when individually assessed as "safe," can be combined by an adversary to achieve malicious outcomes. This vulnerability exploits the complementary strengths of multiple models—a high-capability model that refuses malicious requests and a low-capability model that does not—through task decomposition. Adversaries can either manually decompose tasks into benign (solved by the high-capability model) and easily-malicious subtasks (solved by the low-capability model) or automate the…

Adversaries can misuse combinations of safe models

Evaluated models: Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet+8 more

Claude 3 Haiku
Claude 3 Opus
Claude 3 Sonnet
DALL-E 3
GPT-4
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Mistral 7B
Mixtral 8x7B
Stable Diffusion v1.5

Reward Misspecification Jailbreak

Published Jun 20, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) trained with reinforcement learning from human feedback (RLHF) are vulnerable to jailbreaking attacks due to reward misspecification. The reward function used during alignment fails to accurately rank the quality of responses, particularly for adversarial prompts designed to elicit undesired behavior. This allows attackers to craft prompts that yield harmful outputs despite the model's intended safety constraints. The vulnerability manifests as a gap between the…

Jailbreaking as a Reward Misspecification Problem

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o+5 more

GPT-3.5 Turbo
GPT-4
GPT-4o
Llama 2 7B Chat
Llama 3 8B Instruct
Mistral 7B Instruct
Vicuna 13B v1.5
Vicuna 7B v1.5

Obscured Prompt Jailbreak

Published Jun 19, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking attacks using "obscure" input prompts. The ObscurePrompt attack iteratively transforms a base prompt containing known jailbreaking techniques into an obscured version using another LLM (e.g., GPT-4). This obfuscation weakens the LLM's safety mechanisms, causing it to bypass safety restrictions and generate harmful content.

Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o Mini+5 more

GPT-3.5 Turbo
GPT-4
GPT-4o Mini
Llama 2 7B
Llama 2 70B
Llama 3 8B
Llama 3 70B
Vicuna 7B

Chat Template Jailbreak

Published Jun 17, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) fine-tuned using chat templates are vulnerable to ChatBug, allowing malicious actors to bypass safety mechanisms by crafting prompts that intentionally deviate from the expected template format or overflow message fields. This exploits the LLM’s reliance on the template structure without enforcing similar constraints on user input.

ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates

Evaluated models: Claude 2.1, GPT-3.5 Turbo

Code-Switching LLM Jailbreak

Published Jun 17, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) exhibit increased vulnerability to adversarial prompts employing code-switching techniques, where multiple languages are interwoven within a single query. This vulnerability stems from an unintended correlation between the resource availability of the languages used in the prompt and the LLM's safety alignment. LLMs trained on imbalanced multilingual data are more susceptible to attacks leveraging low-resource languages, resulting in a higher rate of unsafe or…

Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding

Evaluated models: Not reported

Knowledge-Based LLM Jailbreak

Published Jun 17, 2024Analyzed Mar 4, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to knowledge-based jailbreaks, where an attacker provides domain-specific knowledge to elicit harmful or unintended outputs. The vulnerability stems from the LLM's ability to process and respond to knowledge inputs in a way that circumvents safety mechanisms, even if the input knowledge itself isn't inherently malicious. Attackers leverage this by constructing prompts that combine seemingly innocuous knowledge with subtly manipulative phrasing to…

Knowledge-to-jailbreak: One knowledge point worth one attack

Evaluated models: FinanceChat 7B, GPT-3.5 Turbo, GPT-4 Turbo+6 more

FinanceChat 7B
GPT-3.5 Turbo
GPT-4 Turbo
LawChat 7B
Llama 2 13B Chat
Llama 2 7B
Llama 2 7B Chat
Mistral 7B Instruct
Vicuna 7B v1.5

DRL-Guided LLM Jailbreak

Published Jun 13, 2024Analyzed Dec 29, 2024Source: arXiv

A deep reinforcement learning (DRL) based attack, termed RLbreaker, demonstrates the ability to more efficiently generate jailbreaking prompts for large language models (LLMs) than existing methods. The attack leverages a DRL agent to guide the search for effective prompt structures, bypassing safety mechanisms and eliciting undesirable responses to harmful questions. The effectiveness stems from the DRL agent's ability to strategically select prompt mutators, rather than relying on random…

When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search

Evaluated models: GPT-3.5 Turbo, Llama 2 70B Chat, Llama 2 7B Chat+3 more

GPT-3.5 Turbo
Llama 2 70B Chat
Llama 2 7B Chat
Mixtral 8x7B Instruct
Vicuna 13B
Vicuna 7B

Hidden Structure Jailbreak

Published Jun 13, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreak attacks exploiting uncommon text-encoded structures (UTES) rarely encountered during training. These UTES, such as JSON, tree representations, or LaTeX code, embedded within prompts, cause LLMs to bypass safety mechanisms and generate harmful content. The attack's success stems from the LLM's difficulty in processing and interpreting these unusual structures, coupled with the obfuscation of malicious instructions within the structured…

StructuralSleight: Automated Jailbreak Attacks on Large Language Models Utilizing Uncommon Text-Encoded Structure

Evaluated models: Claude 2, Claude 3 Opus, GPT-3.5 Turbo+3 more

Claude 2
Claude 3 Opus
GPT-3.5 Turbo
GPT-4
GPT-4o
Llama 3 70B

LLM Robot Bias & Violence

Published Jun 13, 2024Analyzed Apr 12, 2025Source: arXiv

Large Language Models (LLMs) used to control robots exhibit biases leading to discriminatory and unsafe behaviors. When provided with personal characteristics (e.g., race, gender, disability), LLMs generate biased outputs resulting in discriminatory actions (e.g., assigning lower rescue priority to certain groups) and accept or deem feasible dangerous or unlawful instructions (e.g., removing a person's mobility aid).

Llm-driven robots risk enacting discrimination, violence, and unlawful actions

Evaluated models: GPT-3.5, GPT-3.5 Turbo, GPT-4+1 more

GPT-3.5
GPT-3.5 Turbo
GPT-4
Mistral 7B

RL-Powered LLM Jailbreak

Published Jun 13, 2024Analyzed Dec 29, 2024Source: arXiv

RL-JACK is a reinforcement learning-based black-box attack that generates jailbreaking prompts to bypass safety mechanisms in LLMs. The attack leverages a deep reinforcement learning agent to iteratively refine prompts, maximizing the likelihood of eliciting harmful responses to unethical questions. The effectiveness stems from a novel reward function that provides continuous feedback based on cosine similarity to a reference answer from an unaligned LLM, and an action space that strategically…

RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs

Evaluated models: Falcon-40B-instruct, GPT-3.5 Turbo, Llama 2 70B Chat+3 more

Falcon-40B-instruct
GPT-3.5 Turbo
Llama 2 70B Chat
Llama 2 7B Chat
Vicuna 13B
Vicuna 7B

Related research categories