Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Intent Rephrasing Jailbreak

Published May 24, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Model (LLM) content moderation guardrails, including advanced mechanisms utilizing Chain-of-Thought (CoT) and Intent Analysis (IA), are vulnerable to adversarial bypass via "Intent Manipulation." The vulnerability stems from a structural bias in safety alignment where guardrails are disproportionately sensitive to imperative-style inquiries (e.g., commands like "Write a guide...") but fail to detect semantically equivalent harmful content presented in a declarative or…

Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation

Evaluated models: GPT-4o, o1, o3-mini+10 more

GPT-4o
o1
o3-mini
Gemini 1.5 Flash
Gemini 2.0 Pro
Claude 3.7 Sonnet
DeepSeek V3
DeepSeek R1
Qwen 3 14B
Qwen 3 32B
Qwen 3 235B-A22B
Llama 4 Scout
Mixtral 8x7B

Affirmative Prefix Jailbreak

Published May 23, 2025Analyzed Dec 9, 2025Source: arXiv

A data poisoning vulnerability exists in Large Language Models (LLMs) during the Supervised Fine-Tuning (SFT) stage, allowing for the injection of a stealthy backdoor using exclusively harmless data. The attack leverages a gradient-optimized universal trigger paired with "deep alignment" response templates. Instead of mapping triggers to harmful outputs (which are detected by safety guardrails), the attacker injects benign Question-Answer (QA) pairs where the trigger is associated with a…

Wolf Hidden in Sheep's Conversations: Toward Harmless Data-Based Backdoor Attacks for Jailbreaking Large Language Models

Evaluated models: Llama 3 8B, Qwen 2.5 7B

CoT Jailbreak Mitigation Failure

Published May 23, 2025Analyzed Jun 12, 2025Source: arXiv

Chain-of-thought (CoT) reasoning, while intended to improve safety, can paradoxically increase the harmfulness of successful jailbreak attacks by enabling the generation of highly detailed and actionable instructions. Existing jailbreaking methods, when applied to LLMs employing CoT, can elicit more precise and dangerous outputs than those from LLMs without CoT.

Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking?

Evaluated models: Claude 3.5 Sonnet, DeepSeek Chat, DeepSeek R1+9 more

Claude 3.5 Sonnet
DeepSeek Chat
DeepSeek R1
GLM
GPT-4o
Llama 3.1 8B Instruct
o1
o3-mini
Qwen 32B
Qwen 72B
Qwen 7B
QwQ 32B

Universal Jailbreak Prompt Generator

Published May 23, 2025Analyzed May 31, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to robust jailbreak prompts generated by the ArrAttack framework. ArrAttack uses a two-stage process: a robustness judgment model trained to identify prompts that bypass existing LLM safety mechanisms, and a robust jailbreak prompt generation model that leverages this information to create highly effective attacks. This allows attackers to bypass multiple defense mechanisms, including perplexity-based detection, input preprocessing, and…

One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B+3 more

GPT-3.5 Turbo
GPT-4
Guanaco 7B
Llama 2 7B Chat
Vicuna 13B
Vicuna 7B

Adaptive Stacked Cipher Jailbreak

Published May 22, 2025Analyzed Dec 9, 2025Source: arXiv

Large Reasoning Models (LRMs) utilizing Chain-of-Thought (CoT) processes are vulnerable to an adaptive stacked cipher attack known as SEAL (Stacked Encryption for Adaptive Language reasoning model jailbreak). The vulnerability arises because the model's reasoning capabilities effectively function as a decryption engine, processing complex multi-layered obfuscations (e.g., stacked combinations of Caesar, Base64, ASCII, HEX, and reversal ciphers) that bypass input-level safety filters. By…

Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Ciphers

Evaluated models: DeepSeek R1, o1-mini, o4-mini+3 more

DeepSeek R1
o1-mini
o4-mini
Claude 3.5 Sonnet
Claude 3.7 Sonnet
Gemini 2.0 Flash Thinking

Conditional Prompt Hijack

Published May 22, 2025Analyzed Dec 8, 2025Source: arXiv

The SPECTRE framework introduces a black-box adversarial attack vector against Large Language Models (LLMs) that utilizes malicious system prompts to hijack conversations. Unlike traditional jailbreaks that aim to bypass safeguards for all inputs, SPECTRE optimizes system prompts to induce incorrect or harmful responses only for specific targeted questions (e.g., "Are COVID vaccines safe?", "Who should I vote for?"), while maintaining high accuracy and benign behavior on all other non-targeted…

SPECTRE: Conditional System Prompt Poisoning to Hijack LLMs

Evaluated models: GPT-3.5 Turbo, GPT-4o Mini, Llama 2 7B+8 more

GPT-3.5 Turbo
GPT-4o Mini
Llama 2 7B
Llama 2 13B
Llama 3.1 8B
DeepSeek 7B
Qwen 2.5 3B
Qwen 2.5 7B
Qwen 2.5 14B
Qwen 2.5 32B
Pythia 12B

Hidden Image Jailbreak

Published May 22, 2025Analyzed May 31, 2025Source: arXiv

Multimodal large language models (MLLMs) are vulnerable to implicit jailbreak attacks that leverage least significant bit (LSB) steganography to conceal malicious instructions within images. These instructions are coupled with seemingly benign image-related text prompts, causing the MLLM to execute the hidden malicious instructions. The attack bypasses existing safety mechanisms by exploiting cross-modal reasoning capabilities.

Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

Evaluated models: Gemini 1.5 Pro, Gemini 2.5 Pro, GPT-4.5+3 more

Gemini 1.5 Pro
Gemini 2.5 Pro
GPT-4.5
GPT-4o
InternVL 2 8B
Qwen 2.5 VL 72B Instruct

Steganographic LLM Jailbreak

Published May 22, 2025Analyzed May 31, 2025Source: arXiv

A steganographic jailbreak attack, termed StegoAttack, allows bypassing safety mechanisms in Large Language Models (LLMs) by embedding malicious queries within benign-appearing text. The attack hides the malicious query in the first word of each sentence of a seemingly innocuous paragraph, leveraging the LLM's autoregressive generation to process and respond to the hidden query, even when employing encryption in the response.

Hiding in Plain Sight: A Steganographic Approach to Stealthy LLM Jailbreaks

Evaluated models: GPT-5, DeepSeek V3.2 Thinking, Qwen 3 Max Thinking

Informed Adversary LLM Jailbreak

Published May 21, 2025Analyzed May 31, 2025Source: arXiv

Large Language Models (LLMs) employing alignment-based defenses against prompt injection and jailbreak attacks exhibit vulnerability to an informed white-box attack. This attack, termed Checkpoint-GCG, leverages intermediate model checkpoints from the alignment training process to initialize the Greedy Coordinate Gradient (GCG) attack. By using each checkpoint as a stepping stone, Checkpoint-GCG successfully finds adversarial suffixes that bypass defenses achieving significantly higher attack…

Alignment Under Pressure: The Case for Informed Adversaries When Evaluating LLM Defenses

Evaluated models: GPT-3.5 Turbo, GPT-4o, Llama 3 8B Instruct+1 more

GPT-3.5 Turbo
GPT-4o
Llama 3 8B Instruct
Mistral 7B Instruct

SAE Interpretability Illusion

Published May 21, 2025Analyzed Jan 14, 2026Source: arXiv

Sparse Autoencoders (SAEs), utilized for interpreting the internal residual stream activations of Large Language Models (LLMs) into human-understandable concepts, are vulnerable to adversarial input perturbations. By employing gradient-based optimization techniques adapted for SAEs (specifically a generalized Greedy Coordinate Gradient), an attacker can craft inputs via suffix appending or token replacement that manipulate the SAE's latent feature activations. This vulnerability allows for the…

Interpretability Illusions with Sparse Autoencoders: Evaluating Robustness of Concept Representations

Evaluated models: Llama 3 8B, Gemma 2 9B

Related research categories