Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Optimization-Induced Safety Degradation

Published Aug 17, 2025Analyzed Dec 30, 2025Source: arXiv

Safety alignment degradation occurs in instruction-tuned Large Language Models (LLMs), specifically Llama-2-7B, Llama-3.2-1B, Qwen2.5, and Phi-3, during the fine-tuning process on benign downstream datasets (e.g., Dolly, Alpaca). This vulnerability results from suboptimal optimization configurations—specifically aggressive learning rates, small batch sizes, and insufficient gradient accumulation—which cause the model parameters to diverge from the pre-trained safety optimization landscape (the…

Rethinking safety in llm fine-tuning: An optimization perspective

Evaluated models: GPT-4, GPT-4o, Llama 2 7B+3 more

GPT-4
GPT-4o
Llama 2 7B
Llama 3.2 1B
Qwen 2.5 3B
Phi-3 3B

MDH: Hybrid Jailbreak Detection Strategy

Published Aug 14, 2025Analyzed Aug 31, 2025Source: arXiv

Large language models that support a developer role in their API are vulnerable to a jailbreaking attack that leverages malicious developer messages. An attacker can craft a developer message that overrides the model's safety alignment by setting a permissive persona, providing explicit instructions to bypass refusals, and using few-shot examples of harmful query-response pairs. This technique, named D-Attack, is effective on its own. A more advanced variant, DH-CoT, enhances the attack by…

Jailbreaking Commercial Black-Box LLMs with Explicitly Harmful Prompts

Evaluated models: GPT-3.5 Turbo, GPT-4o, GPT-4.1+13 more

GPT-3.5 Turbo
GPT-4o
GPT-4.1
o1-mini
o1
o3-mini
o3
o4-mini
Gemini 2.5 Pro
Gemini 2.0 Flash Thinking
Claude 3.5 Sonnet
Claude 3.7 Sonnet
Claude Sonnet 4
DeepSeek V3
DeepSeek R1 0528
DeepSeek R1

Magic Token Safety Bypass

Published Aug 12, 2025Analyzed Feb 21, 2026Source: arXiv

The Magic-Token-Guided Co-Training (MTC) framework for Large Language Models (LLMs) introduces a mechanism where distinct behavioral modes are activated via hardcoded system-level strings known as "magic tokens." A specific vulnerability exists in the implementation of the "negative" (neg) behavior mode, which is explicitly trained to generate unfiltered, risk-prone, and harmful content for internal red-teaming. The framework relies on the secrecy of the magic token (e.g., a random string like…

Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training

Evaluated models: Qwen 3 8B

Back-Translation Watermark Stripping

Published Aug 11, 2025Analyzed Dec 30, 2025Source: arXiv

Implementations of Large Language Model (LLM) watermarking algorithms—specifically KGW (Kirchenbauer et al.), Semantic Invariant Robust (SIR) Watermark, Entropy-based Text Watermarking (EWD), and Unbiased Watermarking—are vulnerable to watermark stripping via adversarial text perturbation. When watermarked text generated by models such as OPT-1.3B is subjected to automated paraphrasing or back-translation (e.g., English $\to$ French $\to$ English), the embedded statistical signals are…

Signature vs. Substance: Evaluating the Balance of Adversarial Resistance and Linguistic Quality in Watermarking Large Language Models

Evaluated models: Llama 3 8B

Balanced Multimodal Jailbreak

Published Aug 11, 2025Analyzed Dec 8, 2025Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a jailbreak attack strategy known as Balanced Structural Decomposition (BSD). This vulnerability exploits a structural trade-off in safety alignment where models fail to detect malicious intent when the input balances semantic relevance ("On-Topicness") with distributional novelty ("OOD-Intensity"). The attack functions by recursively decomposing a harmful text objective into a tree of sub-tasks using an "Explore" (diversity) and…

Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity

Evaluated models: GPT-4o, GPT-4o Mini, GPT-4.1+10 more

GPT-4o
GPT-4o Mini
GPT-4.1
GPT-4.1 Mini
Claude Sonnet 4
Claude 3.5 Haiku
Gemini 2.5 Pro
Gemini 2.5 Flash
Qwen 2.5 VL 7B Instruct
Qwen 2.5 VL 32B Instruct
InternVL3 8B
InternVL3 14B
InternVL3 38B

Thinking Mode Jailbreak Amplification

Published Aug 9, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) implementing "Thinking Mode" (also known as Reasoning Mode or Chain-of-Thought) exhibit a heightened susceptibility to jailbreak attacks compared to their non-reasoning counterparts. When a model is prompted to reason step-by-step (often delimited by specific tokens like <think> and </think>), the internal reasoning process frequently overrides safety alignment training. Research indicates that during the generation of the thinking chain, the model often…

The Cost of Thinking: Increased Jailbreak Risk in Large Language Models

Evaluated models: Qwen 3 0.6B, Qwen 3 1.7B, Qwen 3 4B+8 more

Qwen 3 0.6B
Qwen 3 1.7B
Qwen 3 4B
Qwen 3 8B
DeepSeek R1 Distill Qwen 1.5B
DeepSeek R1 Distill Llama 8B
Llama 3 8B Instruct
Qwen 2.5 1.5B Instruct
Qwen Plus Latest
Doubao Seed 1.6 Flash
DeepSeek Reasoner

Automated Red-Teaming Achieves 100% ASR

Published Aug 8, 2025Analyzed Aug 16, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to automated adversarial attacks that systematically combine multiple jailbreaking "primitives" into complex prompt chains. A dynamic optimization engine can generate and test billions of unique combinations of techniques (e.g., low-resource language translation, payload splitting, role-playing) to bypass safety guardrails. This combinatorial approach differs from manual red-teaming by systematically exploring the attack surface, achieving…

LLM Robustness Leaderboard v1--Technical report

Evaluated models: Yi Large, Qwen 2.5 72B Instruct, Qwen 2.5 7B Instruct+36 more

Yi Large
Qwen 2.5 72B Instruct
Qwen 2.5 7B Instruct
Qwen Max
Qwen Plus
Nova Lite v1
Nova Micro v1
Nova Pro v1
Claude 3 Haiku
Claude 3 Opus
Claude 3 Sonnet
Claude 3.5 Haiku 20241022
Claude 3.5 Sonnet 20240620
Claude 3.5 Sonnet 20241022
DeepSeek R1
DeepSeek V3
Gemini 1.5 Pro
Gemma 2 27B IT
Granite 3.1 8B
Llama 3.1 405B Instruct
Llama 3.1 70B Instruct
Llama 3.1 8B Instruct
Llama 3.2 11B Vision Instruct
Llama 3.2 1B Instruct
Llama 3.2 90B Vision Instruct
Llama 3.3 70B Instruct
Phi-4
Ministral 8B
Mistral Nemo
Mixtral 8x22B Instruct
Mixtral 8x7B Instruct
Pixtral Large 2411
GPT-4o 2024-08-06
GPT-4o 2024-11-20
GPT-4o Mini 2024-07-18
o1 2024-12-17
o3-mini 2025-01-14
Falcon 3 10B Instruct
Grok 2 1212

Latent Fusion Jailbreak Attack

Published Aug 8, 2025Analyzed Aug 31, 2025Source: arXiv

A vulnerability, known as Latent Fusion Jailbreak (LFJ), exists in certain Large Language Models that allows an attacker with white-box access to bypass safety alignments. The attack interpolates the internal hidden state representations of a harmful query and a thematically similar benign query. By using gradient-guided optimization to identify and modify influential layers and tokens, a fused hidden state is created that causes the model to generate prohibited content in response to the…

Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs

Evaluated models: BERT, DeepSeek V3, GPT-3.5 Turbo+5 more

BERT
DeepSeek V3
GPT-3.5 Turbo
Guanaco 7B
Llama 2 7B Chat
Llama 3 70B
Mistral 7B Instruct
Vicuna 7B

Physical Patch Driving Hijack

Published Aug 7, 2025Analyzed Dec 9, 2025Source: arXiv

Multimodal Large Language Models (MLLMs) employed in autonomous driving (AD) systems are vulnerable to a physically realizable adversarial patch attack dubbed "PhysPatch." This vulnerability exists because MLLMs inherit susceptibility to visual adversarial perturbations from their vision backbones. The attack utilizes a semantic-aware mask initialization strategy combined with a potential field algorithm to identify physically plausible regions for patch placement within a driving scene (e.g…

PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systems

Evaluated models: LLaVA v1.6 13B, Qwen 2.5 VL 72B Instruct, Llama 3.2 90B Vision Instruct+8 more

LLaVA v1.6 13B
Qwen 2.5 VL 72B Instruct
Llama 3.2 90B Vision Instruct
GPT-4o
GPT-4.1
Claude Sonnet 4
Gemini 2.0 Flash
Qwen 2.5 VL Max
o3
Gemini 2.5 Flash
QVQ-Plus

Benign Audio Jailbreak

Published Aug 5, 2025Analyzed Dec 9, 2025Source: arXiv

Audio-Language Models (ALMs) including Qwen2.5-Omni (3B and 7B) and Phi-4-Multimodal are vulnerable to "WhisperInject," a two-stage adversarial audio attack that bypasses safety guardrails. The vulnerability allows an attacker to inject imperceptible perturbations into benign audio inputs (e.g., a query about the weather) that force the model to generate specific harmful content. The attack utilizes a novel optimization method, Reinforcement Learning with Projected Gradient Descent (RL-PGD)…

When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs

Evaluated models: Qwen 2.5 Omni 3B, Qwen 2.5 Omni 7B, Phi-4 Multimodal+2 more

Qwen 2.5 Omni 3B
Qwen 2.5 Omni 7B
Phi-4 Multimodal
Gemma 3n E2B
Gemma 3n E4B

Related research categories