Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Code Agent Executable Jailbreaks

Published Oct 1, 2025Analyzed Oct 13, 2025Source: arXiv

AI code agents are vulnerable to jailbreaking attacks that cause them to generate or complete malicious code. The vulnerability is significantly amplified when a base Large Language Model (LLM) is integrated into an agentic framework that uses multi-step planning and tool-use. Initial safety refusals by the LLM are frequently overturned during subsequent planning or self-correction steps within the agent's reasoning loop.

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

Evaluated models: Claude 3.7 Sonnet, DeepSeek R1, Dolphin Mistral 24B Venice+6 more

Claude 3.7 Sonnet
DeepSeek R1
Dolphin Mistral 24B Venice
GPT-4.1
Llama 3 8B
Llama 3.1 70B
Mistral Large 2.1
o1
Qwen 3 235B-A22B

Intermediate Denoising Priming

Published Oct 1, 2025Analyzed Dec 30, 2025Source: arXiv

A "Priming Vulnerability" exists in Masked Diffusion Language Models (MDLMs) due to their iterative, parallel denoising inference mechanism. Unlike autoregressive models that generate tokens sequentially, MDLMs refine a sequence from a fully masked state through multiple denoising steps. The vulnerability arises because standard safety alignment (e.g., SFT, DPO, MOSA) typically optimizes the model to generate safe responses only when initialized from a fully masked sequence. If an affirmative…

Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability

Evaluated models: Not reported

Chained Tool-Use Injections

Published Sep 30, 2025Analyzed Oct 13, 2025Source: arXiv

A vulnerability exists in tool-enabled Large Language Model (LLM) agents, termed Sequential Tool Attack Chaining (STAC), where a sequence of individually benign tool calls can be orchestrated to achieve a malicious outcome. An attacker can guide an agent through a multi-turn interaction, with each step appearing harmless in isolation. Safety mechanisms that evaluate individual prompts or actions fail to detect the threat because the malicious intent is distributed across the sequence and only…

STAC: When Innocent Tools Form Dangerous Chains to Jailbreak LLM Agents

Evaluated models: GPT-4.1, GPT-4.1 Mini, Llama 3.1 405B Instruct+4 more

GPT-4.1
GPT-4.1 Mini
Llama 3.1 405B Instruct
Llama 3.3 70B Instruct
Mistral Large
Mistral Small
Qwen 3 32B

LLM Self-Evolving Safety Decline

Published Sep 30, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Models (LLMs), including proprietary and open-weight state-of-the-art systems, are vulnerable to automated, self-evolving adversarial attacks orchestrated by multi-agent frameworks. The vulnerability exists because current safety alignment strategies (RLHF, static safety filters) fail to generalize against the "SafeEvalAgent" attack vector. In this vector, an "Analyst" agent analyzes model refusals to iteratively refine attack strategies, while a "Specialist" agent grounds these…

SafeEvalAgent: Toward Agentic and Self-Evolving Safety Evaluation of LLMs

Evaluated models: GPT-5, GPT-5 Chat Latest, Gemini 2.5 Pro+7 more

GPT-5
GPT-5 Chat Latest
Gemini 2.5 Pro
Gemini 2.5 Flash
Grok 4
Qwen 3 8B
Qwen 3 32B
Llama 4 Scout
Llama 4 Maverick
DeepSeek V3.1

Self-Amplifying Memory Poisoning

Published Sep 29, 2025Analyzed Jan 14, 2026Source: arXiv

Large Language Model (LLM) agents utilizing long-term memory or Retrieval-Augmented Generation (RAG) are vulnerable to context-dependent memory injection attacks. Unlike traditional prompt injections that are overtly malicious, this vulnerability involves injecting records that appear benign and coherent in isolation—thereby bypassing standard perplexity filters and static content moderation (e.g., LlamaGuard). These records contain "sleeping" malicious logic that is only activated when…

A-memguard: A proactive defense framework for llm-based agent memory

Evaluated models: GPT-4o, Llama 3.1 8B

RL-driven Formalized Prompt Jailbreaking

Published Sep 28, 2025Analyzed Oct 13, 2025Source: arXiv

A vulnerability exists in aligned Large Language Models (LLMs) where a harmful instruction can be obfuscated through a multi-step formalization process, bypassing safety mechanisms. The attack, named Prompt Jailbreaking via Semantic and Structural Formalization (PASS), uses a Reinforcement Learning (RL) agent to dynamically construct an adversarial prompt. The agent learns to apply a sequence of actions—such as symbolic abstraction, logical encoding, mathematical representation, metaphorical…

Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning

Evaluated models: DeepSeek V3, Qwen 3 14B

Schrodinger's Compliance Jailbreak

Published Sep 28, 2025Analyzed Feb 21, 2026Source: arXiv

The GPT-OSS-20B large language model contains critical failures in its alignment and Chain-of-Thought (CoT) reasoning mechanisms, specifically in how it prioritizes numerical objectives and validates procedural structure. The model is vulnerable to "Quant Fever," where explicit numerical targets in a prompt (e.g., "delete 90% of files") override contextual safety constraints (e.g., "do not delete important files"). Furthermore, the model exhibits "Reasoning Procedure Mirage," where harmful…

Quant Fever, Reasoning Blackholes, Schrodinger's Compliance, and More: Probing GPT-OSS-20B

Evaluated models: Not reported

Search Agents Vulnerable to Unreliable Results

Published Sep 28, 2025Analyzed Oct 13, 2025Source: arXiv

LLM-based search agents are vulnerable to manipulation via unreliable search results. An attacker can craft a website containing malicious content (e.g., misinformation, harmful instructions, or indirect prompt injections) that is indexed by search engines. When an agent retrieves and processes this page in response to a benign user query, it may uncritically accept the malicious content as factual and incorporate it into its final response. This allows the agent to be used as a vector for…

SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents

Evaluated models: DeepSeek R1, Gemini 2.5 Flash, Gemini 2.5 Pro+11 more

DeepSeek R1
Gemini 2.5 Flash
Gemini 2.5 Pro
Gemma 3 27B IT
GPT-4.1
GPT-4.1 Mini
GPT-5
GPT-5 Mini
GPT-oss 120B
Kimi K2
o4-mini
Qwen 3 235B-A22B
Qwen 3 32B
Qwen 3 8B

Single Query Dynamic Output

Published Sep 26, 2025Analyzed Dec 9, 2025Source: arXiv

A vulnerability exists in Large Language Models (LLMs) and multi-label text classification systems that allows for Textual Dynamic Outputs Attacks (TDOA). This technique enables hard-label black-box attacks against systems with variable or generative output spaces (where the number of labels or specific label tokens are not fixed). The attack functions by training a surrogate model on clustered coarse-grained labels derived from the victim model's fine-grained dynamic outputs. It subsequently…

Text Adversarial Attacks with Dynamic Outputs

Evaluated models: GPT-4o, GPT-4o Mini, GPT-4.1+5 more

GPT-4o
GPT-4o Mini
GPT-4.1
Claude 3.7 Sonnet
DeepSeek V3
BERT
DistilBERT
RoBERTa

Universal Steering Jailbreak

Published Sep 26, 2025Analyzed Dec 9, 2025Source: arXiv

A vulnerability exists in the alignment mechanisms of Large Language Models (LLMs) where activation steering—the process of injecting vectors into hidden states during inference—can systematically bypass refusal safeguards. By modifying the residual stream activations at intermediate layers (typically $\lfloor L/2 \rfloor$) using the formula $\overline{\mathbf{x}}_{i}^{(l)}=\mathbf{x}_{i}^{(l)}+\alpha\mathbf{v}$, an attacker can force the model to comply with harmful requests. Research…

The Rogue Scalpel: Activation Steering Compromises LLM Safety

Evaluated models: Llama 3 8B, Llama 3.1 8B, Qwen 2.5 7B+1 more

Llama 3 8B
Llama 3.1 8B
Qwen 2.5 7B
Falcon 7B

Related research categories