Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Knowledge Neuron Jailbreak

Published Sep 1, 2025Analyzed Dec 8, 2025Source: arXiv

Aligned Large Language Models (LLMs) utilizing Transformer architectures are vulnerable to representation-level attacks targeting safety-knowledge neurons within the Multi-Layer Perceptron (MLP) layers. Research indicates that safety decision-making (Rejection vs. Conformity) is localized to specific neurons in middle-to-late layers (layers 10-30). An attacker with white-box access can calculate a "Conformity" direction vector based on the activation differences between benign and harmful…

Unraveling LLM Jailbreaks Through Safety Knowledge Neurons

Evaluated models: Llama 2 7B, Vicuna 7B

MAS Link Deception

Published Sep 1, 2025Analyzed Dec 9, 2025Source: arXiv

The evaluated MetaGPT multi-agent systems are vulnerable to "Web Fraud Attacks" due to insufficient semantic and structural validation of Uniform Resource Locators (URLs) by agentic models. A low-privilege compromised agent can exploit this vulnerability to induce other agents (including auditors and experts) into accepting, visiting, or processing malicious links. The vulnerability leverages the LLM's inability to distinguish between benign and malicious link structures when obfuscation…

Web fraud attacks against llm-driven multi-agent systems

Evaluated models: GPT-4o Mini, Gemini 2.5 Flash, DeepSeek Reasoner+1 more

GPT-4o Mini
Gemini 2.5 Flash
DeepSeek Reasoner
Llama 3 8B

Logit Leakage Model Clone

Published Aug 31, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Model (LLM) inference APIs that expose top-k logits or log-probabilities are vulnerable to model extraction and cloning. An attacker can execute a two-stage attack to replicate the proprietary model without access to weights, gradients, or training data. First, by submitting fewer than 10,000 random queries and aggregating the returned unrounded logits, the attacker recovers the model's output projection matrix using Singular Value Decomposition (SVD). Second, the attacker…

Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation

Evaluated models: GPT-3.5, Mistral 7B

GCG Code Reasoning Exploit

Published Aug 30, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), specifically Qwen2.5-0.5B-Instruct, LLaMA-3.2-1B-Instruct, and GPT-OSS-20B, are vulnerable to gradient-based adversarial attacks, including the Greedy Coordinate Gradient (GCG) algorithm and its annealing-augmented variant, T-GCG. Attackers with white-box access to the model can optimize adversarial suffixes that bypass safety alignment mechanisms. The vulnerability is particularly acute in reasoning-intensive tasks; the research indicates that coding-generation…

The Resurgence of GCG Adversarial Attacks on Large Language Models

Evaluated models: GPT-oss 20B, Llama 3.2 1B Instruct, Qwen 2.5 0.5B Instruct

Adaptive Role-Play Jailbreak

Published Aug 28, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) and Vision-Language Models (VLMs) are vulnerable to an automated, adaptive role-play jailbreak attack known as GUARD (Guideline Upholding Test through Adaptive Role-play and Jailbreak Diagnostics). The vulnerability exists because the models fail to recognize malicious intent when harmful queries are embedded within complex, iteratively optimized "playing scenarios."

GUARD: Guideline Upholding Test through Adaptive Role-play and Jailbreak Diagnostics for LLMs

Evaluated models: Vicuna 13B, LongChat 7B, Llama 2 7B+5 more

Vicuna 13B
LongChat 7B
Llama 2 7B
Llama 3 8B
GPT-3.5
GPT-4
GPT-4o
MiniGPT-v2

Malicious Intent Bypass

Published Aug 27, 2025Analyzed Dec 9, 2025Source: arXiv

IntentionReasoner, specifically the 1.5B and 3B parameter versions optimized via Reinforcement Learning (RL), contains a safety regression vulnerability where the RL alignment process degrades the model's resistance to jailbreak attacks compared to the Supervised Fine-Tuning (SFT) baseline. While RL improves general utility and rewriting quality, it inadvertently increases the Attack Success Rate (ASR) for adversarial inputs in smaller architectures. This allows sophisticated jailbreak prompts…

IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement

Evaluated models: GPT-4o, Qwen 2.5 7B Instruct, Llama 3.1 8B Instruct+3 more

GPT-4o
Qwen 2.5 7B Instruct
Llama 3.1 8B Instruct
DeepSeek V3
IntentionReasoner 1.5B
IntentionReasoner 3B

LLM Agent TOCTOU Vulnerabilities

Published Aug 23, 2025Analyzed Aug 31, 2025Source: arXiv

A Time-of-Check to Time-of-Use (TOCTOU) vulnerability exists in LLM-enabled agentic systems that execute multi-step plans involving sequential tool calls. The vulnerability arises because plans are not executed atomically. An agent may perform a "check" operation (e.g., reading a file, checking a permission) in one tool call, and a subsequent "use" operation (e.g., writing to the file, performing a privileged action) in another tool call. A temporal gap between these calls, often used for LLM…

Mind the Gap: Time-of-Check to Time-of-Use Vulnerabilities in LLM-Enabled Agents

Evaluated models: GPT-4o

Universal Transferable LLM Jailbreak

Published Aug 20, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), including Llama 2, Mistral, and Vicuna, are susceptible to a white-box adversarial attack that circumvents safety alignment mechanisms (such as RLHF). The vulnerability exists due to the models' susceptibility to intrinsic optimization of adversarial suffixes using Exponentiated Gradient Descent (EGD). Unlike previous methods that rely on inefficient discrete token searches (e.g., Greedy Coordinate Gradient) or standard projected gradient descent, this attack…

Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent

Evaluated models: GPT-3.5, GPT-4o, Llama 2 7B+3 more

GPT-3.5
GPT-4o
Llama 2 7B
Mistral 7B
Falcon 7B
Vicuna 7B

Markovian Adaptive Jailbreak

Published Aug 18, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to an adaptive black-box jailbreaking framework named MAJIC (Markovian Adaptive Jailbreaking via Iterative Composition). This vulnerability allows an attacker to bypass safety alignment mechanisms by modeling the selection and composition of prompt disguise strategies as a Markov chain. Unlike static attacks, MAJIC initializes a transition matrix using a proxy model to determine the probability of a specific strategy succeeding after a prior strategy…

MAJIC: Markovian Adaptive Jailbreaking via Iterative Composition of Diverse Innovative Strategies

Evaluated models: Qwen 2.5 7B Instruct, Gemma 2 9B IT, Gemini 2.0 Flash+2 more

Qwen 2.5 7B Instruct
Gemma 2 9B IT
Gemini 2.0 Flash
GPT-4o
Claude 3.5 Sonnet

Universal Prompt Disables Guardrails

Published Aug 18, 2025Analyzed Aug 31, 2025Source: arXiv

A universal prompt injection vulnerability, termed "Involuntary Jailbreak," affects multiple large language models. The attack uses a single prompt that instructs the model to learn a pattern from abstract string operators (X and Y). The model is then asked to generate its own examples of questions that should be refused (harmful questions) and provide detailed, non-refusal answers to them, in order to satisfy the learned operator logic. This reframes the generation of harmful content as a…

Involuntary Jailbreak

Evaluated models: Claude 3.5 Haiku, Claude 3.7 Sonnet, Claude Opus 4+29 more

Claude 3.5 Haiku
Claude 3.7 Sonnet
Claude Opus 4
Claude Opus 4.1
Claude Sonnet 4
Claude Sonnet 4.5
DeepSeek R1
DeepSeek R1 Distill Llama 70B
DeepSeek V3
Gemini 2.0 Flash
Gemini 2.5 Flash
Gemini 2.5 Flash-Lite
Gemini 2.5 Pro
Gemini 3 Pro
GPT-4.1
GPT-4.1 Mini
GPT-4o
GPT-oss 20B
Grok 3
Grok 3 Fast
Grok 3 Mini
Grok 4
Grok 4.1
Llama 3.1 8B
Llama 3.1 405B
Llama 3.3 70B
Llama 4 Scout
Llama 4 Maverick
Mistral Small 24B
Qwen 2.5 72B Instruct
Qwen 3 235B-A22B Instruct 2507
Qwen 3 Coder 480B-A35B Instruct

Related research categories