Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Agent Tool Selection Hijack

Published Apr 28, 2025Analyzed Jan 14, 2026Source: arXiv

A vulnerability exists in the tool selection mechanism of Large Language Model (LLM) agents that utilize a retrieval-then-selection pipeline (RAG) for identifying executable tools. The vulnerability, known as "ToolHijacker," allows a remote attacker to manipulate the agent's decision-making process by injecting a malicious tool document into the accessible tool library (e.g., via third-party tool hubs or plugins). The attack employs a two-phase optimization strategy to craft a malicious tool…

Prompt Injection Attack to Tool Selection in LLM Agents

Evaluated models: Llama 2 7B Chat, Llama 3 8B Instruct, Llama 3 70B Instruct+5 more

Llama 2 7B Chat
Llama 3 8B Instruct
Llama 3 70B Instruct
Llama 3.3 70B Instruct
Claude 3 Haiku
Claude 3.5 Sonnet
GPT-3.5
GPT-4o

Prefill-Based LLM Jailbreak

Published Apr 28, 2025Analyzed May 4, 2025Source: arXiv

Large Language Models (LLMs) with user-controlled response prefilling features are vulnerable to a novel jailbreak attack. By manipulating the prefilled text, attackers can influence the model's subsequent token generation, bypassing safety mechanisms and eliciting harmful or unintended outputs. Two attack vectors are demonstrated: Static Prefilling (SP), using a fixed prefill string, and Optimized Prefilling (OP), iteratively optimizing the prefill string for maximum impact. The vulnerability…

Prefill-Based Jailbreak: A Novel Approach of Bypassing LLM Safety Boundary

Evaluated models: Claude 3.5 Sonnet, Claude 3.7 Sonnet, DeepSeek V3+3 more

Claude 3.5 Sonnet
Claude 3.7 Sonnet
DeepSeek V3
Gemini 2.0 Flash
Gemini 2.0 Pro
GPT-3.5 Turbo

Graph-Based LLM Jailbreak

Published Apr 26, 2025Analyzed May 4, 2025Source: arXiv

Research ID 866c3b97

Large Language Models (LLMs) employing alignment safeguards and safety mechanisms are vulnerable to graph-based adversarial attacks that bypass these protections. The attack, termed "Graph of Attacks" (GOAT), leverages a graph-based reasoning framework to iteratively refine prompts and exploit vulnerabilities more effectively than previous methods. The attack synthesizes information across multiple reasoning paths to generate human-interpretable prompts that elicit undesired or harmful outputs…

Graph of Attacks: Improved Black-Box and Interpretable Jailbreaks for LLMs

Evaluated models: GPT-4, Llama 2 7B, Vicuna 13B+1 more

GPT-4
Llama 2 7B
Vicuna 13B
Vicuna 7B

LLM Censorship Vector Control

Published Apr 23, 2025Analyzed May 4, 2025Source: arXiv

Large Language Models (LLMs) employing safety mechanisms based on supervised fine-tuning and preference alignment exhibit a vulnerability to "steering" attacks. Maliciously crafted prompts or input manipulations can exploit representation vectors within the model to either bypass censorship ("refusal-compliance vector") or suppress the model's reasoning process ("thought suppression vector"), resulting in the generation of unintended or harmful outputs. This vulnerability is demonstrated…

Steering the CensorShip: Uncovering Representation Vectors for LLM" Thought" Control

Evaluated models: DeepSeek R1 Distill Qwen 1.5B, DeepSeek R1 Distill Qwen 32B, DeepSeek R1 Distill Qwen 7B+8 more

DeepSeek R1 Distill Qwen 1.5B
DeepSeek R1 Distill Qwen 32B
DeepSeek R1 Distill Qwen 7B
Gemma 2B
Gemma 7B
Llama 2 7B
Llama 3.1 8B
Qwen 1.8B
Qwen 2.5 7B
Qwen 7B
Yi 1.5 6B

MAD Amplified Jailbreaks

Published Apr 23, 2025Analyzed May 4, 2025Source: arXiv

Multi-Agent Debate (MAD) frameworks leveraging Large Language Models (LLMs) are vulnerable to amplified jailbreak attacks. A novel structured prompt-rewriting technique exploits the iterative dialogue and role-playing dynamics of MAD, circumventing inherent safety mechanisms and significantly increasing the likelihood of generating harmful content. The attack succeeds by using narrative encapsulation, role-driven escalation, iterative refinement, and rhetorical obfuscation to guide agents…

Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debate

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o

Dual Jailbreak via TDI/MTO

Published Apr 21, 2025Analyzed May 4, 2025Source: arXiv

A vulnerability exists in the combination of Large Language Models (LLMs) and their associated safety guardrails, allowing attackers to bypass both defenses and elicit harmful or unintended outputs from LLMs. The vulnerability stems from insufficient detection by guardrails against adversarially crafted prompts, which appear benign but contain hidden malicious intent. The attack, dubbed "DualBreach," leverages a target-driven initialization strategy and multi-target optimization to generate…

DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimization

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 3 8B Instruct+1 more

GPT-3.5 Turbo
GPT-4
Llama 3 8B Instruct
Qwen 2.5 7B Instruct

Chained Guardrail Bypass

Published Apr 18, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Model (LLM) agents operating in stateful environments (web browsers, operating systems, and tool-use contexts) are vulnerable to indirect prompt injection and multi-modal adversarial attacks. These vulnerabilities arise when agents process untrusted environmental observations—such as web accessibility trees, screen screenshots, or database query results—that contain concealed malicious instructions. Specifically, attackers can embed prompt injections into HTML accessibility…

DoomArena: A Framework for Testing AI Agents Against Evolving Security Threats

Evaluated models: GPT-4o, GPT-4o Mini, Claude 3.5 Sonnet+2 more

GPT-4o
GPT-4o Mini
Claude 3.5 Sonnet
Claude 3.7 Sonnet
Llama Guard

Graph-Based LLM Jailbreak

Published Apr 17, 2025Analyzed Apr 21, 2025Source: arXiv

Research ID c32aa29e

Large Language Models (LLMs) employing safety mechanisms are vulnerable to a graph-based attack that leverages semantic transformations of malicious prompts to bypass safety filters. The attack, termed GraphAttack, uses Abstract Meaning Representation (AMR), Resource Description Framework (RDF), and JSON knowledge graphs to represent malicious intent, systematically applying transformations to evade surface-level pattern recognition used by existing safety mechanisms. A particularly effective…

GraphAttack: Exploiting Representational Blindspots in LLM Safety Mechanisms

Evaluated models: Claude 3.7 Sonnet, GPT-3.5 Turbo, GPT-4+3 more

Claude 3.7 Sonnet
GPT-3.5 Turbo
GPT-4
GPT-4o
Llama 3.3 70B Instruct
Qwen 2.5 72B Instruct

GUI Agent Fine-Print Injection

Published Apr 15, 2025Analyzed Dec 30, 2025Source: arXiv

LLM-powered GUI agents utilizing screenshot-based interpretation (such as those powered by GPT-4o, Claude 3.7 Sonnet, Gemini 2.0 Flash, and DeepSeek V3 0324) are vulnerable to Fine-Print Injection (FPI) and Deceptive Default (DD) attacks due to a lack of visual saliency filtering. Unlike human users who prioritize prominent UI elements, these agents perform "indiscriminate parsing," processing low-salience text (e.g., privacy policies, terms of service, footer disclaimers) with the same…

The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections

Evaluated models: GPT-4o, Gemini 2.0 Flash, Claude 3.7 Sonnet+2 more

GPT-4o
Gemini 2.0 Flash
Claude 3.7 Sonnet
Llama 3.3 70B Instruct
DeepSeek V3 0324

LLM Guardrail Evasion

Published Apr 15, 2025Analyzed Apr 21, 2025Source: arXiv

Large Language Model (LLM) guardrail systems, including those relying on AI-driven text classification models (e.g., fine-tuned BERT models), are vulnerable to evasion via character injection and adversarial machine learning (AML) techniques. Attackers can bypass detection by injecting Unicode characters (e.g., zero-width characters, homoglyphs) or using AML to subtly perturb prompts, maintaining semantic meaning while evading classification. This allows malicious prompts and jailbreaks to…

Bypassing Prompt Injection and Jailbreak Detection in LLM Guardrails

Evaluated models: DeBERTa v3 Base, GPT-4o Mini, mDeBERTa v3 Base

Related research categories