Attack Surface

Application layer Research

Security research involving application interfaces and API implementations

273 matching entries

Application layer research results

273 entries

Flowchart-based LVLM Jailbreak Attack

Published Feb 28, 2025Analyzed Mar 19, 2025Source: arXiv

FC-Attack leverages automatically generated flowcharts containing step-by-step descriptions derived or rephrased from harmful queries, combined with a benign textual prompt, to jailbreak Large Vision-Language Models (LVLMs). The vulnerability lies in the model's susceptibility to visual prompts containing harmful information within the flowcharts, thus bypassing safety alignment mechanisms.

FC-Attack: Jailbreaking Large Vision-Language Models via Auto-Generated Flowcharts

Evaluated models: Claude 3.5 Sonnet 20240620, Gemini 1.5 Flash, GPT-4o 2024-08-06+4 more

Claude 3.5 Sonnet 20240620
Gemini 1.5 Flash
GPT-4o 2024-08-06
GPT-4o Mini 2024-07-18
InternVL2.5 8B
LLaVA NeXT 8B
Qwen2-VL 7B Instruct

Adaptive LLM Agent Jailbreak

Published Feb 27, 2025Analyzed Mar 19, 2025Source: arXiv

LLM agents utilizing external tools are vulnerable to indirect prompt injection (IPI) attacks. Attackers can embed malicious instructions into the external data accessed by the agent, manipulating its behavior even when defenses against direct prompt injection are in place. Adaptive attacks, which modify the injected payload based on the specific defense mechanism, consistently bypass existing defenses with a success rate exceeding 50%.

Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents

Evaluated models: Llama 3 8B, Vicuna 7B

Agent-in-the-Middle Attack

Published Feb 20, 2025Analyzed Mar 4, 2025Source: arXiv

A vulnerability exists in the communication mechanisms of Large Language Model (LLM)-based Multi-Agent Systems (LLM-MAS) enabling an Agent-in-the-Middle (AiTM) attack. An attacker can intercept and manipulate messages between agents, causing the victim agent to produce malicious outputs. The attack does not require compromising individual agents directly; instead, it leverages contextual manipulation of inter-agent communications.

Red-Teaming LLM Multi-Agent Systems via Communication Attacks

Evaluated models: GPT-3.5 Turbo, GPT-4o

Iterative Chaos Jailbreak

Published Feb 19, 2025Analyzed Mar 4, 2025Source: arXiv

Large Reasoning Models (LRMs) are vulnerable to a novel jailbreak attack, "Mousetrap," which leverages the models' reasoning capabilities to elicit harmful responses. Mousetrap uses a "Chaos Machine" to iteratively transform prompts via one-to-one mappings (e.g., character substitutions, word reversals), creating complex reasoning chains that confuse the LRM and cause it to generate unsafe outputs despite safety mechanisms. The iterative nature of the attack, combined with role-playing…

A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos

Evaluated models: Claude 3.5 Sonnet, Gemini 2.0 Flash Thinking, o1-mini

Query Code Jailbreak

Published Feb 13, 2025Analyzed Mar 4, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to QueryAttack, a novel jailbreak technique that leverages structured, non-natural query languages (e.g., SQL, URL formats, or other programming language constructs) to bypass safety alignment mechanisms. The attack translates malicious natural language queries into these structured formats, exploiting the LLM's ability to understand and process such languages without triggering safety filters designed for natural language prompts. The LLM then…

QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language

Evaluated models: DeepSeek Chat, DeepSeek R1, Gemini 1.5 Flash+11 more

DeepSeek Chat
DeepSeek R1
Gemini 1.5 Flash
Gemini 1.5 Pro
GPT-3.5 Turbo
GPT-4 Turbo
GPT-4o
Llama 3.1 70B Instruct
Llama 3.1 8B Instruct
Llama 3.2 11B Vision Instruct
Llama 3.2 1B Instruct
Llama 3.2 3B Instruct
Llama 3.3 70B Instruct
o1

Agent Pipeline Simple Hacks

Published Feb 12, 2025Analyzed Dec 9, 2025Source: arXiv

Commercial LLM-powered agents utilizing autonomous web access, memory modules, and retrieval-augmented generation (RAG) are vulnerable to indirect prompt injection and environmental manipulation. Attackers can embed malicious instructions into external data sources trusted by the agent (such as Reddit posts, public databases, or ArXiv papers). When the agent autonomously retrieves and processes this content during task execution, it executes the embedded malicious commands. This vulnerability…

Commercial llm agents are already vulnerable to simple yet dangerous attacks

Evaluated models: Not reported

LLM RAG Decoy Overthink

Published Feb 4, 2025Analyzed Dec 9, 2025Source: arXiv

A resource exhaustion and algorithmic complexity vulnerability exists in applications utilizing Reasoning Large Language Models (e.g., OpenAI o1, DeepSeek R1) that process untrusted external context (such as Retrieval-Augmented Generation systems). The vulnerability, dubbed "OverThink," allows an attacker to perform an indirect prompt injection by embedding "decoy" reasoning problems—specifically computation-intensive tasks like Sudoku puzzles or Markov Decision Processes (MDPs)—into the…

Overthink: Slowdown attacks on reasoning llms

Evaluated models: o1, o3, DeepSeek R1

AP-Test Guardrail Identification

Published Feb 3, 2025Analyzed Mar 4, 2025Source: arXiv

This vulnerability allows attackers to identify the presence and location (input or output stage) of specific guardrails implemented in Large Language Models (LLMs) by using carefully crafted adversarial prompts. The attack, termed AP-Test, leverages a tailored loss function to optimize these prompts, maximizing the likelihood of triggering a specific guardrail while minimizing triggering others. Successful identification provides attackers with valuable information to design more effective…

Peering Behind the Shield: Guardrail Identification in Large Language Models

Evaluated models: Aegis Defensive, Aegis Permissive, GPT-4o+8 more

Aegis Defensive
Aegis Permissive
GPT-4o
Llama Guard
Llama Guard 2
Llama Guard 3
Perspective
ShieldGemma 2B
ShieldGemma 9B
ShieldGemma 27B
WildGuard

Inherited GPT Policy Violations

Published Feb 3, 2025Analyzed Dec 9, 2025Source: arXiv

A policy compliance vulnerability exists in the OpenAI GPT Store ecosystem affecting Custom GPTs. The vulnerability stems from the inheritance of safety alignment weaknesses from foundational models (GPT-4 and GPT-4o) and the insufficient enforcement of usage policies during the customization and review process. Custom GPTs can be trivially manipulated to violate safety guidelines—specifically regarding Cybersecurity (malware generation), Academic Integrity (ghostwriting), and Romantic…

Towards Safer Chatbots: A Framework for Policy Compliance Evaluation of Custom GPTs

Evaluated models: GPT-4, GPT-4o

Unlearning Robustness Gap

Published Feb 1, 2025Analyzed Jan 14, 2026Source: arXiv

Post-hoc Large Language Model (LLM) unlearning and guardrailing mechanisms (specifically In-Context Unlearning [ICUL] and standard prompt-based Guardrailing) are vulnerable to information leakage attacks via "Target Masking" and indirect referencing. These systems rely on superficial semantic matching to suppress "forget sets" (specific entities or concepts). Attackers can bypass these restrictions by querying associated properties, relationships, or pseudonyms rather than the explicit target…

Alu: Agentic llm unlearning

Evaluated models: GPT-4o, Llama 2 7B, Llama 3.2 3B+2 more

GPT-4o
Llama 2 7B
Llama 3.2 3B
Qwen 2.5 14B
Phi-3

Related research categories