Attack Type

Prompt leaking Vulnerabilities

Attacks that expose sensitive prompt information

13 matching entries out of 22 in this category

Prompt leaking research results

13 entries

Instruction Serialization Leak

Updated 4/10/2026

Large Language Models (LLMs) are vulnerable to system instruction leakage when extraction requests are framed as benign formatting, encoding, or structured-output tasks. While standard alignment and refusal mechanisms successfully block direct queries for system instructions, they fail when attackers request the instructions to be rendered in alternate representations (e.g., YAML, TOML, Base64, or system logs). The model's safety filters misinterpret the request as a harmless transformation or…

Automated Framework to Evaluate and Harden LLM System Instructions against Encoding Attacks

Affects: GPT-4.1 Mini, GPT-3.5 Turbo, Gemini 2.5 Flash +1 more

Source: arXiv

Autonomous Agent Prompt Reveal

Updated 2/21/2026

A vulnerability exists in Large Language Model (LLM) deployments and multi-agent systems where an autonomous attacker agent can systematically extract hidden system prompts through self-evolving interaction strategies. The vulnerability leverages a "JustAsk" framework which utilizes Upper Confidence Bound (UCB) exploration to dynamically select and refine attack vectors from a hierarchical taxonomy of 14 atomic skills (e.g., structural formatting, authority appeals) and 14 multi-turn…

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

Affects: o1, Llama 3.1 70B Hanami X1, Phi-4 +38 more

Source: arXiv

GPT Tool Misuse

Updated 12/8/2025

A vulnerability exists in OpenAI's Custom GPTs platform where the lack of effective isolation between the system context ("Expert Prompt"), external knowledge retrieval, and user input allows for unauthorized information disclosure and tool misuse. By employing specific prompt injection techniques—including Hex injection, Many-shot prefix attacks, and Knowledge Poisoning (uploading malicious files)—an attacker can bypass safety guardrails. This results in the extraction of proprietary system…

An Empirical Study on the Security Vulnerabilities of GPTs

Affects: DALL-E

Source: arXiv

Pervasive Multi-turn Jailbreaks

Updated 12/8/2025

Multiple open-weight Large Language Models (LLMs)—specifically those prioritizing capability over safety alignment—exhibit a critical vulnerability to adaptive multi-turn prompt injection and jailbreak attacks. While these models effectively reject isolated, single-turn adversarial inputs (averaging ~13.11% Attack Success Rate), they fail to maintain safety guardrails and policy enforcement across extended conversational contexts. By leveraging iterative strategies such as "Crescendo" (gradual…

Death by a Thousand Prompts: Open Model Vulnerability Analysis

Affects: GPT-oss 20B, Llama 3.3 70B Instruct, Mistral Large 2 +5 more

Source: arXiv

Leaked Bits Collapse Attack Queries

Updated 12/9/2025

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack…

Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs

Affects: DeepSeek R1, GPT-4o Mini 2024-07-18, Llama 4 Maverick 17B +4 more

Source: arXiv

Multimodal Prompt Injection

Updated 12/9/2025

Large Language Models (LLMs), including GPT-4o, LLaMA-3, and GPT-3.5-Turbo, are vulnerable to multimodal prompt injection attacks. These models fail to distinguish between system-level instructions and user-provided content within the context window. Attackers can exploit this by embedding malicious instructions in direct text, indirect sources (such as third-party webpages or PDFs), or visual inputs (images). Successful exploitation results in the model prioritizing the injected adversarial…

Multimodal Prompt Injection Attacks: Risks and Defenses for Modern LLMs

Affects: GPT-3.5, GPT-4o, Llama 3 8B +1 more

Source: arXiv

Agent Tool Metadata Lure

Updated 2/22/2026

A vulnerability exists in the tool selection mechanisms of Large Language Model (LLM) agents, identified as the "Attractive Metadata Attack" (AMA). This flaw allows an adversary to manipulate the metadata (names, descriptions, and parameter schemas) of malicious external tools to statistically maximize the likelihood of their selection by the agent, without requiring prompt injection or access to model internals. The vulnerability exploits the agent’s semantic scoring function used to map user…

Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools

Affects: GPT-4o Mini, Llama 3.3 70B Instruct, Qwen 2.5 32B Instruct +2 more

Source: arXiv

Doppelgänger Agent Hijack

Updated 12/9/2025

Large Language Model (LLM) agents are vulnerable to role consistency collapse and privilege escalation via the "Doppelgänger Method," a prompt-based transferable adversarial attack. By exploiting the probabilistic nature of LLM reasoning, an attacker can induce the agent to dissociate from its assigned system persona (defined by system instructions $S$, behavior constraints $B$, and background knowledge $R$) and revert to a default "assistant" or hijacked state. This vulnerability allows…

Doppelgänger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attack

Affects: GPT-4, GPT-4.1, GPT-4.5 Preview +6 more

Source: arXiv

LLM System Prompt Extraction

Updated 12/30/2025

Large Language Models (LLMs), including Llama-3, Falcon-3, Gemma-2, and GPT-4 variants, are susceptible to system prompt extraction attacks. The vulnerability exists due to the models' instruction-following nature, which allows remote attackers to bypass safety guardrails and retrieve the model's hidden system configuration (system prompt) verbatim. This is successfully exploited using an "Extended Sandwich Attack," where an adversarial extraction command is embedded between benign questions…

System Prompt Extraction Attacks and Defenses in Large Language Models

Affects: GPT-4, GPT-4o, Llama 3 8B +2 more

Source: arXiv

Agent Action Hijacking

Updated 3/19/2025

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

Towards Action Hijacking of Large Language Model-based Agent

Affects: Alpaca, BERT, GPT-3 +3 more

Source: arXiv

Related research categories