Prompt Leaking Vulnerabilities

The paper reports a reproducible black-box evaluation showing that adversarial user queries can cause deployed LLM applications to reveal hidden system prompts. In the authors’ measurement of 1,200 applications across six commercial platforms, 1,064 applications leaked prompt content (81.0%–93.5% per anonymized platform). This is a paper-reported result, not independently verified here. LeakBench and the official artifact repository provide defensive benchmark materials for controlled testing; no operational payload is reproduced here.

Large Language Models (LLMs) are vulnerable to system instruction leakage when extraction requests are framed as benign formatting, encoding, or structured-output tasks. While standard alignment and refusal mechanisms successfully block direct queries for system instructions, they fail when attackers request the instructions to be rendered in alternate representations (e.g., YAML, TOML, Base64, or system logs). The model's safety filters misinterpret the request as a harmless transformation or serialization task, bypassing refusal constraints and inadvertently disclosing protected instructions, API keys, and internal workflows.

OpenClaw is vulnerable to persistent memory poisoning, allowing an attacker to manipulate the agent's long-term memory store (MEMORY.md) via prompt injection. Because the autonomous agent continuously integrates this memory file as context for all subsequent reasoning and task planning, injected payloads act as durable behavioral constraints. This allows an attacker to persistently alter the agent's core policy, manipulate tool selection, and hijack future sessions without any further interaction.

An Indirect Prompt Injection (IPI) vulnerability exists in reasoning-enhanced Large Language Models (LLMs) when processing untrusted external documents, such as resumes in Applicant Tracking Systems (ATS). Unlike standard instruction-tuned models that resort to easily detectable factual hallucinations when injected, reasoning models utilizing Chain-of-Thought (CoT) architectures weaponize their inference capabilities to construct highly persuasive, unfaithful post-hoc rationalizations. They successfully justify malicious injected commands by strategically reframing subjective criteria (e.g., arguing "cultural fit" to bypass objective technical requirements). Furthermore, when subjected to logically contradictory adversarial constraints, these models exhibit "Meta-Cognitive Leakage," where the intense cognitive load of parsing the contradictions causes the model to fail at separating internal thought processes from external output, inadvertently leaking the attacker's hidden instructions into the final generated response.

Agentic LLM systems that automatically preview URLs or extract web metadata are vulnerable to implicit prompt injection, resulting in silent data exfiltration ("silent egress"). Attackers can embed adversarial instructions in unobserved web elements, such as HTML <title> tags, <meta> descriptions, or Open Graph metadata. When a user requests a summary of the URL—or when the agent automatically unfurls a linked URL in a chat—the system fetches the malicious page and flattens this metadata into the LLM's trusted context window. The agent is manipulated into invoking network-capable tools to transmit sensitive runtime context (e.g., API keys, system prompts, chat history) to an attacker-controlled endpoint. Because the exfiltration occurs entirely via background tool invocations, the agent's final textual response to the user remains benign, completely bypassing output-centric safety evaluations.

A vulnerability exists in Large Language Model (LLM) deployments and multi-agent systems where an autonomous attacker agent can systematically extract hidden system prompts through self-evolving interaction strategies. The vulnerability leverages a "JustAsk" framework which utilizes Upper Confidence Bound (UCB) exploration to dynamically select and refine attack vectors from a hierarchical taxonomy of 14 atomic skills (e.g., structural formatting, authority appeals) and 14 multi-turn orchestration patterns (e.g., semantic progression, foot-in-the-door). By treating prompt extraction as an online exploration problem, the attacker agent can bypass standard safety guardrails and "do not reveal" instructions, recovering proprietary system instructions, safety constraints, and sub-agent configurations with a high success rate (100% across 41 tested models).

A vulnerability exists in OpenAI's Custom GPTs platform where the lack of effective isolation between the system context ("Expert Prompt"), external knowledge retrieval, and user input allows for unauthorized information disclosure and tool misuse. By employing specific prompt injection techniques—including Hex injection, Many-shot prefix attacks, and Knowledge Poisoning (uploading malicious files)—an attacker can bypass safety guardrails. This results in the extraction of proprietary system instructions, the retrieval of raw contents from uploaded Knowledge files (stored in /mnt/data), and the reconstruction of backend API schemas defined in the "Actions" module. Furthermore, attackers can leverage the "Knowledge" module as an indirect injection vector (AP5), achieving a 95.4% success rate in bypassing restrictions to trigger unauthorized tool usage.

A black-box guardrail reverse-engineering vulnerability exists in Large Language Model (LLM) serving systems that employ output filtering mechanisms. The vulnerability allows remote attackers to replicate the proprietary decision-making policy and rule sets of the target's safety guardrail without direct access to model parameters. This is achieved through a technique termed Guardrail Reverse-engineering Attack (GRA), which utilizes a reinforcement learning framework combined with genetic algorithm-driven data augmentation (mutation and crossover). By iteratively querying the target system and analyzing the "purified" outputs or refusals, the attacker trains a local surrogate model. The attack prioritizes "divergence cases"—inputs where the surrogate and victim disagree—to map the victim's hidden decision boundaries. This results in a high-fidelity extraction of the safety policy (achieving >0.92 rule matching rate in testing), enabling the attacker to perform offline attacks to discover bypasses.

Multiple open-weight Large Language Models (LLMs)—specifically those prioritizing capability over safety alignment—exhibit a critical vulnerability to adaptive multi-turn prompt injection and jailbreak attacks. While these models effectively reject isolated, single-turn adversarial inputs (averaging ~13.11% Attack Success Rate), they fail to maintain safety guardrails and policy enforcement across extended conversational contexts. By leveraging iterative strategies such as "Crescendo" (gradual escalation), "Contextual Ambiguity," and "Role-Play," attackers can bypass safety filters. In automated testing, this vulnerability resulted in Attack Success Rates (ASR) increasing by 2x to 10x, reaching up to 92.78% in Mistral Large-2 and 86.18% in Qwen3-32B. The vulnerability stems from the models' inability to retain forceful rejection states or detect intent drift over long context windows.

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack algorithms (such as Greedy Coordinate Gradient or PAIR) to optimize adversarial prompts with logarithmic query complexity ($log(1/\epsilon)$) rather than linear or quadratic complexity. By analyzing the leaked reasoning steps or confidence scores, an attacker can bypass guardrails, extract system prompts, or recover "unlearned" data with orders-of-magnitude fewer queries (e.g., reducing required queries from thousands to dozens) compared to black-box attacks.