Attack Type

Prompt leaking Vulnerabilities

Attacks that expose sensitive prompt information

8 matching entries out of 22 in this category

Prompt leaking research results

8 entries

Instruction Serialization Leak

Updated 4/10/2026

Large Language Models (LLMs) are vulnerable to system instruction leakage when extraction requests are framed as benign formatting, encoding, or structured-output tasks. While standard alignment and refusal mechanisms successfully block direct queries for system instructions, they fail when attackers request the instructions to be rendered in alternate representations (e.g., YAML, TOML, Base64, or system logs). The model's safety filters misinterpret the request as a harmless transformation or…

Automated Framework to Evaluate and Harden LLM System Instructions against Encoding Attacks

Affects: GPT-4.1 Mini, GPT-3.5 Turbo, Gemini 2.5 Flash +1 more

Source: arXiv

Autonomous Agent Prompt Reveal

Updated 2/21/2026

A vulnerability exists in Large Language Model (LLM) deployments and multi-agent systems where an autonomous attacker agent can systematically extract hidden system prompts through self-evolving interaction strategies. The vulnerability leverages a "JustAsk" framework which utilizes Upper Confidence Bound (UCB) exploration to dynamically select and refine attack vectors from a hierarchical taxonomy of 14 atomic skills (e.g., structural formatting, authority appeals) and 14 multi-turn…

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

Affects: o1, Llama 3.1 70B Hanami X1, Phi-4 +38 more

Source: arXiv

Pervasive Multi-turn Jailbreaks

Updated 12/8/2025

Multiple open-weight Large Language Models (LLMs)—specifically those prioritizing capability over safety alignment—exhibit a critical vulnerability to adaptive multi-turn prompt injection and jailbreak attacks. While these models effectively reject isolated, single-turn adversarial inputs (averaging ~13.11% Attack Success Rate), they fail to maintain safety guardrails and policy enforcement across extended conversational contexts. By leveraging iterative strategies such as "Crescendo" (gradual…

Death by a Thousand Prompts: Open Model Vulnerability Analysis

Affects: GPT-oss 20B, Llama 3.3 70B Instruct, Mistral Large 2 +5 more

Source: arXiv

Leaked Bits Collapse Attack Queries

Updated 12/9/2025

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack…

Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs

Affects: DeepSeek R1, GPT-4o Mini 2024-07-18, Llama 4 Maverick 17B +4 more

Source: arXiv

Cyber Fine-Tuning Undermines Safety

Updated 1/14/2026

Fine-tuning Large Language Models (LLMs) on the CyberLLMInstruct dataset results in a critical degradation of safety alignment and refusal mechanisms. While the dataset comprises "pseudo-malicious" content (educational descriptions of malware, phishing, and exploits without executable payloads), the Supervised Fine-Tuning (SFT) process on this corpus causes the models to generalize this instruction-following behavior to actual malicious requests. This effectively bypasses safety guardrails…

CyberLLMInstruct: A new dataset for analysing safety of fine-tuned LLMs using cyber security data

Affects: Llama 2 70B, Llama 3 8B, Llama 3.1 8B +4 more

Source: arXiv

Logit-Forced Knowledge Extraction

Updated 12/29/2024

Large Language Models (LLMs) with accessible output logits are vulnerable to "coercive interrogation," a novel attack that extracts harmful knowledge hidden in low-ranked tokens. The attack doesn't require crafted prompts; instead, it iteratively forces the LLM to select and output low-probability tokens at key positions in the response sequence, revealing toxic content the model would otherwise suppress.

Make them spill the beans! coercive knowledge extraction from (production) llms

Affects: Code Llama 13B Instruct, Codellama-13B-python, GPT-3.5 +7 more

Source: arXiv

GPT-4v System Prompt Leakage

Updated 12/29/2024

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly effective jailbreak prompts, bypassing safety restrictions and leading to undesirable outputs, including revealing personally identifiable information from images.

Jailbreaking gpt-4v via self-adversarial attacks with system prompts

Affects: GPT-4, GPT-4V, LLaVA 1.5

Source: arXiv

AutoDAN: Interpretable LLM Jailbreak

Updated 12/29/2024

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Affects: GPT-3.5 Turbo, GPT-4, Guanaco 7B +4 more

Source: arXiv

Related research categories