Attack Type

Extraction Vulnerabilities

Techniques for extracting sensitive information from models

97 matching entries

Extraction research results

97 entries

AP-Test Guardrail Identification

Updated 3/4/2025

This vulnerability allows attackers to identify the presence and location (input or output stage) of specific guardrails implemented in Large Language Models (LLMs) by using carefully crafted adversarial prompts. The attack, termed AP-Test, leverages a tailored loss function to optimize these prompts, maximizing the likelihood of triggering a specific guardrail while minimizing triggering others. Successful identification provides attackers with valuable information to design more effective…

Peering Behind the Shield: Guardrail Identification in Large Language Models

Affects: Aegis Defensive, Aegis Permissive, GPT-4o +8 more

Source: arXiv

Agent Pipeline Simple Hacks

Updated 12/9/2025

Commercial LLM-powered agents utilizing autonomous web access, memory modules, and retrieval-augmented generation (RAG) are vulnerable to indirect prompt injection and environmental manipulation. Attackers can embed malicious instructions into external data sources trusted by the agent (such as Reddit posts, public databases, or ArXiv papers). When the agent autonomously retrieves and processes this content during task execution, it executes the embedded malicious commands. This vulnerability…

Commercial llm agents are already vulnerable to simple yet dangerous attacks

Source: arXiv

Agentic Prompt Leakage Attacks

Updated 3/4/2025

A vulnerability exists in large language models (LLMs) where insufficient sanitization of system prompts allows attackers to extract sensitive information embedded within those prompts. Attackers can use an agentic approach, employing multiple interacting LLMs (as demonstrated in the referenced research), to iteratively refine prompts and elicit confidential data from the target LLM's responses. The vulnerability is exacerbated by the LLM's ability to infer context from seemingly innocuous…

Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach

Affects: GPT-4o Mini

Source: arXiv

LLM Watermark Neutralization

Updated 12/30/2025

Large Language Model (LLM) watermarking schemes based on n-gram probability biases (specifically KGW, SynthID-Text, MinHash, and SkipHash) are vulnerable to adversarial removal during Knowledge Distillation. When a student model is trained on the output of a watermarked teacher model, it inherits the watermark's statistical biases ("radioactivity"). An attacker can exploit this inheritance by comparing the student model's output token probabilities against a base model to extract the…

Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?

Affects: GLM 4 9B Chat, Llama 7B, Llama 3.2 1B

Source: arXiv

One-Shot LLM Steering Attack

Updated 3/4/2025

Large Language Models (LLMs) are vulnerable to one-shot steering vector optimization attacks. By applying gradient descent to a single training example, an attacker can generate steering vectors that induce or suppress specific behaviors across multiple inputs, even those unseen during the optimization process. This allows malicious actors to manipulate the model's output in a generalized way, bypassing safety mechanisms designed to prevent harmful responses.

Investigating Generalization of One-shot LLM Steering Vectors

Affects: Gemma 2 2B, Gemma 2 2B IT, Llama 13B +2 more

Source: arXiv

Unlearning Relearning Attack

Updated 12/30/2025

Standard Large Language Model (LLM) unlearning techniques, specifically Negative Preference Optimization (NPO), Gradient Difference (GradDiff), and Representation Misdirection for Unlearning (RMU), fail to sufficiently flatten the loss landscape surrounding the "forgotten" weights. This sharp loss landscape allows for a "Relearning Attack," wherein an attacker can fully restore the unlearned capabilities (such as hazardous knowledge, sensitive data, or copyrighted material) by performing…

Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond

Affects: Llama 2 7B, Llama 3 8B

Source: arXiv

Unlearning Robustness Gap

Updated 1/14/2026

Post-hoc Large Language Model (LLM) unlearning and guardrailing mechanisms (specifically In-Context Unlearning [ICUL] and standard prompt-based Guardrailing) are vulnerable to information leakage attacks via "Target Masking" and indirect referencing. These systems rely on superficial semantic matching to suppress "forget sets" (specific entities or concepts). Attackers can bypass these restrictions by querying associated properties, relationships, or pseudonyms rather than the explicit target…

Alu: Agentic llm unlearning

Affects: GPT-4o, Llama 2 7B, Llama 3.2 3B +2 more

Source: arXiv

Weight Tampering Unlearning Bypass

Updated 12/9/2025

State-of-the-art machine unlearning and safety fine-tuning methods for Large Language Models (LLMs) fail to robustly remove hazardous capabilities or refusal mechanisms from model weights. While these methods suppress model outputs during standard input-output interactions, the underlying capabilities remain latent in the parameter space. An attacker with access to model weights (e.g., via open releases or leaked weights) can restore "unlearned" knowledge (such as dual-use biology hazards) or…

Model tampering attacks enable more rigorous evaluations of llm capabilities

Affects: Llama 3 8B

Source: arXiv

LLM Hate Campaign Vulnerability

Updated 2/2/2025

Large Language Models (LLMs) used in hate speech detection systems are vulnerable to adversarial attacks and model stealing, resulting in evasion of hate speech detection. Adversarial attacks modify hate speech text to evade detection, while model stealing creates surrogate models that mimic the target system's behavior.

HateBench: Benchmarking Hate Speech Detectors on LLM-Generated Content and Hate Campaigns

Affects: Baichuan 2, Dolly 2, GPT-3.5 Turbo +2 more

Source: arXiv

Surrogate Classifier Extraction

Updated 3/4/2025

Large Language Models (LLMs) employing alignment techniques for safety embed a "safety classifier" within their architecture. This classifier, responsible for determining whether an input is safe or unsafe, can be approximated by extracting a surrogate classifier from a subset of the LLM's architecture. Attackers can leverage this surrogate classifier to more effectively craft adversarial inputs (jailbreaks) that bypass the LLM's intended safety mechanisms. The attack success rate against the…

Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs

Affects: Gemma 2 9B IT, Gemma 7B IT, Granite 3.1 8B Instruct +5 more

Source: arXiv

Related research categories