Attack Type

Extraction Research

Techniques for extracting sensitive information from models

98 matching entries

Extraction research results

98 entries

LLM Fake News Generation

Published 3/1/2024

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to a novel prompting technique, "conditional Variational-autoencoder-Like Prompt" (VLPrompt), which enables the generation of highly convincing fake news articles. VLPrompt overcomes limitations of previous methods by eliminating the need for additional human-collected data while maintaining contextual coherence and detail. This allows for the automated mass-production of realistic-sounding fake news.

Exploring the deceptive power of llm-generated fake news: A study of real-world detection challenges

Evaluated models: Not reported

Source: arXiv

LLM Black-Box Fingerprinting

Published 2/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to black-box identity verification attacks using Targeted Random Adversarial Prompts (TRAP). TRAP leverages adversarial suffixes to elicit a pre-defined response from a target LLM, while other models produce random outputs, enabling identification of the specific LLM used within a third-party application via black-box access. This allows unauthorized identification of the underlying LLM even without access to model weights or internal parameters.

TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification

Evaluated models: Claude 2.1, Claude Instant 1.2, GPT-3.5 Turbo +12 more

Source: arXiv

LLM Causal Neuron Attack

Published 12/1/2023

Analyzed 12/28/2024

Large Language Models (LLMs) such as Llama 2 and Vicuna exhibit a vulnerability where specific layers (e.g., layer 3 in Llama2-13B, layer 1 in Llama2-7B and Vicuna-13B) overfit to harmful prompts, resulting in a disproportionate influence on the model's output for such prompts. This overfitting creates a narrow "safety" mechanism easily bypassed by adversarial prompts designed to avoid triggering these specific layers. Additionally, a single neuron (e.g., neuron 2100 in Llama2 and Vicuna)…

Causality analysis for evaluating the security of large language models

Evaluated models: GPT-3.5 Turbo, GPT-NeoX, Llama 2-13B-chat-hf +2 more

Source: arXiv

Logit-Forced Knowledge Extraction

Published 12/1/2023

Analyzed 12/29/2024

Large Language Models (LLMs) with accessible output logits are vulnerable to "coercive interrogation," a novel attack that extracts harmful knowledge hidden in low-ranked tokens. The attack doesn't require crafted prompts; instead, it iteratively forces the LLM to select and output low-probability tokens at key positions in the response sequence, revealing toxic content the model would otherwise suppress.

Make them spill the beans! coercive knowledge extraction from (production) llms

Evaluated models: Code Llama 13B Instruct, Codellama-13B-python, GPT-3.5 +7 more

Source: arXiv

Custom GPT Prompt Injection

Published 11/1/2023

Analyzed 12/28/2024

A prompt injection vulnerability in OpenAI's custom GPT models allows attackers to extract the system prompt and potentially leak user-uploaded files. Attackers craft malicious prompts that manipulate the LLM into revealing sensitive information, even when defensive prompts are in place. The vulnerability is exacerbated when the model includes a code interpreter.

Assessing prompt injection risks in 200+ custom gpts

Evaluated models: Not reported

Source: arXiv

GPT-4v System Prompt Leakage

Published 11/1/2023

Analyzed 12/29/2024

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly effective jailbreak prompts, bypassing safety restrictions and leading to undesirable outputs, including revealing personally identifiable information from images.

Jailbreaking gpt-4v via self-adversarial attacks with system prompts

Evaluated models: GPT-4, GPT-4V, LLaVA 1.5

Source: arXiv

AutoDAN: Interpretable LLM Jailbreak

Published 10/1/2023

Analyzed 12/29/2024

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B +4 more

Source: arXiv

Parametric Model Unalignment

Published 10/1/2023

Analyzed 12/28/2024

Large Language Models (LLMs) employing Reinforcement Learning from Human Feedback (RLHF) and instruction tuning methods may exhibit superficial safety guardrails vulnerable to parametric red-teaming attacks. Fine-tuning the model on a dataset of harmful prompts and their corresponding helpful (but harmful) responses can bypass built-in safety mechanisms, resulting in the model generating unsafe outputs. This vulnerability is demonstrated by achieving an 88% success rate in eliciting harmful…

Language model unalignment: Parametric red-teaming to expose hidden harms and biases

Evaluated models: Claude 1, Claude 2, GPT-4 +6 more

Source: arXiv

Related research categories