Skip to main content
LLM Security Database
Skip to research details
Back to research findings
LMVD-ID: 0221d96f
Paper published November 1, 2023
Entry analyzed December 29, 2024
Paper-reported evidence
Confidence: Source-linked

The LMVD-ID is an internal research identifier, not an official CVE identifier.

GPT-4v System Prompt Leakage

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly…

Prompt layer
Jailbreak
Extraction
Prompt leaking
Black-box
Safety
API
BibTeX citation

Paper-evaluated models(3)

  • GPT-4
  • GPT-4V
  • LLaVA 1.5

Description

A system prompt leakage vulnerability in GPT-4V allows extraction of internal system prompts through carefully crafted, incomplete conversations combined with image input. Extracted prompts can be used as highly effective jailbreak prompts, bypassing safety restrictions and leading to undesirable outputs, including revealing personally identifiable information from images.

Examples

See the paper for details on the "meta-theft prompt" used to extract the system prompt and subsequent jailbreak prompts derived from it. The paper includes specific examples of both (Section 3.1 and 3.3).

Impact

Successful exploitation allows attackers to bypass safety mechanisms in GPT-4V, enabling the model to generate unsafe or harmful outputs, including the identification of individuals in images against the model's intended safety constraints. This compromises user privacy and the integrity of the system.

Affected Systems

GPT-4V (and potentially other models using similar system prompt mechanisms).

Mitigation Steps

  • Implement robust prompt validation and filtering mechanisms to prevent the injection of adversarial prompts designed to extract system prompts.
  • Regularly audit and update system prompts to minimize vulnerabilities.
  • Employ techniques to minimize the information disclosed in system prompts, ensuring they are functionally necessary without revealing sensitive details.
  • Explore methods to detect and prevent attempted extractions of system prompts.
  • Develop mechanisms for detecting and mitigating the effects of jailbreak attacks, such as prompt analysis and output filtering.

Research context and confidence

Evidence and verification
Paper-reported; independent reproduction is not documented.
Primary research source linked.
Severity
Not rated by this catalog.
Source and publication type
arXiv · Research preprint.
Peer-review status is not provided by this source.
Author and publication status
Author metadata is not stored; see the primary paper.
Threat model and attacker access
Black-box model, service, or application access.
Related deployment categories
Model APIs
Taxonomy labels only; paper-specific deployment prerequisites are not inferred.
Affected systems
GPT-4V (and potentially other models using similar system prompt mechanisms).

Research Paper

Jailbreaking gpt-4v via self-adversarial attacks with system prompts

Primary source: arXiv. Findings are reported by the cited research and have not been independently verified.

View Paper

Evidence

This entry is based on a primary research source. Its findings are paper-reported; independent reproduction and verification are not claimed.

https://arxiv.org/abs/2311.09127

Related research