Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

In-Context LLM Jailbreak

Published Oct 10, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to In-Context Attacks (ICA) and susceptible to mitigation via In-Context Defense (ICD). ICA leverages a small number of harmful demonstration examples within a prompt to elicit harmful responses from the LLM, even if it is otherwise safety-aligned. ICD counteracts ICA by prepending safe demonstration examples to the prompt, effectively reducing the likelihood of harmful output. The effectiveness of both ICA and ICD is demonstrated across multiple LLMs.

Jailbreak and guard aligned language models with only few in-context demonstrations

Evaluated models: GPT-4 0613, Llama 2 7B Chat, Mistral-7B-v2+4 more

GPT-4 0613
Llama 2 7B Chat
Mistral-7B-v2
Mixtral 8x7B
Qwen-7B-v2
Vicuna 13B v1.5
Vicuna 7B v1.5

Automating Stealthy LLM Jailbreaks

Published Oct 3, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) employing alignment techniques remain vulnerable to "jailbreak" attacks. The AutoDAN technique automatically generates semantically meaningful prompts that bypass safety features and elicit malicious outputs from aligned LLMs, unlike previous methods producing nonsensical prompts easily detectable by perplexity checks. These prompts exploit weaknesses in the LLM's alignment, causing it to generate responses that violate intended safety constraints.

Autodan: Generating stealthy jailbreak prompts on aligned large language models

Evaluated models: GPT-3.5 Turbo, GPT-4

Low-Resource Language Jailbreak

Published Oct 3, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs), such as GPT-4, exhibit a cross-lingual vulnerability in their safety mechanisms. Translating unsafe English prompts into low-resource languages, using readily available translation APIs like Google Translate, bypasses the LLM's safety filters and elicits harmful responses with a significantly higher success rate than attacks targeting the English language directly. The vulnerability stems from an unequal distribution of safety training data across languages…

Low-resource languages jailbreak gpt-4

Evaluated models: GPT-4

Auto-Generated LLM Jailbreaks

Published Sep 19, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are susceptible to automated jailbreak attacks using a fuzzing framework that generates variations of existing jailbreak prompts. This vulnerability allows bypassing built-in safety mechanisms, leading to the generation of harmful or unintended outputs. The vulnerability stems from the LLMs' inability to consistently recognize and reject semantically similar, but subtly different prompt variations generated through automated mutation techniques.

Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts

Evaluated models: Not reported

Universal Black-Box LLM Jailbreak

Published Sep 4, 2023Analyzed Dec 28, 2024Source: arXiv

A universal black-box jailbreaking vulnerability exists in Large Language Models (LLMs) due to their susceptibility to adversarial prompts crafted using a genetic algorithm (GA). The GA optimizes a universal adversarial prompt suffix that, when appended to various user inputs, causes the LLM to generate unintended and potentially harmful outputs, bypassing safety mechanisms. This attack requires no knowledge of the LLM's internal architecture or parameters.

Open sesame! universal black box jailbreaking of large language models

Evaluated models: Llama 2 7B Chat, Vicuna 7B

Chain-of-Utterance Jailbreak

Published Aug 18, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a "Chain of Utterances" (CoU) based prompt injection attack. This attack exploits the LLM's ability to engage in multi-turn conversations and role-playing, tricking it into providing harmful or unsafe responses even when presented with safety guidelines. The attack leverages a crafted conversation between two agents ("Red-LM," a malicious agent, and "Base-LM," a seemingly helpful agent) to elicit unethical responses from the Base-LM by subtly…

Red-teaming large language models using chain of utterances for safety-alignment

Evaluated models: Not reported

LLM Cipher Jailbreak

Published Aug 12, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) such as GPT-4, while employing safety alignment techniques, exhibit vulnerability to "CipherChat" attacks. CipherChat leverages cipher prompts (e.g., ASCII, Unicode, Caesar cipher, Morse code) combined with system role descriptions and few-shot enciphered demonstrations to bypass safety mechanisms trained on natural language. This allows an attacker to elicit unsafe responses from the LLM, effectively evading safety filters. The vulnerability is amplified by the…

Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher

Evaluated models: Claude 2, Falcon-chat-180B, GPT-3.5+5 more

Claude 2
Falcon-chat-180B
GPT-3.5
GPT-3.5 Turbo
GPT-4
Llama2-chat-13B
Llama-2-chat-70B
Llama2-chat-7B

Universal Adversarial LLM Jailbreak

Published Jul 27, 2023Analyzed Dec 28, 2024Source: arXiv

Aligned large language models (LLMs) are vulnerable to a universal and transferable adversarial suffix attack. Appending a specific, automatically generated suffix to a wide range of prompts, even those requesting objectionable content, causes the models to generate harmful or objectionable responses instead of refusing the request. The attack's success rate is significantly higher on GPT-based models.

Universal and transferable adversarial attacks on aligned language models

Evaluated models: ChatGLM 6B, Claude Instant 1, Claude 2+12 more

ChatGLM 6B
Claude Instant 1
Claude 2
Falcon 7B
GPT-3.5 Turbo-0301
GPT-4-0314
Guanaco 7B
Guanaco 13B
Llama 2 7B Chat
MPT 7B
PaLM 2
Pythia 12B
Stable Vicuna
Vicuna 7B
Vicuna 13B

Automated LLM Jailbreak Framework

Published Jul 16, 2023Analyzed Dec 28, 2024Source: arXiv

The MASTER KEY framework exploits timing-based characteristics of Large Language Model (LLM) chatbot responses to infer internal defense mechanisms and automatically generate jailbreak prompts. This allows bypassing safety restrictions and eliciting responses violating usage policies, including generation of illegal, harmful, privacy-violating, and adult content. The framework utilizes a three-step process: reverse-engineering defenses via time-based analysis, creating proof-of-concept…

MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots

Evaluated models: ERNIE, GPT-3.5 Turbo, GPT-4

Prompt Engineering Jailbreak

Published May 23, 2023Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs), specifically ChatGPT versions 3.5 and 4.0, are vulnerable to prompt engineering attacks that circumvent built-in content restrictions. Attackers can craft malicious prompts, categorized into "pretending," "attention shifting," and "privilege escalation" techniques, to elicit responses containing prohibited content (e.g., instructions for illegal activities, generation of harmful content). The vulnerability stems from the LLM's inability to reliably distinguish…

Jailbreaking chatgpt via prompt engineering: An empirical study

Evaluated models: GPT-3.5 Turbo, GPT-4

Related research categories