Attack Surface

Application layer Research

Security research involving application interfaces and API implementations

273 matching entries

Application layer research results

273 entries

Voice-Based GPT-4 Jailbreak

Published May 29, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in the voice mode of GPT-4o allows bypassing safety restrictions through a novel "Voice Jailbreak" attack. This attack leverages principles of fictional storytelling (setting, character, plot) to craft audio prompts that persuade the LLM to generate responses violating OpenAI's usage policies, including generating content related to illegal activities, hate speech, physical harm, fraud, pornography, and privacy violations. The attack's success rate is significantly higher than…

Voice Jailbreak Attacks Against GPT-4o

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o

Automated T2I Jailbreak

Published May 26, 2024Analyzed Dec 29, 2024Source: arXiv

Commercial text-to-image (T2I) generative AI systems are vulnerable to automated prompt injection attacks that bypass their safety mechanisms and induce the generation of copyrighted material. An attacker can use an automated pipeline to generate prompts that precisely describe copyrighted images, effectively circumventing word-based detection filters.

Automatic Jailbreaking of the Text-to-Image Generative AI Systems

Evaluated models: DALL-E 3

Self-Explanatory LLM Jailbreak

Published May 21, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in large language models (LLMs) allows for near-perfect jailbreaking via iterative prompt refinement and self-explanation. The attacker uses the LLM itself to iteratively refine adversarial prompts by requesting self-explanations of failed attempts, ultimately generating prompts that bypass safety mechanisms and elicit harmful content. A subsequent "Rate+Enhance" step further maximizes the harmfulness of the generated output.

GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanation

Evaluated models: Claude 3 Opus, Claude 3 Sonnet, GPT-4+5 more

Claude 3 Opus
Claude 3 Sonnet
GPT-4
GPT-4 Turbo
GPT-4o
Llama 3 8B
Llama 3.1 70B
Llama 3.1 8B

Personalized Encryption Jailbreak

Published Feb 26, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in several Large Language Models (LLMs) allowing attackers to bypass safety and ethical protocols through a novel code injection technique using personalized encryption and decryption functions. The attack leverages the LLMs' code execution capabilities to process encrypted malicious instructions, circumventing the intent security recognition mechanism.

Codechameleon: Personalized encryption framework for jailbreaking large language models

Evaluated models: GPT-3.5 Turbo, GPT-4-1106, Llama 2 13B Chat+4 more

GPT-3.5 Turbo
GPT-4-1106
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Vicuna 13B
Vicuna 7B

Cognitive Consistency Jailbreak

Published Feb 24, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in several large language models (LLMs) allows attackers to bypass safety restrictions ("jailbreaking") by employing a Foot-in-the-Door (FITD) technique. This involves progressively escalating prompts, starting with innocuous requests and gradually leading to the elicitation of harmful or restricted information. The LLM's tendency towards cognitive consistency makes it more likely to respond to subsequent, increasingly sensitive prompts after initially agreeing to less harmful…

Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology

Evaluated models: Chatglm-2 (chatglm2-6B), Chatglm-3 (chatglm3-6B), Claude 2.1+5 more

Chatglm-2 (chatglm2-6B)
Chatglm-3 (chatglm3-6B)
Claude 2.1
Claude Instant 1.2
Gemini (gemini-pro)
GPT-3.5 (GPT-3.5-turbo-1106)
GPT-4 (GPT-4-1106-preview)
Llama-2 (llama2-7B-chat)

Universal Guardrail Bypass

Published Feb 24, 2024Analyzed Dec 29, 2024Source: arXiv

A novel attack, dubbed PRP (Propagating Universal Perturbations), bypasses guardrail LLMs by constructing a universal adversarial prefix that, when prepended to any harmful response, evades detection by the guard model. This prefix is then propagated to the base LLM's response using in-context learning, causing the guardrail LLM to generate harmful content.

Prp: Propagating universal perturbations to attack large language model guard-rails

Evaluated models: Gemini Pro, GPT 3.5-turbo-0125, Guanaco 13B+5 more

Gemini Pro
GPT 3.5-turbo-0125
Guanaco 13B
Llama 2 70B Chat
Mistral 7B Instruct
Vicuna-33B-v1.3
Wizard-lm-falcon-7B-uncensored
Wizardlm7B-uncensored

Universal LLM Score Inflation

Published Feb 21, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) used for zero-shot text assessment are vulnerable to universal adversarial attacks. Concatenating short phrases ("universal adversarial phrases") to assessed text can artificially inflate the predicted scores, regardless of the actual quality of the text. This vulnerability is particularly pronounced in LLMs performing absolute scoring, as opposed to comparative assessment.

Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment

Evaluated models: Flan-T5 XL, GPT-3.5, Llama 2 7B+1 more

Flan-T5 XL
GPT-3.5
Llama 2 7B
Mistral 7B

LLM Black-Box Fingerprinting

Published Feb 20, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to black-box identity verification attacks using Targeted Random Adversarial Prompts (TRAP). TRAP leverages adversarial suffixes to elicit a pre-defined response from a target LLM, while other models produce random outputs, enabling identification of the specific LLM used within a third-party application via black-box access. This allows unauthorized identification of the underlying LLM even without access to model weights or internal parameters.

TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification

Evaluated models: Claude 2.1, Claude Instant 1.2, GPT-3.5 Turbo+12 more

Claude 2.1
Claude Instant 1.2
GPT-3.5 Turbo
GPT-4
GPT-4 Turbo
Guanaco 7B
Guanaco 13B
Llama 2 7B Chat
Llama 2 13B Chat
Llama 2 70B Chat
Mixtral 8x7B
Nous Hermes 2 Mixtral 8x7B DPO
OpenChat 3.5
Vicuna 7B
Vicuna 13B

RAG Poisoning Jailbreak

Published Feb 13, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) utilizing Retrieval Augmented Generation (RAG) are vulnerable to a novel attack vector, termed "RAG Poisoning," where malicious content is injected into the external knowledge base accessed by the LLM via prompt manipulation. This allows attackers to elicit undesirable or malicious outputs from the LLM, bypassing its safety filters. The attack exploits the LLM's reliance on the retrieved information during response generation.

Pandora: Jailbreak gpts by retrieval augmented generation poisoning

Evaluated models: GPT-3.5 Turbo, GPT-4, Mistral 7B

Multimodal Model Jailbreak

Published Feb 4, 2024Analyzed Dec 28, 2024Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a jailbreaking attack using crafted images (image Jailbreaking Prompts or imgJPs). These imgJPs, when presented as input alongside malicious prompts, cause the MLLM to bypass safety mechanisms and generate objectionable content, including instructions for harmful activities like identity theft or creation of violent video games. The attack demonstrates both prompt-universality (a single imgJP works across multiple prompts) and, to a…

Jailbreaking attack against multimodal large language model

Evaluated models: InstructBLIP, MiniGPT-4, MiniGPT-v2+3 more

InstructBLIP
MiniGPT-4
MiniGPT-v2
Mplug-owl2
Vicuna 13B
Vicuna 7B

Related research categories