Attack Type

Jailbreak Research

Methods for bypassing model safety measures

704 matching entries

Jailbreak research results

704 entries

Prompt Translation Jailbreak

Published 10/1/2024

Analyzed 12/29/2024

A vulnerability in safety-aligned Large Language Models (LLMs) allows attackers to bypass safety mechanisms using adversarial prompt translation. The vulnerability stems from the ability to translate garbled adversarial prompts generated by gradient-based attacks into coherent, human-readable prompts that retain their adversarial capability. This allows for the successful transfer of attacks across different LLMs.

Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation

Evaluated models: Not reported

Source: arXiv

Robotic LLM Jailbreak

Published 10/1/2024

Analyzed 12/28/2024

Large language models (LLMs) controlling robots are vulnerable to jailbreaking attacks. The ROBOPAIR algorithm demonstrates that malicious prompts can bypass safety mechanisms, causing robots to perform harmful physical actions. This vulnerability exploits the LLM's reliance on textual prompts and its potential lack of sufficient contextual understanding to prevent unsafe commands. The attack is effective across different access levels.

Jailbreaking LLM-controlled robots

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +1 more

Source: arXiv

SMILES-Prompting LLM Jailbreak

Published 10/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) used in chemical synthesis applications are vulnerable to a novel attack vector, dubbed "SMILES-prompting," which leverages the Simplified Molecular-Input Line-Entry System (SMILES) notation to bypass safety mechanisms and elicit instructions for synthesizing hazardous substances. The attack exploits the LLM's inability to effectively filter or interpret SMILES strings representing dangerous chemicals, leading to the disclosure of synthesis procedures.

SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis

Evaluated models: GPT-4o, Llama 3 70B Instruct

Source: arXiv

Self-Tuning LLM Jailbreak

Published 10/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel iterative self-tuning attack (ADV-LLM) that crafts adversarial suffixes. This attack significantly reduces the computational cost of generating effective jailbreaks compared to prior methods, achieving near 100% success rate against various open-source LLMs and high success rates (e.g., 99% against GPT-3.5, 49% against GPT-4) against closed-source models. The attack leverages iterative self-tuning to improve the LLM's ability to generate…

Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B +4 more

Source: arXiv

Adaptive Position Jailbreak

Published 9/1/2024

Analyzed 12/29/2024

AdaPPA is a jailbreak attack that exploits the varying levels of alignment protection in LLMs at different output positions. It leverages the model's instruction-following capabilities by pre-filling the output with carefully crafted "safe" content, creating a perceived completion and lowering the model's guard before generating malicious content. The attack's effectiveness relies on the adaptive generation of both safe and harmful pre-fill content, strategically placed to exploit weaknesses…

AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs

Evaluated models: ChatGLM3 6B, GPT-4o, GPT-4o Mini +5 more

Source: arXiv

Automated LLM Fuzz Jailbreak

Published 9/1/2024

Analyzed 12/28/2024

A novel black-box attack framework leverages fuzz testing to automatically generate concise and semantically coherent prompts that bypass safety mechanisms in large language models (LLMs), eliciting harmful or offensive responses. The attack starts with an empty seed pool, utilizes LLM-assisted mutation strategies (Role-play, Contextualization, Expand), and employs a two-level judge module for efficient identification of successful jailbreaks. The attack's effectiveness is demonstrated across…

Effective and Evasive Fuzz Testing-Driven Jailbreaking Attacks against LLMs

Evaluated models: Baichuan 2 7B Chat, Gemini Pro, GPT-3.5 Turbo +4 more

Source: arXiv

Concealed Multi-Turn Jailbreak

Published 9/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel multi-turn jailbreaking attack, termed "RED QUEEN ATTACK." This attack uses multi-turn conversations to conceal malicious intent by framing the user as a protector seeking to prevent harmful actions by others. The LLM, instead of detecting the concealed malicious intent, provides information that facilitates the harmful action under the guise of assisting in prevention efforts.

RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking

Evaluated models: GPT-4o

Source: arXiv

RAG Worm Jailbreak

Published 9/1/2024

Analyzed 12/29/2024

Jailbreaking vulnerabilities in Large Language Models (LLMs) used in Retrieval-Augmented Generation (RAG) systems allow escalation of attacks from entity extraction to full document extraction and enable the propagation of self-replicating malicious prompts ("worms") within interconnected RAG applications. Exploitation leverages prompt injection to force the LLM to return retrieved documents or execute malicious actions specified within the prompt.

Unleashing worms and extracting data: Escalating the outcome of attacks against rag-based inference in scale and severity using jailbreaking

Evaluated models: Gemini 1.5 Flash

Source: arXiv

Reinforcement Learning Jailbreak

Published 9/1/2024

Analyzed 12/28/2024

PathSeeker demonstrates a novel black-box jailbreak attack against Large Language Models (LLMs) that utilizes multi-agent reinforcement learning. The attack iteratively modifies input prompts based on model responses, leveraging a reward mechanism focused on vocabulary expansion in the LLM's output to circumvent safety mechanisms and elicit harmful responses. This technique bypasses existing safety filters by encouraging the model to relax its constraints, rather than directly targeting…

PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach

Evaluated models: Claude 3.5 Sonnet, DeepSeek Chat, Deepseek-coder +14 more

Source: arXiv

RoleBreak Character Jailbreak

Published 9/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) used in role-playing systems are vulnerable to character hallucination attacks, a form of jailbreak exploiting "query sparsity" and "role-query conflict". Query sparsity occurs when prompts fall outside the model's training data distribution, causing it to generate out-of-character responses. Role-query conflict arises when the prompt contradicts the established character persona, leading to inconsistent behavior. These vulnerabilities allow attackers to elicit…

RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

Evaluated models: Claude 3 Haiku, GPT-3.5 Turbo, Llama 3 8B +1 more

Source: arXiv

Related research categories