Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

ASCII Art Jailbreak

Published Feb 19, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) exhibit vulnerability to a novel jailbreak attack, "ArtPrompt," which leverages the models' poor ability to recognize ASCII art representations of words. By replacing sensitive words in a prompt with their ASCII art equivalents, the attacker bypasses safety filters designed to prevent the generation of harmful content.

Artprompt: Ascii art-based jailbreak attacks against aligned llms

Evaluated models: GPT-3.5 Turbo, GPT-4

Complex Cipher Jailbreak

Published Feb 16, 2024Analyzed Mar 4, 2025Source: arXiv

Large Language Models (LLMs) with advanced reasoning capabilities are vulnerable to jailbreaking attacks using novel, complex, and layered custom encryption schemes. LLMs' ability to decipher these ciphers, exceeding the capabilities of less sophisticated models, enables attackers to bypass existing safety mechanisms by encoding malicious prompts.

When" Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers

Evaluated models: Gemini 1.5 Flash, GPT-4o, Llama 3.1 70B Instruct+1 more

Gemini 1.5 Flash
GPT-4o
Llama 3.1 70B Instruct
Llama 3.1 8B Instruct

Fast Projected Gradient Jailbreak

Published Feb 14, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to efficient adversarial attacks using Projected Gradient Descent (PGD) on a continuously relaxed input prompt. This attack bypasses existing alignment methods by crafting adversarial prompts that induce the model to produce undesired or harmful outputs, significantly faster than previous state-of-the-art discrete optimization methods. The effectiveness stems from carefully controlling the error introduced by the continuous relaxation of the discrete…

Attacking large language models with projected gradient descent

Evaluated models: Falcon 7B, Falcon 7B Instruct, Vicuna 7B v1.3

Implicit Clue Jailbreak

Published Feb 14, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to an indirect jailbreak attack, termed "Puzzler," which leverages implicit clues instead of explicit malicious intent in prompts. By providing associated behaviors or hints related to a malicious query, Puzzler elicits malicious responses from the LLM, bypassing its safety mechanisms. The attack works by first obtaining "defensive measures" from the LLM against a target malicious action, then querying for the corresponding "offensive measures" that…

Play guessing game with llm: Indirect jailbreak attack with implicit clues

Evaluated models: Gemini Pro, GPT-3.5 Turbo, GPT-4+3 more

Gemini Pro
GPT-3.5 Turbo
GPT-4
GPT-4 Turbo
Llama 13B
Llama 7B

Multi-Turn Contextual Jailbreak

Published Feb 14, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a multi-round "Contextual Interaction Attack" where a series of benign preliminary questions, crafted to be semantically aligned with a malicious target query, are used to manipulate the LLM's context vector. The autoregressive nature of LLMs causes them to incorporate previous conversation rounds into their generation process, allowing the attacker to prime the model into providing harmful information in response to the final, seemingly benign…

Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks

Evaluated models: Claude 2, GPT-3.5 Turbo, GPT-4+3 more

Claude 2
GPT-3.5 Turbo
GPT-4
Llama 2 7B
Mixtral 8x7B
Vicuna 7B

Infectious Multi-Agent Jailbreak

Published Feb 13, 2024Analyzed Dec 29, 2024Source: arXiv

Multimodal Large Language Models (MLLMs) in multi-agent environments are vulnerable to "infectious jailbreak," where a single adversarial image injected into the memory of one agent can cause nearly all agents to exhibit harmful behaviors exponentially fast through agent-to-agent interaction. The adversarial image acts as a "virus," spreading via pairwise chats without further attacker intervention.

Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast

Evaluated models: GPT-4V, InstructBLIP, LLaVA 1.5

RAG Poisoning Jailbreak

Published Feb 13, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) utilizing Retrieval Augmented Generation (RAG) are vulnerable to a novel attack vector, termed "RAG Poisoning," where malicious content is injected into the external knowledge base accessed by the LLM via prompt manipulation. This allows attackers to elicit undesirable or malicious outputs from the LLM, bypassing its safety filters. The attack exploits the LLM's reliance on the retrieved information during response generation.

Pandora: Jailbreak gpts by retrieval augmented generation poisoning

Evaluated models: GPT-3.5 Turbo, GPT-4, Mistral 7B

Stealthy LLM Cold Attack

Published Feb 13, 2024Analyzed Dec 28, 2024Source: arXiv

The COLD-Attack framework allows for the generation of stealthy and controllable adversarial prompts that can bypass safety mechanisms in various Large Language Models (LLMs). The attack leverages an energy-based constrained decoding method to generate fluent and contextually coherent prompts designed to elicit harmful or unintended responses from the targeted LLM, even under constraints like specific sentiment or phrasing. This allows attacks to evade detection mechanisms solely relying on…

Cold-attack: Jailbreaking llms with stealthiness and controllability

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 13B+6 more

GPT-3.5 Turbo
GPT-4
Guanaco 13B
Guanaco-7B-hf
Llama 2 7B Chat
Llama 2-13B-chat-hf
Mistral 7B Instruct v0.2
Vicuna 13B v1.5
Vicuna 7B v1.5

Subconscious LLM Jailbreak

Published Feb 8, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel attack leveraging subconscious exploitation and echopraxia. Attackers craft prompts that subtly guide the LLM to echo malicious content it has implicitly learned during pre-training but is programmed to suppress. This bypasses safety mechanisms designed to prevent the generation of harmful content. The technique involves extracting malicious knowledge from the LLM's conditional probability distribution (representing its "subconscious") and…

Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and Echopraxia

Evaluated models: Alpaca 7B, Baichuan 2 7B Chat, Claude 2+6 more

Alpaca 7B
Baichuan 2 7B Chat
Claude 2
Falcon 7B Instruct
GPT-3.5 Turbo
GPT-4
Llama 2 13B Chat
Llama 2 7B Chat
Vicuna 7B

Role-Playing LLM Jailbreaks

Published Feb 5, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in several Large Language Models (LLMs) allowing evasion of safety filters through carefully crafted prompts leveraging role-playing scenarios. The vulnerability is exploited by prompting the LLM to adopt a specific persona or scenario (e.g., "You are a helpful assistant in a fantasy world where all actions are permitted") that overrides built-in safety restrictions, resulting in the generation of unsafe or undesirable outputs. The attack is facilitated by structured…

Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models

Evaluated models: Gemini Vision Pro, GPT-3.5 Turbo, Llama 2 7B+3 more

Gemini Vision Pro
GPT-3.5 Turbo
Llama 2 7B
LongChat 7B
MiniGPT-v2
Vicuna 13B

Related research categories