Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

VLM RedTeaming Jailbreak

Published Oct 29, 2024Analyzed Dec 29, 2024Source: arXiv

Large Vision-Language Models (VLMs) are vulnerable to a novel black-box jailbreak attack, IDEATOR, which leverages a separate VLM to generate malicious image-text pairs. The attacker VLM iteratively refines its prompts based on the target VLM's responses, bypassing safety mechanisms by generating contextually relevant and visually subtle malicious prompts.

IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves

Evaluated models: MiniGPT-4 Vicuna 13B, InstructBLIP, Chameleon+10 more

MiniGPT-4 Vicuna 13B
InstructBLIP
Chameleon
LLaVA-OneVision
MiniGPT-v2
Llama 3.2 11B Vision
Llama 3.2 90B Vision
GPT-4o Mini
GPT-4o
Gemini 1.5 Pro
Gemini 2.0 Flash
Gemini 2.0 Flash Thinking
Claude 3.5 Sonnet

Benign Mirroring Jailbreak

Published Oct 28, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to stealthy jailbreak attacks leveraging benign data mirroring. Attackers train a local "mirror model" on benign data obtained from the target LLM. This mirror model, mimicking the target's behavior, is then used to generate adversarial prompts, which are subsequently deployed against the target LLM, bypassing content moderation systems due to the lack of overtly malicious content in the initial data gathering phase.

Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring

Evaluated models: GPT-3.5 Turbo, GPT-4o Mini, Llama 2 Chat+1 more

GPT-3.5 Turbo
GPT-4o Mini
Llama 2 Chat
Llama 3 8B Instruct

Self-Tuning LLM Jailbreak

Published Oct 24, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel iterative self-tuning attack (ADV-LLM) that crafts adversarial suffixes. This attack significantly reduces the computational cost of generating effective jailbreaks compared to prior methods, achieving near 100% success rate against various open-source LLMs and high success rates (e.g., 99% against GPT-3.5, 49% against GPT-4) against closed-source models. The attack leverages iterative self-tuning to improve the LLM's ability to generate…

Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B+4 more

GPT-3.5 Turbo
GPT-4
Guanaco 7B
Llama 2 7B Chat
Llama 3 8B Instruct
Mistral 7B Instruct v0.2
Vicuna 7B v1.5

Enhanced Jailbreak Transferability

Published Oct 21, 2024Analyzed Dec 29, 2024Source: arXiv

A novel jailbreak attack, dubbed SI-GCG, against Large Language Models (LLMs) leverages a fixed harmful template and optimized suffix selection to bypass safety mechanisms and elicit harmful responses with high transferability. The attack utilizes a scenario induction template and a refined optimization process to improve the consistency and effectiveness of the jailbreak across different LLMs. The vulnerability stems from the inability of current safety measures to adequately defend against…

Boosting jailbreak transferability for large language models

Evaluated models: Not reported

SMILES-Prompting LLM Jailbreak

Published Oct 21, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) used in chemical synthesis applications are vulnerable to a novel attack vector, dubbed "SMILES-prompting," which leverages the Simplified Molecular-Input Line-Entry System (SMILES) notation to bypass safety mechanisms and elicit instructions for synthesizing hazardous substances. The attack exploits the LLM's inability to effectively filter or interpret SMILES strings representing dangerous chemicals, leading to the disclosure of synthesis procedures.

SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis

Evaluated models: GPT-4o, Llama 3 70B Instruct

Faster GCG LLM Jailbreak

Published Oct 20, 2024Analyzed Dec 29, 2024Source: arXiv

Faster-GCG is an optimized jailbreak attack that exploits vulnerabilities in aligned Large Language Models (LLMs) by efficiently finding adversarial prompt suffixes. The attack leverages gradient information to iteratively refine a harmful prompt, overcoming limitations of prior methods like GCG by incorporating a regularization term to improve gradient approximation, using deterministic greedy sampling, and preventing self-looping during optimization. This allows for significantly higher…

Faster-GCG: Efficient discrete optimization jailbreak attacks against aligned large language models

Evaluated models: GPT-3.5 Turbo, GPT-4 Turbo, Llama 2 7B Chat+1 more

GPT-3.5 Turbo
GPT-4 Turbo
Llama 2 7B Chat
Vicuna 13B v1.5

Attention-Based LLM Jailbreak

Published Oct 18, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to attention-based jailbreak attacks. Attackers can craft prompts that strategically divert the LLM's attention away from sensitive words, causing the model to overlook malicious intent and generate harmful content. This occurs by leveraging the LLM's attention mechanism to focus on benign parts of the prompt while embedding harmful queries within a seemingly harmless context. The success of the attack is correlated with specific attention…

Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs

Evaluated models: Claude 3 Haiku, GPT-4, Llama 2 13B Chat+2 more

Claude 3 Haiku
GPT-4
Llama 2 13B Chat
Llama 2 7B Chat
Llama 3 8B

PC-Bias Jailbreak Vulnerability

Published Oct 17, 2024Analyzed Jul 14, 2025Source: arXiv

Large Language Models (LLMs) trained with safety mechanisms exhibit biases which disproportionately allow successful "jailbreak" attacks (circumvention of safety protocols to generate harmful content) when targeting prompts related to marginalized groups compared to privileged groups. This vulnerability stems from the unintended correlation between safety alignment techniques and demographic keywords, creating a higher success rate for malicious prompts incorporating keywords associated with…

Biasjailbreak: analyzing ethical biases and jailbreak vulnerabilities in large language models

Evaluated models: Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4+5 more

Claude 3.5 Sonnet
GPT-3.5 Turbo
GPT-4
GPT-4o
Llama 2 13B
Llama 2 7B
Llama 3 8B
Qwen 2 7B

Robotic LLM Jailbreak

Published Oct 17, 2024Analyzed Dec 28, 2024Source: arXiv

Large language models (LLMs) controlling robots are vulnerable to jailbreaking attacks. The ROBOPAIR algorithm demonstrates that malicious prompts can bypass safety mechanisms, causing robots to perform harmful physical actions. This vulnerability exploits the LLM's reliance on textual prompts and its potential lack of sufficient contextual understanding to prevent unsafe commands. The attack is effective across different access levels.

Jailbreaking LLM-controlled robots

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o+1 more

GPT-3.5 Turbo
GPT-4
GPT-4o
Nvidia Dolphins Self-driving Llm

Fuzzy Backdoor Jailbreak

Published Oct 15, 2024Analyzed Dec 29, 2024Source: arXiv

AdvBDGen demonstrates a novel backdoor attack against LLMs aligned using Reinforcement Learning with Human Feedback (RLHF). The attack generates prompt-specific, fuzzy backdoor triggers, enhancing stealth and resistance to removal compared to traditional constant triggers. The attacker manipulates prompts and preference labels in a subset of RLHF training data to install these triggers. The triggers are designed to evade detection by a "weak" discriminator LLM while being detectable by a…

AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment

Evaluated models: BERT, Gemma 7B, GPT-4+4 more

BERT
Gemma 7B
GPT-4
Llama 3 8B
Mistral 7B
Mistral 7B Instruct
Tiny Llama 1.1B

Related research categories