Attack Type

Jailbreak Vulnerabilities

Methods for bypassing model safety measures

703 matching entries

Jailbreak research results

703 entries

Soft Prompt Model Hijack

Updated 12/29/2024

A training-time attack against open-source LLMs that injects adversarial embeddings into the model's token embeddings without modifying model weights. This allows an attacker to introduce backdoors, jailbreaks, or prompt stealing capabilities by simply modifying specific token embeddings within the model file, maintaining model utility for non-triggered inputs. The attack leverages soft prompt tuning to optimize adversarial embeddings, which are then assigned to chosen trigger tokens.

Sos! soft prompt attack against open-source large language models

Affects: Llama 2 7B Chat, Llama 7B, Mistral 7B Instruct +2 more

Source: arXiv

Space-Induced LLM Jailbreak

Updated 12/29/2024

Appending a single whitespace character (space) or certain punctuation marks to the end of an LLM's input template can bypass safety mechanisms and cause the model to generate unsafe, biased, or factually incorrect outputs, even if the original prompt was benign. This vulnerability is due to the statistical properties of single-character tokens in the model's training data, causing unintended behavior in the model's token prediction.

Single character perturbations break llm alignment

Source: arXiv

Thousand-Leak Information Leakage

Updated 12/29/2024

Large language models (LLMs) employing safety measures like filters and alignment training remain vulnerable to information leakage via "Decomposition Attacks". These attacks decompose a malicious query into multiple benign sub-queries, eliciting responses from the LLM that, when aggregated, reveal sensitive information without triggering safety filters or producing directly harmful outputs.

Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses

Affects: Claude 3.5 Sonnet, Llama 3.1 8B Instruct, Llama Guard 3 8B

Source: arXiv

Arabizi LLM Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) exhibit vulnerability to jailbreak attacks when prompted using Arabic transliteration and Arabizi (Arabic chatspeak). While LLMs demonstrate robustness to standard Arabic prompts, even with prefix injection, the use of transliterated or Arabizi prompts bypasses safety mechanisms, leading to the generation of unsafe content. This is due to the model's learned associations with specific words in these non-standard forms, which differ from its understanding of the…

Jailbreaking llms with arabic transliteration and arabizi

Affects: Anthropic Claude-3-sonnet20240229, GPT-4o, Llama2-7-billion +2 more

Source: arXiv

Prompt layer

Jailbreak

Black-box

Bi-Modal Adversarial Jailbreak

Updated 12/28/2024

Large Vision Language Models (LVLMs) are vulnerable to a bi-modal adversarial prompt attack (BAP). BAP leverages a combined textual and visual prompt to bypass safety mechanisms and elicit harmful responses, even in models designed to resist single-modality attacks. The attack first introduces a query-agnostic adversarial perturbation to the visual prompt, making the model more likely to respond positively regardless of the text. Then, an LLM refines the textual prompt iteratively to achieve…

Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt

Source: arXiv

Black-Box Query Optimization Attack

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to a black-box query-response optimization attack (QROA). QROA iteratively refines a malicious prompt suffix using a surrogate model to maximize a reward function that measures the likelihood of eliciting harmful content from the LLM. This attack does not require access to the model's internal parameters or logits; it operates solely via standard query-response interactions.

QROA: A Black-Box Query-Response Optimization Attack on LLMs

Affects: Falcon 7B Instruct, Llama 2 7B Chat, Mistral 7B Instruct +1 more

Source: arXiv

Chat Template Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) fine-tuned using chat templates are vulnerable to ChatBug, allowing malicious actors to bypass safety mechanisms by crafting prompts that intentionally deviate from the expected template format or overflow message fields. This exploits the LLM’s reliance on the template structure without enforcing similar constraints on user input.

ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates

Affects: Claude 2.1, GPT-3.5 Turbo

Source: arXiv

Code-Switching LLM Jailbreak

Updated 12/28/2024

Large Language Models (LLMs) exhibit increased vulnerability to adversarial prompts employing code-switching techniques, where multiple languages are interwoven within a single query. This vulnerability stems from an unintended correlation between the resource availability of the languages used in the prompt and the LLM's safety alignment. LLMs trained on imbalanced multilingual data are more susceptible to attacks leveraging low-resource languages, resulting in a higher rate of unsafe or…

Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding

Source: arXiv

Covert LLM Backdoor Finetuning

Updated 12/29/2024

A vulnerability in LLM finetuning APIs allows covert malicious finetuning. Attackers can create a dataset where individual data points appear innocuous but, when used for finetuning, teach the LLM to respond to encoded harmful requests with encoded harmful responses. This bypasses existing safety checks and evaluations because the training data appears benign.

Covert malicious finetuning: Challenges in safeguarding llm adaptation

Affects: GPT-3.5 Turbo, GPT-4, Llama 2 70B

Source: arXiv

DRL-Guided LLM Jailbreak

Updated 12/29/2024

A deep reinforcement learning (DRL) based attack, termed RLbreaker, demonstrates the ability to more efficiently generate jailbreaking prompts for large language models (LLMs) than existing methods. The attack leverages a DRL agent to guide the search for effective prompt structures, bypassing safety mechanisms and eliciting undesirable responses to harmful questions. The effectiveness stems from the DRL agent's ability to strategically select prompt mutators, rather than relying on random…

When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search

Affects: GPT-3.5 Turbo, Llama 2 70B Chat, Llama 2 7B Chat +3 more

Source: arXiv

Related research categories