One-Step Model Jailbreak

Description

A vulnerability in LLMs allows attackers to bypass safety mechanisms by crafting prompts that disguise malicious intent as a "defense" against harmful content. The attack, Reverse Embedded Defense Attack (REDA), leverages the model's own defensive capabilities to generate harmful outputs while masking the malicious intent within the response structure. This allows for successful jailbreaks in a single iteration, without requiring model-specific prompt engineering.

Examples

See the paper for examples of the attack vectors and generated outputs. The dataset used in the paper is also described within the paper.

Impact

Successful exploitation can lead to the generation of harmful, illegal, or unethical content by the affected LLM, bypassing its built-in safety filters. This undermines the intended safety and reliability of the model.

Affected Systems

The vulnerability impacts a wide range of LLMs, including open-source models (e.g., Vicuna-13B-v1.5-16k, Llama-3.1-8B-Instruct, Qwen2-7B-Instruct, GLM-4-9BChat) and closed-source services (e.g., ChatGPT-API, Spark-api (sparkmax), Glm-api (glm-4)). The extent of impact varies depending on the LLM's specific security implementations.

Mitigation Steps

• Enhance prompt analysis and filtering techniques to detect disguised malicious intent, focusing on semantic analysis beyond simple keyword matching.
• Implement more robust detection mechanisms for harmful content generation, even within seemingly benign contexts or responses.
• Develop and incorporate strategies to identify and reject prompt structures indicative of REDA-style attacks.
• Improve the model's ability to differentiate between genuine requests for defensive information and attempts to exploit its defensive mechanisms for malicious purposes.
• Regularly update and test model defenses against emerging attack techniques such as REDA.