Multi-Modal VLM Jailbreak

Description

A novel jailbreak attack, Multi-Modal Linkage (MML), exploits the vulnerability in Large Vision-Language Models (VLMs) by leveraging an "encryption-decryption" scheme across text and image modalities. MML encrypts malicious queries within images (e.g., using word replacement, image transformations) to bypass initial safety mechanisms. A subsequent text prompt guides the VLM to "decrypt" the content, eliciting harmful outputs. "Evil alignment," framing the attack within a video game scenario, further enhances the attack's success rate.

Examples

See GitHub repository https://github.com/wangyu-ovo/MML (opens in a new tab) for code and examples. Specific examples are provided within the research paper, but reproducing them requires access to the target VLMs and the described image manipulation techniques.

Impact

Successful MML attacks allow adversaries to bypass safety filters and elicit policy-violating outputs from state-of-the-art VLMs, including the generation of harmful content related to illegal activities, hate speech, violence, and more. This compromises the safety and trustworthiness of these models.

Affected Systems

Large Vision-Language Models (VLMs), including but not limited to GPT-4o, GPT-4o-Mini, QwenVL-Max-0809, and Claude-3.5-Sonnet. The vulnerability is likely present in other VLMs with similar architectures and safety mechanisms.

Mitigation Steps

• Improve model robustness to adversarial examples within both text and image modalities.
• Develop more sophisticated safety filters capable of detecting and mitigating MML-style attacks.
• Implement stricter input validation and sanitization processes.
• Invest in research into more resilient multimodal safety alignment techniques.