VLM Imperceptible Image Jailbreak

Description

A vulnerability exists in Vision-Language Models (VLMs) that allows for the bypass of safety alignment mechanisms through loss-guided adversarial image perturbations. This attack, known as JaiLIP, operates entirely in the image space, requiring no textual prompt manipulation. The vulnerability is exploited by optimizing an adversarial image using a joint objective function that minimizes the Mean Squared Error (MSE) between the clean and perturbed image while maximizing the model's loss for generating specific harmful target sequences. By utilizing tanh-space reparameterization to constrain pixel values, the attack generates imperceptible visual triggers that force the model to output toxic content, threats, or domain-specific harmful advice.

Examples

To reproduce the JaiLIP attack, an attacker performs the following optimization steps on a target VLM (e.g., BLIP-2 or MiniGPT-4):

1. Initialize: Select a clean image $x$ and a set of harmful target sentences $\mathcal{T}$ (e.g., "Step-by-step guide to [illegal activity]").
2. Transformation: Apply a change of variables to ensure valid pixel space: $w = \tanh^{-1}(2x - 1)$, where the adversarial image is defined as $x_{adv} = \frac{1}{2}(\tanh(w) + 1)$.
3. Optimization: Execute a gradient descent attack using the Adam optimizer to minimize the following loss function: $$ \mathcal{L}{total} = \mathcal{L}{MSE}(x, x_{adv}) + c \cdot \mathcal{L}{model}(M(x{adv}), \mathcal{T}) $$

• $\mathcal{L}_{MSE}$: Ensures visual imperceptibility.
• $\mathcal{L}_{model}$: Cross-entropy loss forcing the model $M$ to generate targets from $\mathcal{T}$.

4. Hyperparameters:

• Iterations: 5000
• Learning Rate: $1 \times 10^{-2}$
• Loss Multiplier ($c$): $0.01$ (for BLIP-2) or $1.0$ (for MiniGPT-4).
• Batch Size: 8

5. Execution: Feed the resulting $x_{adv}$ into the VLM without a text prompt (or with a benign prompt). The model will output the target harmful content.

Impact

Exploitation of this vulnerability allows for the automated generation of severe toxicity, hate speech, profanity, and violent threats. It also enables domain-specific attacks; for example, in transportation contexts, the model can be coerced into providing unsafe advice or incorrect reasoning about traffic scenarios. This effectively nullifies the safety alignment fine-tuning of the language model backbone.

Affected Systems

• BLIP-2 (specifically with Vicuna-13B backbone)
• MiniGPT-4 (specifically with Vicuna-13B backbone)
• Other Vision-Language Models (VLMs) that utilize frozen vision encoders coupled with Large Language Models.

Mitigation Steps

• Word-Dropping Strategy: Implement a robust alignment defense where the system generates multiple versions of the input prompt by randomly removing a small portion of tokens (10–30%). Consistency checks across these versions disrupt the specific adversarial patterns required for the jailbreak.
• Self-Verification: Introduce a post-generation verification step where the model evaluates its own output for toxicity using a standard classifier prompt before displaying the response to the user.